I have one server whre SSHing to it is bloody slow, how do I troubleshoot this?

[XavierMace]

That sounds way more complicated and involved per machine than I want to get into.

LOL, I don't get you sometimes. Why do you have all this equipment? Every single thread of yours where you ask for help with your network and people make suggestions, you respond with that's too complicated/hard. Yet you're making your own monitoring software rather than using one of the many widely available free applications? You've got a very convoluted setup. If you don't want complicated, get rid of your excess hardware. Just run a single host with internal storage. If you're actually wanting to learn proper network storage management (which doesn't seem to be the case), you can always make an "inception" SAN where you pass through all your storage to a VM and have the VM hand out NFS/iSCSI shares. That's still vastly simpler than your current setup. Otherwise you may have to start doing more complicated/involved things.

 

[Red Squirrel]

Local storage is not really scalable. Also there is no point to virtualizing storage. No matter what you need a physical machine with physical drives somewhere along the line, it only makes sense to make that machine the file server. My goal is to eventually have multiple VM hosts with centralized storage. But free vmware is very limited support (no HA etc) so I want to eventually figure out KVM/Qemu but that too is rather complicated to setup properly. I'll probably want to write some kind of front end. Just because I have equipment does not mean I want to overcomplicate my life, I still like to seek the easiest way of doing stuff.

Often times writing your own software is easier than trying to figure out an existing one, especially ones that are more complicated and involved to setup. My philosophy with any software I write is that everything is server side so you only need to set it up once, and it's setup to be easy to deploy by having as little dependencies as possible. For my monitoring software the only thing I need to do on the client is extract the agent, compile it, point it to the IP of the server and then everything else works from there. I can then setup monitoring points on the server and it updates the agent. Most of the solutions I had looked at were complex to setup because of all the dependencies, and then you had to do a lot of stuff client side too each time you wanted to make changes. Basically the time spent trying to figure out an existing solution was spent coding my own.

I eventually want to write some kind of management system for Linux in general (Ex: setting up mail accounts, virtual hosts etc) to make it so settings for each server can be centralized and it would manage everything. Would make it easier to deploy new servers, or move to a new server etc. Basically something like cpanel without the price tag.

But anyway with all this, the original problem is fixed now. That setting in SSH to tell it not to use DNS seems to have made the logins faster. The storage latency issue is a whole different beast and has been since the start. I kind of gave up on it. When I upgrade my UPS to dual conversion (more reliable than standby) then I'm just going to use async mode if I can figure out a way to make that setting server side. That is probably going to make it faster. I just don't want to use it now since if power goes out and the UPS does not catch it fast enough (had it happen a few times) then all my stuff will go corrupt.

I also have a 4 port nic coming and already have another switch so I will move storage to it's own network to see if it helps. Basically treating the file server more as a SAN. It was my original intent anyway. It's just that it also serves lot of raw files so it made no sense to make a VM for that when I can just access it directly. Will probably still keep doing that and it will just use the management interface.