Linux Server Load Help

Discussion in 'Operating Systems' started by Paperlantern, Jun 14, 2012.

  1. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Ive been using Linux for years, and have been administering a couple of my own Minecraft servers now for over a year. They are all working, im not concerned too much on Minecraft itself but rather why the server seems to always have a significant load on it. This is CentOS 5.6. I'm wondering if it is because there is an open session on the server now to run one server in an active terminal window rather than in the background like a service.

    Basically there are two Minecraft servers running on the same system, one runs as as service since it is just a basic server i was able to do that. The other is a modded server that needs to be kicked off from a script in a terminal window. Each runs on its own port, and both run just fine, BUT, even when NO ONE is playing on EITHER server, the box sits at a load of between 0.25 and 0.85. Most often i see it around 0.38 or 0.42, somewhere in there, the range given before are the extremes i have seen while sitting idle. I expect there to be a MINOR load on it since its sitting in an active session with windows open, but I didnt expect it to exceed around 0.10 for something like that.

    if i look at top, i see the following


    Code:
    top - 14:57:34 up 14 days, 23:10,  3 users,  load average: 0.54, 0.41, 0.37
    Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie
    Cpu(s):  1.8%us,  0.3%sy,  0.0%ni, 94.9%id,  3.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   1554812k total,  1495300k used,    59512k free,   125800k buffers
    Swap:  1048568k total,   678356k used,   370212k free,   211956k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     3579 root      18   0 1332m 940m 3652 S  2.7 61.9 411:51.24 java
     4009 root      19   0 1231m  54m 3492 S  1.7  3.6 518:10.34 java
    17545 root      15   0 10100 2960 2392 S  0.3  0.2   0:01.00 sshd
    17573 root      15   0  2428 1060  800 R  0.3  0.1   0:09.04 top
        1 root      15   0  2160  576  552 S  0.0  0.0   0:03.33 init
        2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.94 migration/0
        3 root      34  19     0    0    0 S  0.0  0.0   0:00.11 ksoftirqd/0
        4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.73 migration/1
        5 root      34  19     0    0    0 S  0.0  0.0   0:00.02 ksoftirqd/1
        6 root      10  -5     0    0    0 S  0.0  0.0   0:00.30 events/0
        7 root      10  -5     0    0    0 S  0.0  0.0   0:00.18 events/1
        8 root      10  -5     0    0    0 S  0.0  0.0   0:00.31 khelper
        9 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
       13 root      10  -5     0    0    0 S  0.0  0.0   0:00.32 kblockd/0
    
    Obviously i expect to see java sitting near the top but its only using about 2.5% in one instance, and 1.5 or so (showing 2.7 and 1.7 respectively in this capture). As i understand load averages, the three numbers are 1 minute, 5 minute and 15 minute averages, and 1.00 represents one pegged processor core. This is a dual core machine so 2.00+ would be 100%, which means to me that 0.50 would be about a 25% load, give or take. Except the top processes really are only using 4% - 5% cpu. Where is all the rest of the load coming from?

    Is there some techniques or tools I can use that only being a low intermediate linux user i may not know about that will show me a better representation of where my load is coming from. I use the System monitor in the actual session and i see more like what I'd expect to see with those loads, one core at 15% - 20% and the other core at 1% - 6% or so.

    If I stop both instances of the Minecraft server i see no difference in load. It is very odd and its making me feel like maybe someones compromised it and is using it for something else. Maybe that's just my paranoia, though I did disabled my incoming ssh and RDP rules that I had in place for ease of access to my servers. i plan on putting in a VPN solution now.

    I just remember when I first loaded up the server and was hosting these servers for my friends, the loads were very often less than 0.05, and often times all three figures were 0.00 when i logged in and checked top via ssh.

    Any suggestions or things to look for, ideas or concerns voiced will be appreciated. Thanks.
     
    #1 Paperlantern, Jun 14, 2012
    Last edited: Jun 14, 2012
  2. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Hmmm, I'm almost thinking its the open session or at least 2/3 of it is. If i reboot, leave the terminal window server shut down and just let the service instance run, so there is no active session, the load average stays pretty well at or below 0.25, and often stays below 0.20. but i swear it was 0.00 0.00 0.00 NEARLY every time i logged into it before to check it.

    Still freaks me out to see 0.20 on the load for the system when its doing basically nothing but being idle.
     
  3. cppguru

    cppguru Member

    Joined:
    Jun 14, 2012
    Messages:
    26
    Likes Received:
    0
    First thing I would like to point out - You are on a completely wrong forum to be posting this to begin with. Infact you are on a wrong URL altogether. You are better off posting this here: http://ubuntuforums.org/.

    Back to your question, Being a long time linux user myself, I would suggest you get htop. If you are on ubuntu, pop open your terminal and type this:

    Code:
    sudo apt-get install htop
    See if that helps and report back.

    It is.

    The gnome desktop takes a few cpu cycles if you are not running a GPGPU with proper drivers staged into linux kernel. You are freaking out in ignorance.
     
    #3 cppguru, Jun 14, 2012
    Last edited: Jun 14, 2012
  4. mv2devnull

    mv2devnull Senior member

    Joined:
    Apr 13, 2010
    Messages:
    939
    Likes Received:
    0
    CPU usage is not the only cause for load. Processes waiting for a device (IO, disk) do count into it too, and they obviously cannot use CPU. For example, you do have swap in use and that means additional background disk activity.

    I would check SMART from the drives.

    SSH is essentially a form of VPN. VPN-encryption of SSH-encrypted packets does not add strength to the encryption. It is good practice though to narrow down what SSH connection attempts are accepted.
    http://wiki.centos.org/HowTos/Network/SecuringSSH
     
  5. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    First, it isn't Ubuntu, it's CentOS, in retrospect I did omit that from the OP and will add it, though I fail to see how a post about an operating system, LINUX in general, is in a COMPLETELY WRONG FORUM on a techie forum categorized under Operating Systems. Would the new user that has been here for less than a month and has 10 posts, please explain this to the 1800+ post user that's been posting here for 9 years. The only other place I could have put it was perhaps *nix Software, though I'm not REALLY asking so much about any specific software, its more of a Linux question on the whole, ie the "operating system".

    I will say I don't much like the condescending structure to your post, most people here, like the post by mv2devnull, stay neutral in "tone" and give people positive points and offer help in a lighter manor. I do the same when I help people on topics that I am knowledgeable in, expecting to receive the same treatment when I am looking for guidance. I would advise you to try to be a little less brash in future posts. Often people looking for help DON'T know what they are talking about or have much knowledge on it in general and often may use incorrect terms, look for things in wrong places, make incorrect assumptions, or misinterpret things. That is why they are here asking a question. Immediately bashing how and where they asked the question wont put you high on thier list of people to listen to, no matter how much you know.

    As for the information you did provide I do thank you for pointing out that Gnome uses some cycles, that confirms my suspicion in my post above yours that just having the session open is enough to at least tick up the load a little. I also feel a little better that the server has not been compromised so thank you for that.

    To mv2devnull, thanks I'll see if i can look at SMART on the drives, though its a VM running on an ESXi host, so that could be tricky. I will also look at the wiki page you provided, though I wasn't planning on VPNing SSH, just access to the Windows server I have. I simply turned SSH off because well, i was paranoid.
     
  6. Crusty

    Crusty Lifer

    Joined:
    Sep 30, 2001
    Messages:
    12,688
    Likes Received:
    0
    What is the hardware config of your VM?
     
  7. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Do you mean just that it has 1 processor with 2 cores, 1.5GB of memory alloted and a 60GB drive? or are you looking for more detail, like the TYPE of drive I configured, the type of network card etc etc?
     
  8. theevilsharpie

    theevilsharpie Platinum Member

    Joined:
    Nov 2, 2009
    Messages:
    2,321
    Likes Received:
    0
    Load average calculations in Linux can be confusing, because it sums the run queue with the number of processes in an uninterruptible sleep state. If you have varying I/O latency (particularly network latency), the load average can be higher expected.

    The load average that you're reporting is actually fairly low, so unless you're having an unexplained performance problem, I wouldn't read too much into it.
     
  9. cppguru

    cppguru Member

    Joined:
    Jun 14, 2012
    Messages:
    26
    Likes Received:
    0
    That's good and you're welcome.

    My intent was not to brush you off but instead to redirect you to a better place to take your question where you can get a proper legitimate answer as opposed to just people's opinions. But you are immediately brushing my advise right off the bat. My advise is 100% genuine because I am a member in ubuntu forums and also here as you can see. And Yes I know that your question is "Operating System" related and not ubuntu related specifically but my advise still stands. People there are better equipped with dealing with any linux questions than anandtech forums anytime of anyday - it's just a fact.

    But it seems you were crossed by the fact that someone with less number of posts than you was trying to give you an advise. Either way you are not doing me any favour by listening or not listening. Good luck.
     
    #9 cppguru, Jun 15, 2012
    Last edited: Jun 15, 2012
  10. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    I didn't brush your advice off, I just havent had time to sit down and create an account on the Ubuntu forums. I dont use Ubuntu so I've never created an account there, but with your advice here it may be worth while.

    My specific point was the way you answered the question was quite condescending for someone who has hardly been here for 5 minutes. I get that you are a long time user of Linux, and I respect your answers, but to say

    rather than something like

    See the difference? Still tells me that I would likely get better answers there than I would here, but as a helpful suggestion rather than a snooty remark. Either way I do appreciate the educated suggestion of using that forum as a resource for all things Linux, not just Ubuntu, from a member there.

    Moving forward, I left the Gnome session closed overnight to see how the server would settle, IF it would settle, and top still shows about a 0.25 load at any one time. Makes sense that the average dropped by ABOUT half since i shut one server off. Minecraft has evolved, perhaps its just doing more now at idle?

    I also went ahead and used wget to get htop and installed it. It is a lot better than regular top that is for damn sure, thank you cppguru. I will likely use this on most installations of Linux I use in the future.
     
  11. mv2devnull

    mv2devnull Senior member

    Joined:
    Apr 13, 2010
    Messages:
    939
    Likes Received:
    0
    I'd say that http://www.centos.org/ is better for a CentOS user than the Ubuntu forums.

    Do you need Gnome / X? I'd assume that the Minecraft server needs no GUI and can be administered from command line (i.e. ssh). I would run at runlevel 3 rather than 5.

    You should be able to run both server instances as "services", so they would start the same way. You just need an unique service script (copy&change from the existing one, chkconfig --add, etc).
     
  12. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    See Ive never tinkered, or been ambitious enough yet to try and mess with the runlevels. Sure i would love to turn off the gui altogether, this server doesnt do much else and the only thing i really use the gui for is installing updates, i will have to research how to get the other server to run as a service, and if I can do that, then look into changing the run level on the server to 3 so gui doesnt load. I administer it mostly from a ssh session anyway so there would be no difference there.

    i agree on the centos.org comment, though im thinking of registering at both forums.
     
  13. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Looking at it again, been sitting idle all day, only the service instance of Minecraft running and no Gnome session active, it was up around 1.08 for a minute and has come back down to 0.40 or so. So weird.Just want to understand better WHY it is the way it is.
     
  14. Jodell88

    Jodell88 Diamond Member

    Joined:
    Jan 29, 2007
    Messages:
    9,437
    Likes Received:
    27
    Switch to a TTY and see if the load goes down. If it does, that means it most likely has to do with Xorg/graphics drivers.
     
  15. KillerBee

    KillerBee Golden Member

    Joined:
    Jul 2, 2010
    Messages:
    1,425
    Likes Received:
    3
    Have you run Sysstat?
     
  16. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Not familiar with the term TTY, though i guess it means like a text only mode.

    Nope. Not familiar with that either.
     
  17. Jodell88

    Jodell88 Diamond Member

    Joined:
    Jan 29, 2007
    Messages:
    9,437
    Likes Received:
    27
    Yes, ctrl alt and F1 - F6 would drop youto a console, referred to as a TTY. F7 and up would give you an Xorg server.
     
  18. KillerBee

    KillerBee Golden Member

    Joined:
    Jul 2, 2010
    Messages:
    1,425
    Likes Received:
    3
  19. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    Oh thats what that is, ive been using that, but I also thought the XORG server continued to run, that the cli it dropped you to was just like an instance, like a workspace.


    Thanks vtx! This is good stuff, i installed sysstat and will let it gather information so I can parse it later and look at what the system is doing over a period of time. Though without that modded version of Minecraft running in a Gnome session it seems to have gone below 0.15 load and is hovering between 0.05 and 0.15 for the most part. Maybe the mods just needed extra oomf or just arent coded quite as neatly as Minecraft itself.
     
    #19 Paperlantern, Jun 16, 2012
    Last edited: Jun 16, 2012
  20. serpretetsky

    serpretetsky Senior member

    Joined:
    Jan 7, 2012
    Messages:
    549
    Likes Received:
    0
    im curious about that as well. I dont think it actually stops xwindows or gnome since in any of those terminal sessions, if you try to type STARTX it warns you the x-windows server is already running.
     
  21. Paperlantern

    Paperlantern Platinum Member

    Joined:
    Apr 26, 2003
    Messages:
    2,188
    Likes Received:
    0
    As a little update to this, I dropped the runlevel of the server to 3 so it would not start X at all. It has improved the loads by a smidge. I also bumped the RAM to 2.25GB to full accommodate the two 1024MB chunked instances of Java that are running (The two separate minecraft servers). It now hovers between 0.15 and 0.5 load, usually keeping a 15 minute average of about 0.35. I'm not terribly worried about it anymore like i was. Though I'm still curious why it went from near NOTHING to a constant load at idle, albeit even a relatively inconsequential load, its still there.

    Still learning what makes Linux tick, it's still a lot of fun, regardless of the frustration.