Load average sky high but cpu usage low?

[Red Squirrel]

What would cause the load average to skyrocket on a server even though total cpu usage is very low? Ex:

top - 22:14:46 up 500 days,  9:45,  2 users,  load average: 7.04, 8.96, 8.37
Tasks: 395 total,   1 running, 394 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  1.4%sy,  0.0%ni, 98.3%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8031548k total,  7902432k used,   129116k free,   186680k buffers
Swap:  8175612k total,     5060k used,  8170552k free,  6841072k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                        
20923 p2puser   20   0  102m 2096  984 S  1.7  0.0   1:26.39 sshd                                           
11686 root      20   0  3932  236  192 S  1.3  0.0   6603:02 netresolv                                      
 2458 root      20   0     0    0    0 S  0.3  0.0 316:43.62 kondemand/0                                    
 2465 root      20   0     0    0    0 S  0.3  0.0  85:49.09 kondemand/7                                    
20924 p2puser   20   0 57492 2560 1652 S  0.3  0.0   0:21.98 sftp-server                                    
28677 root      20   0 15300 1496  944 R  0.3  0.0   0:00.21 top                                            
    1 root      20   0 19356  536  292 S  0.0  0.0   0:01.11 init                                           
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.06 kthreadd                                       
    3 root      RT   0     0    0    0 S  0.0  0.0   1:41.90 migration/0                                    
    4 root      20   0     0    0    0 S  0.0  0.0   6:06.15 ksoftirqd/0                                    
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0                                      
    6 root      RT   0     0    0    0 S  0.0  0.0   0:32.29 watchdog/0                                     
    7 root      RT   0     0    0    0 S  0.0  0.0   0:25.89 migration/1                                    
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1                                      
    9 root      20   0     0    0    0 S  0.0  0.0   3:11.19 ksoftirqd/1                                    
   10 root      RT   0     0    0    0 S  0.0  0.0   0:24.29 watchdog/1                                     
   11 root      RT   0     0    0    0 S  0.0  0.0   0:14.96 migration/2                                    
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/2                                      
   13 root      20   0     0    0    0 S  0.0  0.0   4:57.36 ksoftirqd/2                                    
   14 root      RT   0     0    0    0 S  0.0  0.0   0:19.71 watchdog/2                                     
   15 root      RT   0     0    0    0 S  0.0  0.0   1:05.47 migration/3                                    
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/3                                      
   17 root      20   0     0    0    0 S  0.0  0.0   3:49.03 ksoftirqd/3                                    
   18 root      RT   0     0    0    0 S  0.0  0.0   0:21.47 watchdog/3                                     
   19 root      RT   0     0    0    0 S  0.0  0.0   2:42.98 migration/4                                    
   20 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/4                                      
   21 root      20   0     0    0    0 S  0.0  0.0   3:38.70 ksoftirqd/4                                    
   22 root      RT   0     0    0    0 S  0.0  0.0   0:22.77 watchdog/4                                     
   23 root      RT   0     0    0    0 S  0.0  0.0   3:16.96 migration/5

This is my NFS server which I've always had performance issues with and I can't figure out why. Was in middle of watching a movie and it just died on me because it can't keep up with the stream. Trying to figure out if some kind of job is running but there's nothing that jumps out at me even though the load is crazy high.

Is there any way to troubleshoot this further to figure out what's hitting the system so hard?

I really need to redo this setup one of these days, it's always had crap performance. Going to try iSCSI next time as NFS has been nothing but trouble, it's too damn slow.


This is the output of nfsstat but that output is too ugly to really make much of it and I don't really know what any of it means, but it might help?

Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
258600560   0          0          0          0       

Server nfs v3:
null         getattr      setattr      lookup       access       readlink     
369       0% 13037471  0% 264994    0% 2745731   0% 6513496   0% 4190      0% 
read         write        create       mkdir        symlink      mknod        
2924090640 64% 1604801168 35% 105677    0% 11540     0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
43677     0% 8259      0% 71880     0% 0         0% 493       0% 339331    0% 
fsstat       fsinfo       pathconf     commit       
195677    0% 697       0% 339       0% 22489     0% 

Server nfs v4:
null         compound     
3         0% 1486640  99% 

Server nfs v4 operations:
op0-unused   op1-unused   op2-future   access       close        commit       
0         0% 0         0% 0         0% 57826     1% 8543      0% 334       0% 
create       delegpurge   delegreturn  getattr      getfh        link         
7         0% 0         0% 8518      0% 1016757  27% 324285    8% 0         0% 
lock         lockt        locku        lookup       lookup_root  nverify      
0         0% 0         0% 0         0% 315992    8% 0         0% 0         0% 
open         openattr     open_conf    open_dgrd    putfh        putpubfh     
8543      0% 0         0% 4         0% 0         0% 1481601  39% 0         0% 
putrootfh    read         readdir      readlink     remove       rename       
3         0% 464118   12% 334       0% 0         0% 2         0% 0         0% 
renew        restorefh    savefh       secinfo      setattr      setcltid     
4761      0% 0         0% 0         0% 0         0% 19        0% 2         0% 
setcltidconf verify       write        rellockowner bc_ctl       bind_conn    
2         0% 0         0% 68232     1% 0         0% 0         0% 0         0% 
exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     
0         0% 0         0% 0         0% 0         0% 0         0% 0         0% 
set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 
0         0% 0         0% 0         0% 0         0% 0         0%

 

[Fallen Kell]

You need to remember what "load average" is actually counting. It is simply the number of threads per second that the kernel has cycled through the various CPU cores averaged over time (1min, 5mins, 15mins).

A system can have a load average of 100 and still be responsive and underutilized if the processes that are running are things that might be hitting I/O waits (in which you would see the %wa state on the CPU go up), or could be simply waking from sleep to check if some condition has happened and going back to sleep which would show up as ideal. Having more CPU cores/threads also affects the meaning of the load average, since if you have 32 threads, a load average of 32 with the values of %us and %sy combined being close to 1 simply means you are filling all the threads, but on a system with just 4 CPUs/threads, that means you have 8 times the number processes cycling through your CPU than it can give full time for processing...

Looking at your numbers, I would say you need to show a few more things to find out what is happening. On your NFS client, run "nfsstat -rc". You want to look for the "retrans". This number will tell you how many times the client has requested some kind of operation that the server did not handle the first time, typically because the server was under too much load (i.e. typically it didn't have an available NFS server thread that was getting CPU time to handle the request).

The other piece of info you need is an statistics on your hard drives on the NFS server. Something like "iostat -xtc 2" will give some decent info and may show you that you are hitting a bottleneck on a particular disk drive.


That said, the nfsstat info you posted seems to show things are pretty good. Most of your clients (or at least most of your operations) seem to be using NFS v3, and the bulk of them are reads and writes (which is what you typically want to see). You can try tuning things a little better on your client side by changing your mount options for rsize and wsize to something like "rsize=262144,wsize=262144", but that is something that you will have to tweak to find out what works best with your hardware/network/usage (if it is dealing with large files such as movies, and you have a gigabit network with jumboframes, setting it to 262144 or higher should help out).

Another important tunable that you need to do for NFS server performance is to increase the read and write buffer sizes for the TCP stack. I won't go into specifics in this thread, but simply link to the first article that came up on google: Network Tuning

 

[Red Squirrel]

Thanks for the info, the issue kinda subsided but in Kodi I kept getting a circle with a 0 in it and then it would say that the source can't keep up, but I wonder if it's just that particular video format it did not like, since I ended up just watching the movie on my desktop machine (through NFS still) and then it worked fine... so issue may have been more with Kodi, maybe that was somehow causing the load to go high too.

Is there a way to setup NFS tunable values server side? I don't like the idea of having to do it on every single client as if they don't all match I imagine I could get issues like corruption. Right now everything is all default. I also want to experiment with setting the shares to async mode but not sure if that's wise.

 

[Fallen Kell]

Sorry, most all NFS tuning is client side. If you are using automount tables (through NIS, NIS+, LDAP, sssd, etc...) you can edit just those for updating all your clients, but from the sounds of things, I don't suspect this to be the case. The only mount options that you risk corruption issues with are sync/async and some of the client side caching with the nolock option enabled. The risk only occurs if you have multiple clients writing to the same file at the same time on the caching (typically seen when using NFS as the backstore for a database), and in the case of the async option, when you don't have battery backup on your server and you lose power after the server automatically sent the ACK response to the client after the client sent a write and before the server actually committed the write to disk (see the man pages for more details).

Also, looking more closely at your top output, it looks like your server is just starting to hit memory starvation. I don't think it is there yet, but it is close. However, that could be normal operations if you are using something like ZFS (which will use all available RAM for read/write cache). To really know what is going on there you would have needed to show the processes sorted by "VIRT", "RES", or "%MEM" (you can use the "<" and ">" keys in top to change the column it uses to sort the output).

 

[Fallen Kell]

One more item is to take a look at the NFS threads on your NFS server. By default, I believe it only creates 8 threads. Depending on your hardware, this number may be much lower than it can really handle. Each open file over NFS will take up a NFS thread on the server. A modern x86_64 CPU can easily handle multiple NFS threads per CPU thread/core due to the speed of the operations (i.e. I/O from disk is extremely slow compared to the CPU, and it will put the threads in I/O wait state while they are waiting for the disk to read/write, and the CPU can be cycling through and handling multiple other NFS threads because of this).

To figure out if your system needs more NFS threads, "cat /proc/net/rpc/nfsd" and look for the line that begins with "th" (for thread). This line will end with a group of 10 numbers. These ten numbers are a histogram of your NFS threads broken down into 10 percentile groups (to make this easy, if you have 10 NFS threads, each of those 10 numbers would equate to 1 thread, if you have 100 NFS threads, each of the numbers would represent the average of 10 NFS threads). Since they work in order (i.e. the 1st 10%, 2nd 10%, 3rd 10%, ... , 9th 10%, and 10th 10% of NFS threads), the important ones to look at when performing tuning is to look at the last 2-3 on the line. What you are shooting for is for the last 2-3 numbers in the "th" line to be at or very close to "0.000". If the last number is above "0", it means that all the NFS thread on your system are busy. This means you should increase the NFS server threads in /etc/sysconfig/nfs (edit the RPCNFSDCOUNT value which is the threads). For reference, at work we have systems that we have increased that value to over 4096 (grant it these are servers with 128 CPU threads, but that is roughly 32 NFS threads per CPU thread and they easily handle that value).

 

[Red Squirrel]

Not sure what any of those things you mentioned are (NIS etc) but I'll have to look into it. Have looked into LDAP before but looks quite involved to setup as you need to go through all the steps on each client as well and it's far from turn key. I eventually want to write a front end that sits on top of a distro that manages all this stuff and is web based though so I could use it to automate a lot of the stuff that's complicated and/or tedius to setup. I eventually want to look into what it takes to essentially roll my own distro, it would just be CentOS or other distro but with my own customizations to it including my custom app.

As for ram, that server only has 8 gigs of ram so I could bump that up next time there is an unexpected shutdown (this is not a server I shut down for any reason). Ram is way cheaper than it was when this box was originally built and I believe the motherboard supports up to 32. I could probably put 16 at least. I also have a quad gig nic waiting to be installed in it when the opportunity arises. Not that the network is my bottleneck... but hey it can't hurt.


nfsd output looks ok as well.

[root@isengard ~]# cat /proc/net/rpc/nfsd
rc 0 1614674252 2956979868
fh 601 0 0 0 0
io 1707979472 3378205495
th 16 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 2931464589 0 0 0 0 0 0 0 0 0 1175264
net 276964073 0 276537088 53962
rpc 276630635 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 378 13100422 265115 2750959 6540388 4190 2932835860 1613989961 105733 11542 0 0 43677 8259 71931 0 493 340044 195889 713 347 22489
proc4 2 3 1486640
proc4ops 59 0 0 0 57826 8543 334 7 0 8518 1016757 324285 0 0 0 0 315992 0 0 8543 0 4 0 1481601 0 3 464118 334 0 2 0 4761 0 0 0 19 2 2 0 68232 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Though I should probably recheck that during backup jobs or when I'm running into high load again.

Actually is there a good resource to find out of various things like this to look for, not just NFS, but like in general? These are all things I could setup alarm points for in my monitoring system. Been meaning to update that app, I want to add graphing capabilities in it so I can have a better overview of various performance parameters on all my systems. Kinda been neglecting when it comes to that stuff.

 

[Red Squirrel]

Here's some more output. Though right now everything seems fine... it's really strange as sometimes there's like a blocking that goes on and VMs crash etc. But I'm pushing it as hard as I can (running backup jobs etc) and it does not seem to be acting up.

25/11/19 08:16:45 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.05    0.00    4.87   18.90    0.00   72.19

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdv               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00   39.00   23.50 38552.00    18.00   617.12     0.83   13.95   11.97   17.23   8.14  50.85
sdf               0.50     0.00   47.50   22.50 46272.00    14.00   661.23     1.22   17.37   17.01   18.13   8.53  59.70
sdg               0.00     0.00   52.50   21.50 50216.00     6.00   678.68     0.83   11.25    9.57   15.35   6.62  49.00
sdh               0.00     0.00   36.50   21.00 33048.00     6.00   574.85     0.73   12.53   11.81   13.79   8.77  50.45
sdc               0.00     1.00   38.50   32.50 38176.00   106.00   539.18     1.49   20.65   22.23   18.78   8.84  62.75
sdb               0.00     2.00   41.50   35.00 40200.00   134.00   527.24     1.17   15.27   17.35   12.80   7.45  57.00
sdi             268.00  4844.50   35.50  174.00  2452.00 41792.00   211.19    15.90   75.42   46.37   81.35   1.13  23.75
sda               0.00     1.00   48.50   32.50 47120.00   106.00   583.04     1.68   20.86   19.03   23.58   7.95  64.40
sdd               0.00     2.00   45.00   35.00 43244.00   134.00   542.23     2.23   27.66   27.46   27.91   9.15  73.20
sdk             236.50  4749.00   32.50  297.00  2168.00 41816.00   133.49    10.56   31.85   33.28   31.69   0.64  21.15
sdo             250.00  4751.00   31.50  294.00  2268.00 41816.00   135.43     8.18   24.93   35.16   23.83   0.59  19.25
sdp             228.00  4751.50   35.50  298.00  2128.00 41820.00   131.78     8.52   25.44   36.37   24.13   0.55  18.25
sdj             238.00  4762.00   34.50  283.00  2200.00 41812.00   138.62    11.66   36.52   38.58   36.27   0.69  21.75
sdn             244.00  4746.00   35.50  290.00  2264.00 41752.00   135.23    12.51   38.25   43.23   37.64   0.62  20.05
sdm             215.50  4750.00   37.50  297.50  2040.00 41812.00   130.90     7.97   23.63   46.49   20.74   0.56  18.60
sdl             238.50  4767.00   34.50  269.00  2200.00 41780.00   144.91    12.40   40.59   45.35   39.98   0.74  22.45
sds             228.00  4880.00   29.00   70.50  2080.00 27152.00   293.79    25.05  105.52   72.53  119.09   2.42  24.10
sdr               0.00    61.50    0.00  407.00     0.00  2968.50     7.29     1.00    2.45    0.00    2.45   1.01  41.05
sdq               0.00    72.00    0.00  337.00     0.00  2496.50     7.41     1.03    3.07    0.00    3.07   0.83  27.95
sdu               0.00    57.50    1.00  411.00     8.00  2968.50     7.22     1.08    2.62    3.00    2.61   0.92  37.90
sdt               0.00    64.50    0.00  345.00     0.00  2496.50     7.24     1.20    3.47    0.00    3.47   0.88  30.30
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md3               0.00     0.00  349.00   50.50 336060.00   252.00   841.83     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00 41798.00     0.00 334384.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    1.00  780.00     8.00  5464.00     7.01     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

^C
[root@isengard ~]# 
[root@isengard ~]# 
[root@isengard ~]# 
[root@isengard ~]# cat /proc/net/rpc/nfsd
rc 0 1614752991 2957284146
fh 601 0 0 0 0
io 3207767867 492723704
th 16 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 2931760915 0 0 0 0 0 0 0 0 0 1175507
net 277347107 0 276920107 53965
rpc 277013657 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 378 13103180 265241 2751409 6541821 5796 2933132427 1614068522 105771 11551 0 0 43677 8259 71932 0 493 341223 195932 713 347 22499
proc4 2 3 1486640
proc4ops 59 0 0 0 57826 8543 334 7 0 8518 1016757 324285 0 0 0 0 315992 0 0 8543 0 4 0 1481601 0 3 464118 334 0 2 0 4761 0 0 0 19 2 2 0 68232 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[root@isengard ~]# 
[root@isengard ~]# 
[root@isengard ~]# top

top - 20:17:35 up 503 days,  7:48,  3 users,  load average: 11.52, 6.70, 5.93
Tasks: 397 total,   3 running, 394 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  1.6%sy,  0.0%ni, 91.4%id,  5.9%wa,  0.0%hi,  0.4%si,  0.0%st
Mem:   8031548k total,  7907432k used,   124116k free,   132996k buffers
Swap:  8175612k total,     5060k used,  8170552k free,  6980344k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                           
12504 root      20   0  143m  26m  784 R 57.2  0.3   0:14.64 rsync                                                                                                             
 1854 root      20   0     0    0    0 S 49.3  0.0 595:07.67 md1_raid5                                                                                                         
 3480 root      20   0     0    0    0 D 25.7  0.0  98:48.14 kjournald                                                                                                         
27362 root      20   0     0    0    0 D  7.9  0.0   4163:20 nfsd                                                                                                              
  125 root      20   0     0    0    0 S  5.9  0.0   2803:33 kswapd0                                                                                                           
27365 root      20   0     0    0    0 D  5.9  0.0   4189:29 nfsd                                                                                                              
27368 root      20   0     0    0    0 D  5.9  0.0   4083:34 nfsd                                                                                                              
11686 root      20   0  3932  236  192 S  2.0  0.0   6658:04 netresolv                                                                                                         
11694 root      20   0  3932  212  192 S  2.0  0.0   6658:25 netresolv                                                                                                         
27353 root      20   0     0    0    0 D  2.0  0.0   4129:05 nfsd                                                                                                              
27354 root      20   0     0    0    0 D  2.0  0.0   4151:59 nfsd                                                                                                              
27355 root      20   0     0    0    0 D  2.0  0.0   4211:51 nfsd                                                                                                              
27356 root      20   0     0    0    0 D  2.0  0.0   4227:29 nfsd                                                                                                              
27357 root      20   0     0    0    0 D  2.0  0.0   4129:18 nfsd                                                                                                              
27358 root      20   0     0    0    0 D  2.0  0.0   4180:40 nfsd                                                                                                              
27359 root      20   0     0    0    0 D  2.0  0.0   4107:55 nfsd                                                                                                              
27360 root      20   0     0    0    0 D  2.0  0.0   4158:21 nfsd                                                                                                              
27361 root      20   0     0    0    0 D  2.0  0.0   4082:08 nfsd                                                                                                              
27363 root      20   0     0    0    0 D  2.0  0.0   4069:10 nfsd                                                                                                              
27366 root      20   0     0    0    0 D  2.0  0.0   4030:22 nfsd                                                                                                              
    1 root      20   0 19356  624  380 S  0.0  0.0   0:01.11 init                                                                                                              
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.07 kthreadd                                                                                                          
    3 root      RT   0     0    0    0 S  0.0  0.0   1:41.96 migration/0                                                                                                       
    4 root      20   0     0    0    0 S  0.0  0.0   6:07.36 ksoftirqd/0                                                                                                       
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0                                                                                                         
    6 root      RT   0     0    0    0 S  0.0  0.0   0:32.46 watchdog/0                                                                                                        
    7 root      RT   0     0    0    0 S  0.0  0.0   0:25.95 migration/1                                                                                                       
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/1                                                                                                         
    9 root      20   0     0    0    0 S  0.0  0.0   3:12.20 ksoftirqd/1                                                                                                       
   10 root      RT   0     0    0    0 S  0.0  0.0   0:24.42 watchdog/1                                                                                                        
   11 root      RT   0     0    0    0 S  0.0  0.0   0:14.99 migration/2                                                                                                       
[root@isengard ~]#

Most of my servers are running CentOS 6.x though and they're at 8 now, so I do want to look at slowly upgrading and migrating stuff over as my issues could potentially be fixed simply by upgrading everything, and just the act of having fresh new systems/VMs. But I also want to move away from ESXi and look at Proxmox as it's more open, so in that process I will upgrade all the VMs as I will need to rebuild them anyway. Then eventually do the storage server too.

Basically I want to build my own version of CentOS 8.x with all optimizations/changes etc then deploy that throughout so all the installs are more consistent. Big enough project though.

 

[Fallen Kell]

Yeah, everything is looking pretty good according to those outputs. The svctm in the iostat is looking good with all disks under 10 (really only starts to be a concern as it jumps into the 50's and above, which it will typically start growing exponentially up to a couple hundred to a couple thousand as things grind to a halt).

As for NIS/NIS+/LDAP, I would suggest looking at IdM (Identity Management) which is Red Hat's name for FreeIPA. About as easy to setup as possible for central management of all your systems for your accounts/passwords, NFS shares, Sudo rules, SELinux rules, etc..., all easy to configure with a webui (as well as command line for scripting).