0

I have two Debian servers connected to a shared NFS server.

Mounts:

nfs.internal.com:/volume1/SHARE on /share type nfs (rw,nosuid,relatime,sync,vers=3,rsize=131072,wsize=131072,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,soft,noac,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.100,mountvers=3,mountport=892,mountproto=tcp,fsc,lookupcache=none,local_lock=none,addr=172.16.0.100)

Sometimes, the load on the server will increase. There seems to be a breakpoint where the load causes NFS operations to slow down enough that a runaway load condition occurs and we eventually reach a limit on processes configured for the service run on this server (1500).

When the load increases on one server, the other is not affected, so I don't believe networking or the NFS server to be the culprit.

Under normal conditions, load ~10

ops/s       rpc bklog
20637.000           0.000

getattr:           ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
               10453.000        2334.176           0.223        0 (0.0%)           0.339           0.467
access:            ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                4893.000        1139.078           0.233        0 (0.0%)           0.338           0.469

Under "medium" load of ~100:

ops/s       rpc bklog
13374.200           0.000

read:              ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                  10.600        1202.169         113.412        0 (0.0%)           0.623          26.868
write:             ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)
                   7.600           5.462           0.719        0 (0.0%)           0.553           7.395

I don't have it saved, but at the highest load levels of 1500, the avg exe time will reach 4-5 seconds.

From what I understand, an exe value that is much higher than RTT means that the requests are queuing on the local server rather than waiting on the NFS server.

It sounds like the number of simultaneous requests to the NFS server is specified by sunrpc.tcp_slot_table_entries. In recent kernel versions this value is dynamically increased up tosunrpc.tcp_max_slot_table_entries, however I have never seen it go above the default of two.

Could this be the cause of the performance issues, and what would prevent this value from scaling like it should be?

4
  • Try out increasing the slot tables manually via sudo sysctl -w sunrpc.tcp_slot_table_entries=64. There's a decent writeup here explaining what exe time is and why it can balloon when requests start queueing on the client: mail-archive.com/linux-nfs@vger.kernel.org/msg00828.html
    – Cpt.Whale
    Sep 21 at 14:11
  • Try the mount with the async option. With sync the server only confirms write operations after they are actually written to the disk, which slows down confirmations. Your rsize and wsize may perhaps be too high, so try 65536.
    – harrymc
    Sep 21 at 15:24
  • @harrymc async is not an option due to the data involved. Is there evidence that lowering the values for rmem and wmem (negotiated between the devices by default) will help? From what I've read this value should be autoscaling. I will be updating this minimum value by hand but was curious why this value wasn't autoscaling. Sep 27 at 13:31
  • Unfortunate that you can't use async. You may find more advice in Identifying NFS performance bottlenecks.
    – harrymc
    Sep 29 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.