0

After a recent update to my ubuntu 20.04 server, my website started behaving badly.

Pre-update, it would respond in less than a second.

Post-update, it would respond in a few minutes.

After digging around, I discovered very high network usage by the NFS clients that are used to share the site between two application servers. The clients are stuck in iowait which is what's causing the web server to fail.

My monitoring graphs show that at the time of the update, network utilization tripled per client.

I've tried hard to understand what configs might be necessary to fix the excessive network usage, but to no avail.

Here is an example of my exports:

/volume/exports/html     *(rw,async,no_subtree_check,no_root_squash)

and my current fstab for the clients:

10.x.x.x:/volume/exports/html /var/www/html    nfs4    fsc,noatime,vers=4.2,nodev,nosuid,tcp,hard,intr,async      0   0

=====

nfsstat shows ~ 25% calls are access, 25% getattr, 25% sequence and 25% putfh.

Running tcpdump is not helpful, because the stream is approximately 35mb/s and I don't see any obvious pattern to the accesses.

4
  • What NFS operations did you see the clients performing over the network? Mar 4, 2023 at 23:58
  • Not sure how I can tell that. Is there some tool to view the stream of operations? Mar 5, 2023 at 0:09
  • Yes, you have network packet capture tools such as tcpdump and Wireshark, for one (both of which can dissect NFS packets, assuming you aren't using encryption via sec=krb5p). Linux clients also have the 'rpcdebug' tool that could enable diagnostic logs in 'dmesg'. Mar 5, 2023 at 0:10
  • Pick a site and don't ask the same question on more than one - unix.stackexchange.com/questions/738674/…
    – Bib
    Mar 5, 2023 at 10:50

1 Answer 1

1

Check if you are not on latest kernel that broke nfs kernel module: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325

2
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Mar 8, 2023 at 11:37
  • Yeah, that's it. I was suspicious when it happened RIGHT AFTER an upgrade. I've modified my network plan until after the patch is out. Mar 15, 2023 at 21:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .