After a recent update to my ubuntu 20.04 server, my website started behaving badly.
Pre-update, it would respond in less than a second.
Post-update, it would respond in a few minutes.
After digging around, I discovered very high network usage by the NFS clients that are used to share the site between two application servers. The clients are stuck in iowait which is what's causing the web server to fail.
My monitoring graphs show that at the time of the update, network utilization tripled per client.
I've tried hard to understand what configs might be necessary to fix the excessive network usage, but to no avail.
Here is an example of my exports:
/volume/exports/html *(rw,async,no_subtree_check,no_root_squash)
and my current fstab for the clients:
10.x.x.x:/volume/exports/html /var/www/html nfs4 fsc,noatime,vers=4.2,nodev,nosuid,tcp,hard,intr,async 0 0
=====
nfsstat shows ~ 25% calls are access, 25% getattr, 25% sequence and 25% putfh.
Running tcpdump is not helpful, because the stream is approximately 35mb/s and I don't see any obvious pattern to the accesses.