The cause of this problem turned out to be a bug (LU-5726) in one of the Lustre kernel modules running on the system. This was causing the kernel to leak memory every time a file was deleted from the Lustre filesystem the machine was hosting. Upgrading to Lustre 2.5.4 (in which a fix had been merged) solved the problem.
Quoting from the bug report (which was someone else's, not mine) :
When deleting large numbers of files, memory usage on the
MDS server grows significantly. Attempts to reclaim memory by dropping
caches only results in some of the memory being freed. The buffer
usage continues to grow until eventually the MDS server starts OOMing.
The rate at which the buffer usage grows seems to vary but looks like
it might be based on the number of clients that are deleting files and
the speed at which the files are deleted.
In response to the patch it was reported that:
We just applied the patch today to our production file system (Lustre
2.4.3) and are running some heavy purges right now. I collected some info about the memory usage. Prior to the patch, it seemed like the
memory growth was dominated by the "Inactive(file)" in /proc/meminfo.
I dropped the cache in the MDS server (echo 3 >
/proc/sys/vm/drop_caches) and collected Inactive(file) usage every
inactive(file): 1146656 kB ..... The number initially ramped up
fast, but then leveled off a bit. Just to double check, I dropped the
Inactive(file): 401152 kB ....
We got the same behavior, and more importantly, we seem to be
reclaiming the memory from Inactive(file). I also checked MemFree and
Buffers before/after dropping caches:
(Before) MemTotal: 66053640 kB MemFree: 51291028 kB Buffers:
(After) MemTotal: 66053640 kB MemFree: 63239432 kB Buffers: 198148
Buffer usage dropped below 200 MB. Given the rate at which we are
purging, that never would have happened prior to applying the patch.
It feel 90% confident this patch solved the problem. If we can
continue purging at this rate over the couple of days without
increased memory usage, then I think I will be 100% confident.
I checked the MDS mem usage again this morning:
MemTotal: 66053640 kB MemFree: 5568288 kB Buffers:
55504980 kB Active: 22374284 kB Inactive: 33260116 kB
After I dropped caches:
MemTotal: 66053640 kB MemFree: 63146420 kB Buffers:
59788 kB Active: 57960 kB Inactive: 93452 kB
Looks like the patch is successful.