1

I have a Linux machine that reports almost all of its memory and swap are in use, yet I can’t identify any processes using a significant amount of memory on the machine. We are running Linux kernel 2.6.32-431.40.2.el6.x86_64, built unmodified from the RHEL srpm. This is a physical—not virtual—system; a Dell C6100 with two 6-core Intel X5650 CPUs. The system has been up for 12 days and here is the memory and swap usage.

Mem:  49416180k total, 48890424k used,   525756k free,    18300k buffers
Swap: 12582908k total, 12582908k used,        0k free,   727252k cached

I’ve run the following commands and received the following outputs, none of which seem to indicate that any process on the machine is consuming a significant amount of memory:

ps -eo pid,args,pmem --sort pmem 

Here is the full output of that ps command on Pastebin.

I also ran slabtop -o and cat /proc/meminfo and here is the full output of those commands on Pastebin as well.

I’m pretty stumped at this point. If anyone has any ideas as to what might be going on, or would like to see the output of any further commands, I'd be glad to hear it.

  • Is the memory column supposed to be all zeroes on ps? – Daniel B Feb 26 '15 at 22:35
  • It's the same output you get if you run 'top' and sort by memory, indicating that no process is taking more than 0.05% (or whatever the rounding threshold to one digit is) of system RAM. Looking at the VIRT and RES columns upholds this observation. – Juffo-Wup Feb 26 '15 at 23:05
1

The cause of this problem turned out to be a bug (LU-5726) in one of the Lustre kernel modules running on the system. This was causing the kernel to leak memory every time a file was deleted from the Lustre filesystem the machine was hosting. Upgrading to Lustre 2.5.4 (in which a fix had been merged) solved the problem.

Quoting from the bug report (which was someone else's, not mine) :

Description:

When deleting large numbers of files, memory usage on the MDS server grows significantly. Attempts to reclaim memory by dropping caches only results in some of the memory being freed. The buffer usage continues to grow until eventually the MDS server starts OOMing.

The rate at which the buffer usage grows seems to vary but looks like it might be based on the number of clients that are deleting files and the speed at which the files are deleted.

In response to the patch it was reported that:

We just applied the patch today to our production file system (Lustre 2.4.3) and are running some heavy purges right now. I collected some info about the memory usage. Prior to the patch, it seemed like the memory growth was dominated by the "Inactive(file)" in /proc/meminfo. I dropped the cache in the MDS server (echo 3 > /proc/sys/vm/drop_caches) and collected Inactive(file) usage every minute:

inactive(file): 1146656 kB ..... The number initially ramped up fast, but then leveled off a bit. Just to double check, I dropped the cache again:

Inactive(file): 401152 kB ....

We got the same behavior, and more importantly, we seem to be reclaiming the memory from Inactive(file). I also checked MemFree and Buffers before/after dropping caches:

(Before) MemTotal: 66053640 kB MemFree: 51291028 kB Buffers: 10685976 kB

(After) MemTotal: 66053640 kB MemFree: 63239432 kB Buffers: 198148 kB

Buffer usage dropped below 200 MB. Given the rate at which we are purging, that never would have happened prior to applying the patch.

It feel 90% confident this patch solved the problem. If we can continue purging at this rate over the couple of days without increased memory usage, then I think I will be 100% confident.

Additionally

I checked the MDS mem usage again this morning:

MemTotal: 66053640 kB MemFree: 5568288 kB Buffers:
55504980 kB Active: 22374284 kB Inactive: 33260116 kB

After I dropped caches:

MemTotal: 66053640 kB MemFree: 63146420 kB Buffers:
59788 kB Active: 57960 kB Inactive: 93452 kB

Looks like the patch is successful.

https://jira.hpdd.intel.com/browse/LU-5726

0

From what I can see here is that it's Linux' disk caching. Your memory is just fine, when a process needs it, it can just use it.

More information about this here: http://www.linuxatemyram.com/

The only thing I'm not too sure about, is the SWAP usage. I haven't seen that high usage on any of my machines.

erik@xenon ~] $ free -m total used free shared buffers cached Mem: 7883 7756 126 0 201 5702 -/+ buffers/cache: 1852 6030 Swap: 8068 11 8057

  • I am familiar with that site and with Linux's caching mechanism, but that doesn't appear to be the problem here. The Mem: and Swap: lines in the post indicate that only approximately 700 MB out of the 48 GB of RAM on the machine are being used as cache. – Juffo-Wup Feb 26 '15 at 21:25
  • After your edit, the output from your machine shows that almost 6 GB out of your 8 GB total RAM are being used as buffers/cache, meaning that those 6 GB are available for use if needed. On the machine I'm asking about, only ~700 MB out of its 48 GB are in use as buffers/cache, and all of the ~12 GB swap space is full. This is an abnormal situation given that nothing running on the machine appears to be using much RAM. – Juffo-Wup Feb 26 '15 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.