Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Before actually asking, just to be clear: yes, I know about disk cache, and no, it is not my case :) Sorry, for this preamble :)

I'm using CentOS 5. Every application in the system is swapping heavily, and the system is very slow. When I do free -m, here is what I got:

             total       used       free     shared    buffers     cached
Mem:          3952       3929         22          0          1         18
-/+ buffers/cache:       3909         42
Swap:        16383         46      16337

So, I actually have only 42 Mb to use! As far as I understand, -/+ buffers/cache actually doesn't count the disk cache, so I indeed only have 42 Mb, right? I thought, I might be wrong, so I tried to switch off the disk caching and it had no effect - the picture remained the same.

So, I decided to find out who is using all my RAM, and I used top for that. But, apparently, it reports that no process is using my RAM. The only process in my top is MySQL, but it is using 0.1% of RAM and 400Mb of swap. Same picture when I try to run other services or applications - all go in swap, top shows that MEM is not used (0.1% maximum for any process).

top - 15:09:00 up  2:09,  2 users,  load average: 0.02, 0.16, 0.11
Tasks: 112 total,   1 running, 111 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4046868k total,  4001368k used,    45500k free,      748k buffers
Swap: 16777208k total,    68840k used, 16708368k free,    16632k cached

 3214 ntp       15   0 23412 5044 3916 S  0.0  0.1   0:00.00  17m ntpd
 2319 root       5 -10 12648 4460 3184 S  0.0  0.1   0:00.00 8188 iscsid
 2168 root      RT   0 22120 3692 2848 S  0.0  0.1   0:00.00  17m multipathd
 5113 mysql     18   0  474m 2356  856 S  0.0  0.1   0:00.11 472m mysqld
 4106 root      34  19  251m 1944 1360 S  0.0  0.0   0:00.11 249m yum-updatesd
 4109 root      15   0 90152 1904 1772 S  0.0  0.0   0:00.18  86m sshd
 5175 root      15   0 90156 1896 1772 S  0.0  0.0   0:00.02  86m sshd

Restart doesn't help, and, by they way is very slow, which I wouldn't normally expect on this machine (4 cores, 4Gb RAM, RAID1).

So, with that - I'm pretty sure that this is not a disk cache, who is using the RAM, because normally it should have been reduced and let other processes to use RAM, rather then go to swap.

So, finally, the question is - if someone has any ideas how to find out what process is actually using the memory so heavily?

share|improve this question
Did you ever find the answer to this? – Hackeron Aug 23 '15 at 12:23
@Hackeron: OP accepted this answer. I know that answer doesn't address your question, though. I was able to reproduce your issue on one of my servers, and I'm currently researching if there is a way to troubleshoot it. – Deltik Aug 23 '15 at 12:49
@Deltik Ah, ok. Thank you :) - I have 2 servers here that leak all available memory in the space of around 12 hours, let me know if there is anything I can do to help diagnose this. I'm reachable as the nickname "hackeron" on IRC ( – Hackeron Aug 23 '15 at 13:54
@Hackeron: I wasn't able to find you as "hackeron" on I did create a chatroom for extended discussion here. – Deltik Aug 23 '15 at 14:52
up vote 64 down vote accepted

On Linux in the top process you can press < key to shift the output display sort left. By default it is sorted by the %CPU so if you press the key 4 times you will sort it by VIRT which is virtual memory size giving you your answer.

Another way to do this is:

ps -e -o pid,vsz,comm= | sort -n -k 2

should give you and output sorted by processes virtual size.

Here's the long version:

ps --everyone --format=pid,vsz,comm= | sort --numeric-sort --key=2
share|improve this answer
That gives me Warning: bad ps syntax, perhaps a bogus '-'? See on Ubuntu server 11.10. – Oliver Salzburg Mar 9 '12 at 14:36
@OliverSalzburg The issue is -o options. RHEL4 this works. RHEL5: ps -e -o pid,vsz,comm= | sort -n -k 2 works. I'll try 11.10 later tonight but if you find the right sort options before please let me know. ps -e -o pid,vsz,comm | sort -n -k 2 might work but I don't have a place to verify at the moment. – Karlson Mar 9 '12 at 14:40
I'm not really familiar with the -ef option. But this seems to produce reasonable output: sudo ps axo pid,vsz,comm=|sort -n -k 2 – Oliver Salzburg Mar 9 '12 at 14:43
@OliverSalzburg Sorry. Amended (thought I changed it already). It should be ps -e or ps -a – Karlson Mar 9 '12 at 14:54
Ty, I like the top suggestion of < I didn't know that was possible, fedora – SSH This Jun 6 '12 at 0:29

Just a side note on a server showing the same symptoms but still showing memory exhaustion. What ended up finding was a sysctl.conf from a box with 32 GB of RAM and setup for a DB with huge pages configured to 12000. This box only has 2 GB of RAM so it was assigning all free RAM to the huge pages (only 960 of them). Setting huge pages to 10, as none were used anyway, freed up all of the memory.

A quick check of /proc/meminfo to look for the HugePages_ settings can be a good start to troubleshooting at least one unexpected memory hog.

share|improve this answer
I recently had another server where this was the problem. If your organization has ex-Oracle employees in it, this setting may be your culprit. – fields Jul 15 '14 at 14:20

You can also use ps command to get more information about process.

ps aux | less
share|improve this answer
Out of curiosity, what is the correct way to escape this command? It shows END ocne i reach the last line, it does not kill the process when i Ctrl+C it. – KingsInnerSoul Nov 13 '15 at 16:09
@KingsInnerSoul press 'q' – enobayram Nov 16 '15 at 13:41

Show the processes memory in megabytes and the process path.

ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -n
share|improve this answer
Welcome to Super User. Can you expand your answer to explain what this code does and how it addresses the problem? Unexplained code is discouraged, because it doesn't teach the solution. Thanks. – fixer1234 Feb 9 at 22:03
I'm surprised this answer is downvoted and has a comment asking to explain it.. it's short enough that it should be clear what it does (pipes ps aux into awk and then sort), and in the context of the question, it shows which processes are using the most RAM. I think it's a fine answer. – John May 10 at 22:51

Make a script called with content:

ps -eo rss,pid,user,command | sort -rn | head -10 | awk '{ hr[1024**2]="GB"; hr[1024]="MB";
 for (x=1024**3; x>=1024; x/=1024) {
 if ($1>=x) { printf ("%-6.2f %s ", $1/x, hr[x]); break }
 } } { printf ("%-6s %-10s ", $2, $3) }
 { for ( x=4 ; x<=NF ; x++ ) { printf ("%s ",$x) } print ("\n") }
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .