I have a production system running centos 6.3 with 96 Gb RAM and swap of 2 Gb. The intention is that there is enough RAM on the box so that no processes should get swapped.

Over the last week or so the amount of swap space used has increased to 100%, as reported by /proc/meminfo SwapTotal and SwapFree.

However, the amount of swap used by processes as reported by top and by the script given here https://stackoverflow.com/questions/479953/how-to-find-out-which-processes-are-swapping-in-linux is only about 26 Mb.

The system doesn't appear to be swapping, so I'm not sure how seriously to take the /proc/meminfo SwapFree value.

Why is there this discrepency between what /proc/meminfo reports and the per-process Swap?

How can I find out which process(es) are responsible for /proc/meminfo thinking that all the swap is used (except by killing each process in turn and seeing when the SwapFree increases, which isn't really a good idea on a production system)?

Update: here is the current output of free:

             total       used       free     shared    buffers     cached
Mem:      99052152   98330684     721468          0     577400   82224692
-/+ buffers/cache:   15528592   83523560
Swap:      2097144     959580    1137564

and cat /proc/meminfo:

MemTotal:       99052152 kB
MemFree:          731560 kB
Buffers:          577400 kB
Cached:         82222116 kB
SwapCached:       174456 kB
Active:         69762456 kB
Inactive:       25778300 kB
Active(anon):   53783608 kB
Inactive(anon):  8973048 kB
Active(file):   15978848 kB
Inactive(file): 16805252 kB
Unevictable:       31424 kB
Mlocked:           31424 kB
SwapTotal:       2097144 kB
SwapFree:        1137556 kB
Dirty:              2076 kB
Writeback:             0 kB
AnonPages:      12598632 kB
Mapped:         43958780 kB
Shmem:          50010060 kB
Slab:            1390148 kB
SReclaimable:    1308692 kB
SUnreclaim:        81456 kB
KernelStack:        6400 kB
PageTables:       171412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    51623220 kB
Committed_AS:   68521168 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      478428 kB
VmallocChunk:   34307842136 kB
HardwareCorrupted:     0 kB
AnonHugePages:  11145216 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5604 kB
DirectMap2M:     2078720 kB
DirectMap1G:    98566144 kB

The total swap usage recorded in /proc/*/status VmSwap is 11832 Kb. (calculated by awk '/^VmSwap/ {t+=$2} END {print t" kB"}' /proc/*/status )

  • You could add to the post all the output of meminfo, as well as that of vmstat and even run Collectl. – harrymc Nov 28 '13 at 20:41
  • Swapping out the least used 2GB of application RAM over a week won't kill performance. Swapping it out leaves more room for cache so can be a win depending on your workload. What is the output of cat /proc/sys/vm/swappiness, it should be lower than the default of 60 to if you want it to swap less. Some suggest a value of 10. – Brian Nov 28 '13 at 23:10
  • swappiness is set to 60 at the moment, and I agree that reducing it might help, but my question is about how to find out what is using the swap space that isn't accounted for by the values in /proc/*/status VmSwap. – Chris Card Nov 29 '13 at 8:23
  • 1
    I have to ask: Why don't you just disable swap completely? – Gabe Nov 29 '13 at 11:08
  • 1
    See if this stackexchange answer doesn't throw some light on your numbers. – harrymc Nov 30 '13 at 17:47

One source of used swap that won't appear in each process is SwapCached pages.

The SwapCached: 174456 kB line are pages that are in both swap and ram.

This happens when pages are sent to swap then needed again but are never modified. The on disk copy is still valid so is kept. It can in effect be swapped to disk again latter without incurring a write to disk.

  • Even if that's the case, there's still nearly 1 Gb of swap usage unaccounted for. – Chris Card Dec 2 '13 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.