I have a PostgreSQL-driven web service on two Debian servers, one a vhost running Linux kernel 3.2.0-2 with 2GB RAM and the other a dedicated server running Linux kernel 2.6.26-1 with 6 GB RAM. Although the application has a very small memory footprint, over the time the cache fills up to a certain amount at which processes gets swapped out increasing the system's load, finally run out of memory and eventually crash the kernel (same behavior on both systems, only different time periods). Before the crashes,
free -m reports something simliar to:
$ free -m total used free shared buffers cached Mem: 1978 1583 394 0 147 1208 -/+ buffers/cache: 227 1751
If I reboot the system, it runs smoothly for some time (often weeks or months, but sometimes only days) until the cache got filled up again. After the reboot,
free -m reports:
$ free -m total used free shared buffers cached Mem: 1978 452 1526 0 6 302 -/+ buffers/cache: 143 1835
It would be acceptable for me if the server slows down somewhat due to disk I/O if there are peaks due to excessive usage, but it is not acceptable that the system crashes then and when because disk cache memory is not reclaimed to processes if they temporarily need more memory.
So on one server I disabled swapping at all (since with swapping the load goes up to 300 - 400 (in sort of a chain reaction if processes are being killed) rendering the system unuseable even for a remote login to initiate a reboot. In addition I did set the kernel tuning parameter
vm.overcommit_memory=2 as recommended in the PostgreSQL High Performance Tuning guide to force the kernel to reclaim memory from the cache and avoid over-committing.
But this seems to not have the desired effect - the above output from
free did not show a shrinking cache at all while programs again begin to crash with
no memory available error messages (the output above from
free was produced just seconds after the first
no memory available messages appeared in the syslog after about two months of smooth operation).
Is there any chance to avoid this excessive caching to be able to use the memory for the applications rather than for disc I/O?
Increasing physical memory didn't help since on the 6GB-equipped system I get the same results sooner or later when more clients connect to the service (it's a hotspot auth server for free public use with an unknown number of users and I need to stick with the two servers for some time because of various reasons). On the vhost I first used 1GB, now 2GB - being the limit for my vhost - and all I got so far was an increase in cache storage, but not in memory for the processes. Is this the way its supposed to work on Linux? Just 12% of memory for processes? Or are the above suggestions from the HPT book incorrect?
Thanks in advance for any hint!
Hello Michael, yes, it seems so. I also thought that kernel cache memory should be freed if userland applications need more memory. But look at this output from
free(1) I did occasionaly note when the system became unresponsive again, which is a sign that the situation leads to the OOM killer getting kicked in sooner or later:
total used free shared buffers cached Mem: 1978 981 997 0 142 638 -/+ buffers/cache: 200 1777 Swap: 0 0 0
I can see the
free Mem going down to zero while
-/+ buffers/cache still shows some high value before the systems eventually crashes. I immediately cleared the cache by
syncing and writing 3 to
/proc/sys/vm/drop_caches resulting in kernel memory beeing freed:
total used free shared buffers cached Mem: 1978 481 1496 0 1 283 -/+ buffers/cache: 196 1782 Swap: 0 0 0
So as a work-around I am clearing the cache manually from time to time. Unfortunately I cannot just update the OS on the production system, but have to change the systems before updates to ensure 99% availability. Testing kernels on our staging system is possible, but useless since the situation described will only appear after a certain time on our production system serving tens of thousands of login requests each day.
BTW, this is the output of
free -m for today:
total used free shared buffers cached Mem: 1978 1107 871 0 146 753 -/+ buffers/cache: 207 1771 Swap: 0 0 0
If I understand correctly, applications and kernel uses about 190 to 227 MB, while buffers and cache consume the rest of the memory. I am sure that the problem is not related to some application since nothing changes if I restart the applications, but it helps to clear the cache manually.