1

I have a PostgreSQL-driven web service on two Debian servers, one a vhost running Linux kernel 3.2.0-2 with 2GB RAM and the other a dedicated server running Linux kernel 2.6.26-1 with 6 GB RAM. Although the application has a very small memory footprint, over the time the cache fills up to a certain amount at which processes gets swapped out increasing the system's load, finally run out of memory and eventually crash the kernel (same behavior on both systems, only different time periods). Before the crashes, free -m reports something simliar to:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1978       1583        394          0        147       1208
-/+ buffers/cache:        227       1751

If I reboot the system, it runs smoothly for some time (often weeks or months, but sometimes only days) until the cache got filled up again. After the reboot, free -m reports:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1978        452       1526          0          6        302
-/+ buffers/cache:        143       1835

It would be acceptable for me if the server slows down somewhat due to disk I/O if there are peaks due to excessive usage, but it is not acceptable that the system crashes then and when because disk cache memory is not reclaimed to processes if they temporarily need more memory.

So on one server I disabled swapping at all (since with swapping the load goes up to 300 - 400 (in sort of a chain reaction if processes are being killed) rendering the system unuseable even for a remote login to initiate a reboot. In addition I did set the kernel tuning parameter vm.swappiness=0 and vm.overcommit_memory=2 as recommended in the PostgreSQL High Performance Tuning guide to force the kernel to reclaim memory from the cache and avoid over-committing.

But this seems to not have the desired effect - the above output from free did not show a shrinking cache at all while programs again begin to crash with no memory available error messages (the output above from free was produced just seconds after the first no memory available messages appeared in the syslog after about two months of smooth operation).

Is there any chance to avoid this excessive caching to be able to use the memory for the applications rather than for disc I/O?

Increasing physical memory didn't help since on the 6GB-equipped system I get the same results sooner or later when more clients connect to the service (it's a hotspot auth server for free public use with an unknown number of users and I need to stick with the two servers for some time because of various reasons). On the vhost I first used 1GB, now 2GB - being the limit for my vhost - and all I got so far was an increase in cache storage, but not in memory for the processes. Is this the way its supposed to work on Linux? Just 12% of memory for processes? Or are the above suggestions from the HPT book incorrect?

Thanks in advance for any hint!

Hello Michael, yes, it seems so. I also thought that kernel cache memory should be freed if userland applications need more memory. But look at this output from free(1) I did occasionaly note when the system became unresponsive again, which is a sign that the situation leads to the OOM killer getting kicked in sooner or later:

             total       used       free     shared    buffers     cached
Mem:          1978        981        997          0        142        638
-/+ buffers/cache:        200       1777
Swap:            0          0          0

I can see the free Mem going down to zero while -/+ buffers/cache still shows some high value before the systems eventually crashes. I immediately cleared the cache by syncing and writing 3 to /proc/sys/vm/drop_caches resulting in kernel memory beeing freed:

             total       used       free     shared    buffers     cached
Mem:          1978        481       1496          0          1        283
-/+ buffers/cache:        196       1782
Swap:            0          0          0

So as a work-around I am clearing the cache manually from time to time. Unfortunately I cannot just update the OS on the production system, but have to change the systems before updates to ensure 99% availability. Testing kernels on our staging system is possible, but useless since the situation described will only appear after a certain time on our production system serving tens of thousands of login requests each day.

BTW, this is the output of free -m for today:

             total       used       free     shared    buffers     cached
Mem:          1978       1107        871          0        146        753
-/+ buffers/cache:        207       1771
Swap:            0          0          0

If I understand correctly, applications and kernel uses about 190 to 227 MB, while buffers and cache consume the rest of the memory. I am sure that the problem is not related to some application since nothing changes if I restart the applications, but it helps to clear the cache manually.

  • Wait, you're saying that the OOM killer kicks in because kernel cache memory is using too much? That doesn't make sense; cache should be evicted when the system comes under memory pressure. If that's really the case, you've run into a kernel bug, and probably should file a bug report with whoever packages the kernel you're running. Are you sure it's not something else that is gobbling up memory? – a CVn Aug 4 '14 at 18:28
  • Please edit your question to include additional information. Comments are second-class citizens on Stack Exchange, as well as that they can become almost unreadable in cases like this (or worse). – a CVn Oct 6 '14 at 9:20
  • @MichaelKjörling I have the same issue with an embedded system IO operations, unfortunately no one will help me due to use of non-free third party software (tainted kernel). So I can confirm that topic author problem does happen. Problem still exists in kernel 3.3.8. – RoughTomato Jun 24 '16 at 12:38
  • @RoughTomato Linux 3.3.8 is ancient (specifically, from the first week of June 2012; the commit is dated June 1, the tarball is dated June 4). That version is four years old. – a CVn Jun 24 '16 at 13:43
  • @MichaelKjörling charms of working in embedded I guess. I'm working with a project that's years old but still supported and unfortunately can't be moved to newer linux which causes all sorts of problems (and headaches for me :( ). Still, hoping to find out whether somebody noticed that and maybe fixed it somehow in newer versions and maybe made my work a bit easier. I don't believe this problem exists on new 4.6.2 x86_64 which I'm currently running at home. Although it might be platform specific in my case (mips). If I ever figure it out I'll make sure to share! maybe it'll help. Cheers :) ! – RoughTomato Jun 24 '16 at 14:45
-1

Kernel 3.15 add more control over page cache. Many application servers does not require heavy disk io. If that's the case, just add a cron job doing:

    echo 1 > /proc/sys/vm/drop_caches

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.