I try to tweak my development system to maximal reliability. I disabled swap, because for GUI usage it mostly renders the machine unresponsive in such a way not useable anymore. Nevertheless, if agressive appications eat up the memory, some mechanisms seem to kick in that making the most out of it on cost of speed. There is no harddrive swap operation, but the system is getting unresponsive likewise. So I want to let the OOM killer kick in before the system make any special efforts on memory gain. Is it possible to configure the OOM killer to act if there is less than 100 MB free physical memory for example?
9
-
2I think the real issue here is, there's not enough ram to start with. You won't use swap unless there's no ram. By turning off swap... you run out of ram and have no where to page it to. Which causes ugly things to happen. Your system seems to be set up badly, and no amount of tweaking will fix that.– Journeyman Geek ♦Mar 29, 2012 at 8:51
-
14I don't agree. Development and 'power use' often involves experimental usage. For example, when using a command line image processing tool, there are no specs how much memory it's operation take in relation to the image size. So I just give it a run. And I don't expect it to render my whole machine useless. For a single experiment, I could use ulimit to keep it secured, but for whole system operation with sometimes plenty of operations, the containment of one process is not so usefull but a 'life insurance' for the whole machine definetly is.– dronusMar 29, 2012 at 11:14
-
10@JourneymanGeek, you are off in left field. Disks are slow compared to ram, period, hence heavy swapping always grinds the system to a halt. Of course he is out of memory because he tried running a program that uses a lot of memory. The question is what to do when out of memory? Kill the hog, or slow down due to having no memory left for the disk cache.– psusiMar 29, 2012 at 17:38
-
3@TomWijsman, Disk IO is many orders of magnitude slower than memory IO, so using disk swap has always meant a huge slow down. Sometimes ( especially in the old days where ram was expensive and so most people didn't have much ) that's preferable to not being able to do what you were trying at all. These days the disk is SO much slower than ram, and ram is cheap enough that most people have plenty, so on the rare occasion where they accidentally run something that uses more ram than they have, it is often better to give up than take 1000 times as long to do it.– psusiMar 29, 2012 at 18:33
-
3I would also like OOM killer to trigger a little earlier. Sometimes it is obvious that the I have overloaded my system, but it takes minutes of swapping and mouse/keyboard jitter before the killer acts.– joeytwiddleJan 18, 2013 at 6:23
|
Show 4 more comments
5 Answers
I also struggled with that issue. I just want my system to stay responsive, no matter what, and I prefer losing processes to waiting a few minutes. There seems to be no way to achieve this using the kernel oom killer.
However, in the user space, we can do whatever we want. So I wrote the Early OOM Daemon ( https://github.com/rfjakob/earlyoom ) that will kill the largest process (by RSS) once the available RAM goes below 10%.
Without earlyoom, it has been easy to lock up my machine (8GB RAM) by starting http://www.unrealengine.com/html5/ a few times. Now, the guilty browser tabs get killed before things get out of hand.
-
3Thanks for scratching this itch! Loving earlyoom so far. Feb 18, 2016 at 13:15
-
1Just figured out Android does the same for a long time. I am not sure if it is using custom code like yours for that.– dronusMay 23, 2016 at 12:08
-
1I am testing
earlyoomnow, it does well in a first trigger test. I just wonder why this can't be implemented by kernel configuration or system tools.– dronusMay 23, 2016 at 12:23
The default policy of the kernel is to allow applications to keep allocating virtual memory as long as there is free physical memory. The physical memory isn't actually used until the applications touch the virtual memory they allocated, so an application can allocate much more memory than the system has, then start touching it later, causing the kernel to run out of memory, and trigger the out of memory (OOM) killer. Before the hogging process is killed though, it has caused the disk cache to be emptied, which makes the system slow to respond for a while until the cache refills.
You can change the default policy to disallow memory overcommit by writing a value of 2 to /proc/sys/vm/overcommit_memory. The default value of /proc/sys/vm/overcommit_ratio is 50, so the kernel will not allow applications to allocate more than 50% of ram+swap. If you have no swap, then the kernel will not allow applications to allocate more than 50% of your ram, leaving the other 50% free for the cache. That may be a bit excessive, so you may want to increase this value to say, 85% or so, so applications can allocate up to 85% of your ram, leaving 15% for the cache.
-
1Changing these values from there defaults without theoretical background is not going to reach in a more reliable system, you can only justify that change with proper statistics. Just because you can change it doesn't mean you should. If you constantly in low memory conditions that means that you are using more memory than you have and should buy more memory, it doesn't mean you should fiddle with your settings and kill random applications. Interrupting with your daily working or introducing corruption, that's really not the way to go... Mar 29, 2012 at 18:56
-
3@TomWijsman, the question makes it clear that he isn't constantly in low memory conditions; he just sometimes runs a command that takes an unexpectedly large amount of memory. Buying more memory is not the the only solution when you run out. Other potential solutions include finding better ways to make use of the memory you have, or just not doing whatever needs that much memory. The question makes it clear that the latter is more acceptable than going out and buying more ram.– psusiMar 29, 2012 at 19:02
-
Which line in the question makes this clear? I see the opposite given in
I disabled swap, because for GUI usage it mostly renders the machine unresponsive in such a way not useable anymore.. He mentioned GUI, while you are assuming he runs a command. Buying more memory is the first solution, using less memory yourself is the second solution, making your system unstable by fiddling with the stable defaults is the last solution. The question doesn't have to be answered literally, so I don't see what's your problem that you have to bother both of us in the comments. Rant doesn't help... Mar 29, 2012 at 19:06 -
5Hey, this answer sounded quite cool. Unfortunately, the 'commit' refers to virtual memory demand it seems, which is quite bad estimated by application programmers. For example with my (no swap) desktop running, there is about 400 of 2000mb physical memory used, but 1600mb 'commit'ted as
/proc/meminfo'sCommitted_ASstates. With some applications running, this value easily exceeds the physical memory so it's hard to set a feasable limit by this.– dronusMar 29, 2012 at 22:17 -
4Save your work before trying this! :P I had immediate failures from everything (bash, window manager etc).– jozxyqkMar 12, 2015 at 5:18
For me setting vm.admin_reserve_kbytes=262144 does exactly this thing. OOM killer intervents before system goes completely unresponsive.
-
2I like idea, but does it means you have 256MiB of physical memory never used? Jun 14, 2018 at 13:05
-
1256MiB will be used for caches. Caches are really important, it's not about just running faster, system wouldn't work at all if there's no enough memory for caches. Code of every running program can be unloaded from memory because it's mmaped and can be read back from disk. Without caches every task switch will require disk read and system will become completely unresponsive. Jun 15, 2018 at 14:41
-
@MichaelVigovsky Do you have any proof of caching statement? It isn't mentioned at all in kernel docs:
The amount of free memory in the system that should be reserved for users with the capability cap_sys_admin.Feb 22, 2020 at 15:58
The other answers have good automatic solutions, but I find it can be helpful to also enable the SysRq key for when things get out of hand. With the SysRq key, you'd be manually messaging the kernel, and you can do things like a safe reboot (with SysRQ + REISUB) even if userspace has completely frozen.
To allow the kernel to listen to requests, set kernel.sysrq = 1, or enable just the functions you're likely to use with a bitmask (documented here). For example kernel.sysrq = 244 will enable all the combos needed for the safe reboot above as well as manual invocation of the OOM killer with SysRq + F.
Reliability isn't reached by low memory conditions and an OOM killer.
It is wrong to organize a party in a closet and place "cleaning out my closet" on your small playlist.
Is it possible to make the OOM killer intervent earlier?
Doing this will have unintended side results, because you have no control over what is killed.
I try to tweak my development system to maximal reliability.
Maximal reliability involves testing your system and improving your system based on these tests.
Just tweaking random things won't get you anywhere...
I disabled swap, because for GUI usage it mostly renders the machine unresponsive in such a way not useable anymore. Nevertheless, if agressive appications eat up the memory, some mechanisms seem to kick in that making the most out of it on cost of speed.
Due to low memory conditions, disabling the swap won't improve the behavior, it does the opposite.
To increase reliability in this situation, add more memory such that your system is more responsive and there are no random processes being killed without the user's intention. You shouldn't resort on low memory conditions and a mechanism like this, especially not in a development environment...
There is no harddrive swap operation, but the system is getting unresponsive likewise.
Low memory conditions indeed result in unresponiveness, whether you have a swap or not.
So I want to let the OOM killer kick in before the system make any special efforts on memory gain.
Special efforts that will do more harm than good, as I explained above. Instead, you could kill processes you don't need yourself, but I guess you can't do that so the OOM will kill processes that you need.
Is it possible to configure the OOM killer to act if there is less than 100 MB free physical memory for example?
Might be, but you get a higher return on investment if you just buy some extra memory which doesn't really cost much these days. Consider that you're going to hit yourself in the foot on the long run if you continue to work on low memory conditions. OOM is like a bailiff, it doesn't assist you, it assists the OS...
answered Mar 29, 2012 at 14:01
-
8Of course disabling swap improves the behavior because instead of thrashing the disk, the OOM kicks in and kills the memory hog. Running out of ram isn't the problem ( and adding more just means you have to try harder to run out ). The problem is what to do when you DO run out. You want the OOM to kill the hog, and thus relieve the low memory condition.– psusiMar 29, 2012 at 17:43
-
9Because killing an application that is trying to use more memory than you have is preferable to bringing the entire system to its knees. In a perfect world you would have unlimited memory and never run out, but in reality, sometimes you run out by accident and would rather be told "not enough memory" than have the system grind to a halt.– psusiMar 29, 2012 at 18:25
-
6Buying some extra memory might solve some problems, depending on the amount bought. But it doesn't change the fact that there may be inexpected usages by orders of magnitude. So I want the application to fail, but NOT the system under those conditions. Some examples: Process a folder full of compressed images, most of them "normal" size, but some of them really large. A small mistake could make a dead loop with memory runaway eating 1GB/s. Accidentally open a video file in a text editor. Usually this ends with symptoms like jerky mouse and almost dead UI until the OOM kicks in.– dronusMar 29, 2012 at 22:03
-
7@TomWijsman there are also almost-dead loops as there are algorithms that behave linear in mean case but exponential in worst case, depending on input data. And I cannot send a kill signal if the mouse is jerky and clicks as well as keyboard input shows a one minute latency. I usually change to a text mode terminal then and wait minutes for the login to proceed just to issue a
killblindly typed.– dronusMar 29, 2012 at 22:23 -
8I have no problems with killing applications that would run dead either. Consider a system with 2GB physical + 2GB swap. An application that quickly runs out the physical memory can easily eat the swap too. It would just die later, after rendering the system unresponsive for minutes to hours. So why not kill it quickly before GUI operation get flaky? Much processes do all their work with 10mb, some take 1gb, and some rare would need 10gb, that's life.– dronusMar 29, 2012 at 22:43