We've all experienced it--some program is asked to do something that requires a huge amount of memory. It dutifully tries to allocate all this memory, and the system immediately begins thrashing, swapping endlessly and becoming sluggish or non-responsive.
I most recently experienced this on my Ubuntu laptop due to a Matlab script trying to allocate a ridiculously huge matrix. After ~5+ minutes of thrashing, I was able to Ctrl-F1 to a console and kill Matlab. I would much rather have some hot-key that would have given me control of the system immediately and allowed me to kill the offending process; or, perhaps, simply silently refuse to allocate such a large buffer.
What is the quickest way to regain control of a Linux system that has become nonresponsive or extremely sluggish due to excessive swapping?
Is there an effective way to prevent such swapping from occurring in the first place, for instance by limiting the amount of memory a process is allowed to try to allocate?