I've been having this severe performance issue, both in Ubuntu 11.10, and Debian Wheezy:
If one process uses up a great deal of RAM (say, 3700 MB out of 3900 MB), the system quickly becomes unusable, with constant disk thrashing. It gets to the point where the window manager doesn't respond, and killing the offending process takes several minutes. (Actually getting to a terminal, issuing pkill, waiting a while longer until the signal actually gets through)
This problem occurs even if the swap partition is completely disabled.
Investigating with iotop, it looks like most of the disk activity is disk read, going to processes such as google chrome, and the X window renderer. However, this is only limited information, as iotop no longer updates while the system is in full-thrash mode.
My question is:
1) what is causing the massive disk read? 2) how can I prevent this from making the system unusable?