Is there alternative allocator that can be LD_PRELOADed into a process, overrides "malloc", "calloc", etc. and allocates things using it's own swap files (mmaped to files instead of MAP_ANONYMOUS)?
I hope it will prevent a memory-hungry process from slowing down and thrashing the whole system.