I have a custom program that performs analysis on some data. This is a very memory-hungry operation that can potentially use thousands of gigabytes of RAM. My computer only has 16gb of RAM, so my only solution is to use swap memory on large hard drives.
I managed to do this already and it worked well, however the process eats up all of my real RAM first before starting to use the swap space. This makes my computer completely unusable during the very long time it takes to execute the operation. It also makes it difficult to monitor the operation because everything becomes extremely slow and unresponsive.
So is there a way in Linux to force a single process to only use memory in swap space? (without having to implement a custom allocator)
Ideally, I could leave the program running in the background (potentially for days or weeks) while still having access to the computer. The benefit of an extra ~16gb of RAM to the program seems hardly worth the loss of a computer for multiple days.