I run a bunch of CPU-hungry processes in parallel; they normally use a few GB of memory each. From time to time they allocate large amount of memory too (say 150-250GB). Usually at most one of the processes does so, so they fit in the available RAM (384GB on my machine). However, it sometimes happens that more of them allocate this large amount at the same time and (obviously) everything slows down because of swapping.
In such cases I stop all but one of the memory-hog processes, to allow it to compute effectively. But it takes ages to swap-in a stopped process, since it means loading tens of gigabytes from disk in random access pattern. Hence the question is: how I can force a single process to sequentially load entire core from swap?
So far I've only found a swappiness kernel hint, which (with a help of cgroups) can prevent a process from swapping more, but doesn't help on performance of de-swapping. Turning off all the swap is obviously not possible since the other, stopped processes have to occupy space there.
Building my own mini-scheduler is also not an option - the processes are various small scripts/programs in python and the memory peaks generally happen in library calls, so I cannot predict when a peak will occur.
Just to make clear: I don't consider buying terabytes of RAM, at this scale it is too expensive. Putting swap on SSD/SSD array will help only a bit (measured), so it is also not a solution I'm looking for.
(partial self-answer):
It seems that really sequential swap reading (only pages that belong to single process) is hardly possible without kernel hacking: I measured swapoff -a
and it certainly did not read swap sequentially. And it would be logical to read it faster if such an optimization was easy to implement.
Currently my best approach is to read the whole process memory through /proc/[pid]/mem
pseudo-file using the script below (which must be run as root):
#!/usr/bin/python2
import re
import sys
pid=str(sys.argv[1]) # process pid given by the first arg
print(pid) # just to aviod mistakes
CHUNKSIZE=10485760 # single read() invocation block size
total=0
maps_file = open("/proc/"+pid+"/maps", 'r')
mem_file = open("/proc/"+pid+"/mem", 'r', 0)
for line in maps_file.readlines(): # for each mapped region
m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])', line)
if m.group(3) == 'r': # if this is a readable region
start = int(m.group(1), 16)
end = int(m.group(2), 16)
mem_file.seek(start) # seek to region start
togo = end-start # number of bytes to read
while togo > CHUNKSIZE: # read sequential memory from region one block at the moment
mem_file.read(CHUNKSIZE)
togo -= CHUNKSIZE
total += CHUNKSIZE
print(total/1048576) # be verbose, print megabytes read so far
mem_file.read(togo) # read remaining region contents
total+=togo # dump contents to standard output
print(total/1048576) # be verbose...
maps_file.close()
mem_file.close()
It happens to fail at the very last bytes of memory, but generally works with the same performance as swapoff
and loads only the given process. Script is a modified snippet from this answer.