5

I run a bunch of CPU-hungry processes in parallel; they normally use a few GB of memory each. From time to time they allocate large amount of memory too (say 150-250GB). Usually at most one of the processes does so, so they fit in the available RAM (384GB on my machine). However, it sometimes happens that more of them allocate this large amount at the same time and (obviously) everything slows down because of swapping.

In such cases I stop all but one of the memory-hog processes, to allow it to compute effectively. But it takes ages to swap-in a stopped process, since it means loading tens of gigabytes from disk in random access pattern. Hence the question is: how I can force a single process to sequentially load entire core from swap?

So far I've only found a swappiness kernel hint, which (with a help of cgroups) can prevent a process from swapping more, but doesn't help on performance of de-swapping. Turning off all the swap is obviously not possible since the other, stopped processes have to occupy space there.

Building my own mini-scheduler is also not an option - the processes are various small scripts/programs in python and the memory peaks generally happen in library calls, so I cannot predict when a peak will occur.


Just to make clear: I don't consider buying terabytes of RAM, at this scale it is too expensive. Putting swap on SSD/SSD array will help only a bit (measured), so it is also not a solution I'm looking for.


(partial self-answer):

It seems that really sequential swap reading (only pages that belong to single process) is hardly possible without kernel hacking: I measured swapoff -a and it certainly did not read swap sequentially. And it would be logical to read it faster if such an optimization was easy to implement.

Currently my best approach is to read the whole process memory through /proc/[pid]/mem pseudo-file using the script below (which must be run as root):

#!/usr/bin/python2
import re
import sys
pid=str(sys.argv[1]) # process pid given by the first arg

print(pid) # just to aviod mistakes

CHUNKSIZE=10485760  # single read() invocation block size

total=0
maps_file = open("/proc/"+pid+"/maps", 'r')
mem_file = open("/proc/"+pid+"/mem", 'r', 0)
for line in maps_file.readlines():  # for each mapped region
    m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])', line)
    if m.group(3) == 'r':  # if this is a readable region
        start = int(m.group(1), 16)
        end = int(m.group(2), 16)
        mem_file.seek(start)  # seek to region start
        togo = end-start # number of bytes to read
        while togo > CHUNKSIZE: # read sequential memory from region one block at the moment
            mem_file.read(CHUNKSIZE) 
            togo -= CHUNKSIZE
            total += CHUNKSIZE
            print(total/1048576) # be verbose, print megabytes read so far
        mem_file.read(togo) # read remaining region contents
        total+=togo # dump contents to standard output
        print(total/1048576) # be verbose...
maps_file.close()
mem_file.close()

It happens to fail at the very last bytes of memory, but generally works with the same performance as swapoff and loads only the given process. Script is a modified snippet from this answer.

7
  • what about RAID0 array of 16 SSDs for swap space? If you can get hands on SAS raid card 16e it should provide you with up to 24Gbps bandwidth (3GB/sec) Jan 29, 2016 at 8:19
  • I've measured and SSD is not fast enough (in terms of random access) to handle such a load more effectively than a sequential reading from a RAID of HDDs. It's a single threaded load and random access even on fast SSDs doesn't count in hundreds MB per second. RAID won't help here at all. I hoped there will be no answers like this or "buy more RAM":(
    – bzaborow
    Jan 29, 2016 at 9:03
  • just crossing the t's and dotting the i's. You didn't say you can't get more RAM (1.5TB?) or if you considered RAIDing swap file. I am interested for our stuff, might come in handy Jan 29, 2016 at 9:08
  • OK, you're right. I'm adding this explicitely to the description. I'm sorry if you feel offended.
    – bzaborow
    Jan 29, 2016 at 9:15
  • 1
    As far as I know you are right, it is block-level. In my question I mean loading all swapped-out blocks owned by a process, not anything like per-process swap. EDIT: a separate VM for each process might be an option as I can hibernate it. I'll try it if I get no better answer. Thanks!
    – bzaborow
    Jan 29, 2016 at 9:31

0

You must log in to answer this question.

Browse other questions tagged .