0

I have a Python multiprocessing program and it runs on processing a text file of 40GB. It seems it becomes slower when it runs longer and longer on a Ubuntu machine. In particular, I noticed that many subprocesses are using 0% of cpu power. Occasionally, those child processes are using cpu too, but most of the time they are not using cpu at all.

What might be the program's problem? The memory is huge of this machine, so it's not a memory issue.

vmstat output:

:~# vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  0      0 125372376 513092 86170112    0    0    18    97    0    0  8 10 82  0  0
 3  0      0 125088040 513092 86170112    0    0     0    32 23057 5634  8  4 89  0  0
 4  0      0 125617664 513092 86170104    0    0     0     5 23279 6038  8 14 78  0  0
 4  0      0 125224624 513092 86170112    0    0     0    43 24269 6734  8  4 88  0  0
 5  0      0 126028016 513092 86170128    0    0     0   172 23112 5805  8 12 81  0  0
 4  0      0 125310704 513092 86170136    0    0     0   236 24020 6637 10  4 86  0  0
12  0      0 126350656 513092 86170288    0    0     0   572 27158 6431  9 12 79  0  0
 4  0      0 125345584 513092 86170336    0    0     0   332 30882 6405  9  5 86  0  0
 3  0      0 125047856 513092 86170336    0    0     0    52 24165 6408  7  3 90  0  0
 4  0      0 125438048 513092 86170336    0    0     0     1 21120 5704  8 14 77  0  0
 3  0      0 125139064 513092 86170336    0    0     0    24 24790 6116  7  3 89  0  0
 4  0      0 125601800 513092 86170304    0    0     0    11 21626 5594  8 12 80  0  0
 3  0      0 125213312 513092 86170304    0    0     0    56 26952 6679  8  4 88  0  0
 4  0      0 126086096 513092 86170304    0    0     0    12 23574 5922  8 11 81  0  0
 3  0      0 125342456 513092 86170344    0    0     0    33 26524 6022  9  4 87  0  0
35  0      0 125595744 513092 86170336    0    0     0    16 17467 6340  7  5 88  0  0
 4  0      0 125431152 513092 86170336    0    0     0   209 23077 5967  8 10 82  0  0
 5  0      0 125092640 513092 86170336    0    0     0   571 22415 6603  9  4 87  0  0
 5  0      0 125586400 513092 86170456    0    0     0   168 24643 6681 10 12 78  0  0
 3  0      0 125207424 513092 86170464    0    0     0   227 32891 6183  8  4 89  0  0
 4  0      0 125813376 513092 86170448    0    0     0    15 24456 6451  9 14 76  0  0
 3  0      0 125302176 513092 86170456    0    0     0    28 26945 5461  8  4 88  0  0
 8  0      0 126314288 513092 86170456    0    0     0    11 21473 5201  7 13 80  0  0
Improve this question
6
  • Don't rule out a memory issue so fast. Run vmstat while the programming is running and look for the bottleneck. My total guess is you need to tune VM.swappiness
    – davidgo
    Aug 19, 2020 at 19:52
  • But I have a 200GB memory and free -m shows that only about 20G are used. Would that be a still memory issue possibly?
    – martin
    Aug 19, 2020 at 20:00
  • Your output shows its not swap/disk io. It seems like the program is blocking/waiting, so not taking good advantage of parellellism.
    – davidgo
    Aug 19, 2020 at 20:19
  • you have many more threads than CPU cores (well, probably) so most of them are going to be waiting. Also each thread is likely performing DMA requests so it will be paused while the data from secondary storage is retrieved into ram, and your file is larger than your physical ram, so the result is soft locks where lots of threads are waiting for other threads to complete, so ram is free so a dma request can be fufilled, allowing a thread to run, and stop blocking other threads.
    – Frank Thomas
    Aug 19, 2020 at 21:28
  • I'd advise you set a limit on the number of threads (perhaps 2x cores), and make sure you are processing the file in reasonable chunks that are semi-adjascent.
    – Frank Thomas
    Aug 19, 2020 at 21:30

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.