I have a fairly IO-heavy GPU-limited process that needs to read in random files from a folder in a local hard drive. When the process is running by itself, I get a consistent throughput of around 30 MB/s, but when there are two competing processes, the total throughput drops to barely 7 MB/s.
How can I maximise throughput when having two programs? Latency is not a problem.
Each file is in the order of 1-20 MB, The processes are running on independent GPUs, and they use very little CPU. The same effect is observed if I launch one GPU and one pure IO process at the same time.
There are no differences between the available schedulers: deadline, cfq, and noop. I also tried to increase the read deadline time to 5 s, without changes.
- Fedora Linux with kernel 4.16.7-200.fc27.x86_64
- i7-4770 CPU @ 3.40GHz
- 32 GB of RAM, of which 20 are taken by running processes.
- Swap is enabled, but empty.
- The drive is a WDC WD2003FYYS-0, 2TB, but I see the same if I move everything to other drives.
- cat big_file > /dev/null gives a throughput of nearly 100 MB/s, so there is bandwidth for both.
- The whole data is around 500 GB.
- I moved the files to a different, faster drive that is not being used for anything else, and used compression. The overall throughput is slightly improved.
- Giving maximum io priority to one of the processes improved performance by 10%.
iostat -x 1shows that utilisation is around 87 % when running one process, and 100 % when running two.
- The processes are reading random files. If I had only one process, it can provide more than double the throughput each of the individual ones can consume.