1

I have a fairly IO-heavy GPU-limited process that needs to read in random files from a folder in a local hard drive. When the process is running by itself, I get a consistent throughput of around 30 MB/s, but when there are two competing processes, the total throughput drops to barely 7 MB/s.

How can I maximise throughput when having two programs? Latency is not a problem.

Each file is in the order of 1-20 MB, The processes are running on independent GPUs, and they use very little CPU. The same effect is observed if I launch one GPU and one pure IO process at the same time.

There are no differences between the available schedulers: deadline, cfq, and noop. I also tried to increase the read deadline time to 5 s, without changes.

Machine details:

  • Fedora Linux with kernel 4.16.7-200.fc27.x86_64
  • i7-4770 CPU @ 3.40GHz
  • 32 GB of RAM, of which 20 are taken by running processes.
  • Swap is enabled, but empty.
  • The drive is a WDC WD2003FYYS-0, 2TB, but I see the same if I move everything to other drives.
  • cat big_file > /dev/null gives a throughput of nearly 100 MB/s, so there is bandwidth for both.
  • The whole data is around 500 GB.

More info:

  • I moved the files to a different, faster drive that is not being used for anything else, and used compression. The overall throughput is slightly improved.
  • Giving maximum io priority to one of the processes improved performance by 10%.
  • Running iostat -x 1 shows that utilisation is around 87 % when running one process, and 100 % when running two.
  • The processes are reading random files. If I had only one process, it can provide more than double the throughput each of the individual ones can consume.
  • 3
    Get an SSD. Hard drives are awful with concurrent access. – Mokubai Jul 17 '18 at 16:43
  • @Mokubai I don't need more than what a HD can provide, and since my requests are queued, I don't mind latency, so the scheduler could, for example, give a full second dedicated to each process, at maximum throughput. – Davidmh Jul 17 '18 at 20:49
  • A shot in the dark: experiment with ionice -c best-effort with different -n levels for the two processes; or even try -c realtime for one of them. – Kamil Maciorowski Jul 17 '18 at 21:21
  • @KamilMaciorowski it seems to help marginally (~10% improvement). Thanks! – Davidmh Jul 20 '18 at 12:55
  • But you do need more than your hard drives can provide. SSDs are several orders of magnitude faster at random access than mechanical hard drives. You could of course create a custom daemon to manage access to files, with a larger read and write buffer, but why bother? – Daniel B Jul 20 '18 at 13:48
2

Use cfq shceduler for those data disks and set slice_async and slice_sync to pretty high values (e.g. 500) and slice_idle to around 20. You may also need to increase fifo_expire_async and fifo_expire_sync to around 4000 each. (See https://unix.stackexchange.com/a/41831/20336 for details).

The idea is to allow each process to get total control of the device for 0.5 sec at a time so that seeking between different areas of disk does not take all the time.

If you can afford it, the best choice would be to get a good SSD drive that can hold your data. For random read performance I'd suggest Samsung 860 EVO series because year 2018 that seems to be best balance between costs and performance. If price is not an issue, go with the biggest Intel Optane SSD available in the market.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.