I have the same processes running on 2 (or more machines) for which we notice a massive slow down in disk i/o on some machines. The PCs have very similar hardware (I am happy to post dmidecode if necessary).
We are having trouble even diagnosing the issue. So let me describe it the best I can. All PCs run Ubuntu 16.04. (Note: The same results were found when testing on one of these PCs with Ubuntu 18.04).
The program we are running randomly picks 32 files (images) to read. These files are about 512K each. On the 'fast' machines we can read all files in about 0.5 sec. (about 320+MB/s) and on the 'slow' PCs it takes about 2-5 seconds to read 32 files.
Some things we have figured out so far:
- Read-time output of
hdparamon slow and fast machines is comparable.
- Slowdowns consistent for 5400, 7200 RPM disks as well as SSDs.
- slowdown happens on both RAID-5 and RAID-0
- the slowdown seems to go away when we don't shuffle images (no random access)
Here is the output of atop during the tests:
MDD | md1 | busy 0% | read 22474 | write 0 | KiB/w 0 | MBr/s 350.73 | MBw/s 0.00 | avio 0.00 ms | DSK | sdh | busy 20% | read 7251 | write 0 | KiB/w 0 | MBr/s 86.58 | MBw/s 0.00 | avio 0.27 ms | DSK | sdf | busy 20% | read 7375 | write 0 | KiB/w 0 | MBr/s 88.31 | MBw/s 0.00 | avio 0.27 ms | DSK | sde | busy 19% | read 7399 | write 0 | KiB/w 0 | MBr/s 89.36 | MBw/s 0.00 | avio 0.26 ms | DSK | sdg | busy 19% | read 7303 | write 0 | KiB/w 0 | MBr/s 86.50 | MBw/s 0.00 | avio 0.25 ms |
MDD | md0 | busy 0% | read 4018 | write 0 | KiB/w 0 | MBr/s 140.46 | MBw/s 0.00 | avio 0.00 ms | DSK | sdd | busy 100% | read 1181 | write 0 | KiB/w 0 | MBr/s 36.34 | MBw/s 0.00 | avio 8.45 ms | DSK | sdc | busy 99% | read 1153 | write 0 | KiB/w 0 | MBr/s 35.29 | MBw/s 0.00 | avio 8.55 ms | DSK | sde | busy 99% | read 1150 | write 0 | KiB/w 0 | MBr/s 34.84 | MBw/s 0.00 | avio 8.57 ms |
If I am reading it correctly
slow machine has maxed out percentages, and ~1/7th of the read speed.
We've tried schedulers which did not change anything.
I am looking for help even diagnosing the issue. I am happy to help improve the question with any extra details I can. We are at a loss in even figuring out what the issue is, let alone solving it...
UPDATE Per recommendation from @agtoever, posting atop+d results: