Cannot diagnose slow disk read speed, bottleneck, in Ubuntu

Question

I have the same processes running on 2 (or more machines) for which we notice a massive slow down in disk i/o on some machines. The PCs have very similar hardware (I am happy to post dmidecode if necessary).

We are having trouble even diagnosing the issue. So let me describe it the best I can. All PCs run Ubuntu 16.04. (Note: The same results were found when testing on one of these PCs with Ubuntu 18.04).

The program we are running randomly picks 32 files (images) to read. These files are about 512K each. On the 'fast' machines we can read all files in about 0.5 sec. (about 320+MB/s) and on the 'slow' PCs it takes about 2-5 seconds to read 32 files.

Some things we have figured out so far:

• Read-time output of hdparam on slow and fast machines is comparable.
• Slowdowns consistent for 5400, 7200 RPM disks as well as SSDs.
• slowdown happens on both RAID-5 and RAID-0
• the slowdown seems to go away when we don't shuffle images (no random access)

****.

Here is the output of atop during the tests:

Fast:

MDD |          md1  | busy      0%  | read   22474  | write      0 |  KiB/w      0 |  MBr/s 350.73 |  MBw/s   0.00 |  avio 0.00 ms |
DSK |          sdh  | busy     20%  | read    7251  | write      0 |  KiB/w      0 |  MBr/s  86.58 |  MBw/s   0.00 |  avio 0.27 ms |
DSK |          sdf  | busy     20%  | read    7375  | write      0 |  KiB/w      0 |  MBr/s  88.31 |  MBw/s   0.00 |  avio 0.27 ms |
DSK |          sde  | busy     19%  | read    7399  | write      0 |  KiB/w      0 |  MBr/s  89.36 |  MBw/s   0.00 |  avio 0.26 ms |
DSK |          sdg  | busy     19%  | read    7303  | write      0 |  KiB/w      0 |  MBr/s  86.50 |  MBw/s   0.00 |  avio 0.25 ms |

Slow:

MDD |          md0  | busy      0%  | read    4018  | write      0 |  KiB/w      0 |  MBr/s 140.46 |  MBw/s   0.00 |  avio 0.00 ms |
DSK |          sdd  | busy    100%  | read    1181  | write      0 |  KiB/w      0 |  MBr/s  36.34 |  MBw/s   0.00 |  avio 8.45 ms |
DSK |          sdc  | busy     99%  | read    1153  | write      0 |  KiB/w      0 |  MBr/s  35.29 |  MBw/s   0.00 |  avio 8.55 ms |
DSK |          sde  | busy     99%  | read    1150  | write      0 |  KiB/w      0 |  MBr/s  34.84 |  MBw/s   0.00 |  avio 8.57 ms |

If I am reading it correctly slow machine has maxed out percentages, and ~1/7th of the read speed.

We've tried schedulers which did not change anything.

I am looking for help even diagnosing the issue. I am happy to help improve the question with any extra details I can. We are at a loss in even figuring out what the issue is, let alone solving it...

UPDATE Per recommendation from @agtoever, posting atop+d results:

Slow Machine:

Fast Machine: