I want to be able to test the read/write performance of my disks on Windows and in Linux, and get as accurate a comparison as possible.
On Windows I'm using CrystalDiskMark, because hey, it's OSS and everyone else uses it...
So I want to run the same benchmark on Linux, but CrystalDiskMark is Windows only.
So the question is, how would I go about getting the closest possible equivalent of the CrystalDiskMark results on Linux?
I've taken a stab at this myself, but I'm nowhere near qualified to say if this is an accurate comparison.
Here is the CrystalDiskMark result from a Windows VM on my PC:
That's pretty terrible.
Here is my attempt at recreating the same benchmark on the Linux host:
➜ fio --loops=5 --size=1000m --filename=/home/tomas/disktest.tmp --stonewall --ioengine=libaio --direct=1 --group_reporting \ --name=Seq-Q32-Read --bs=1m --iodepth=32 --numjobs=1 --rw=read \ --name=Seq-Q32-Write --bs=1m --iodepth=32 --numjobs=1 --rw=write \ --name=4KiB-Q8-Read --bs=4k --iodepth=8 --numjobs=8 --rw=randread \ --name=4KiB-Q8-Write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite \ --name=4KiB-Q32-Read --bs=4k --iodepth=32 --numjobs=1 --rw=randread \ --name=4KiB-Q32-Write --bs=4k --iodepth=32 --numjobs=1 --rw=randwrite \ --name=4KiB-Q1-Read --bs=4k --iodepth=1 --numjobs=1 --rw=randread \ --name=4KiB-Q1-Write --bs=4k --iodepth=1 --numjobs=1 --rw=randwrite Seq-Q32-Read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 Seq-Q32-Write: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 4KiB-Q8-Read: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... 4KiB-Q8-Write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... 4KiB-Q32-Read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 4KiB-Q32-Write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 4KiB-Q1-Read: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 4KiB-Q1-Write: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.19 Starting 22 processes Jobs: 1 (f=1): [_(21),w(1)][100.0%][w=304MiB/s][w=77.7k IOPS][eta 00m:00s] Seq-Q32-Read: (groupid=0, jobs=1): err= 0: pid=152468: Mon May 4 19:59:09 2020 read: IOPS=8143, BW=8143MiB/s (8539MB/s)(5000MiB/614msec) slat (usec): min=15, max=209, avg=18.86, stdev=11.40 clat (usec): min=535, max=7497, avg=3882.28, stdev=581.71 lat (usec): min=553, max=7517, avg=3901.21, stdev=580.07 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 3326], 10.00th=[ 3490], 20.00th=[ 3589], | 30.00th=[ 3687], 40.00th=[ 3785], 50.00th=[ 3851], 60.00th=[ 3949], | 70.00th=[ 4047], 80.00th=[ 4178], 90.00th=[ 4359], 95.00th=[ 4621], | 99.00th=[ 6128], 99.50th=[ 6652], 99.90th=[ 7242], 99.95th=[ 7308], | 99.99th=[ 7504] lat (usec) : 750=0.22%, 1000=0.26% lat (msec) : 2=0.90%, 4=64.36%, 10=34.26% cpu : usr=0.49%, sys=15.99%, ctx=4878, majf=0, minf=8203 IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=96.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=5000,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Seq-Q32-Write: (groupid=1, jobs=1): err= 0: pid=152476: Mon May 4 19:59:09 2020 write: IOPS=8561, BW=8562MiB/s (8978MB/s)(5000MiB/584msec); 0 zone resets slat (usec): min=20, max=252, avg=37.24, stdev= 8.51 clat (usec): min=1042, max=7200, avg=3674.07, stdev=787.35 lat (usec): min=1080, max=7235, avg=3711.41, stdev=787.63 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 3392], 10.00th=[ 3458], 20.00th=[ 3490], | 30.00th=[ 3490], 40.00th=[ 3523], 50.00th=[ 3556], 60.00th=[ 3556], | 70.00th=[ 3589], 80.00th=[ 3621], 90.00th=[ 3949], 95.00th=[ 5211], | 99.00th=[ 6980], 99.50th=[ 6980], 99.90th=[ 7177], 99.95th=[ 7177], | 99.99th=[ 7177] lat (msec) : 2=2.06%, 4=89.38%, 10=8.56% cpu : usr=17.32%, sys=16.64%, ctx=4993, majf=0, minf=13 IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=96.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,5000,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 4KiB-Q8-Read: (groupid=2, jobs=8): err= 0: pid=152477: Mon May 4 19:59:09 2020 read: IOPS=688k, BW=2689MiB/s (2819MB/s)(39.1GiB/14877msec) slat (nsec): min=1920, max=1768.7k, avg=3175.67, stdev=2519.59 clat (usec): min=12, max=4979, avg=89.03, stdev=27.34 lat (usec): min=15, max=4982, avg=92.28, stdev=27.45 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 73], | 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 102], 90.00th=[ 117], 95.00th=[ 129], | 99.00th=[ 159], 99.50th=[ 176], 99.90th=[ 253], 99.95th=[ 379], | 99.99th=[ 635] bw ( MiB/s): min= 2393, max= 2761, per=100.00%, avg=2700.80, stdev=10.83, samples=232 iops : min=612654, max=706936, avg=691405.41, stdev=2772.56, samples=232 lat (usec) : 20=0.01%, 50=0.01%, 100=78.31%, 250=21.58%, 500=0.07% lat (usec) : 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.76%, sys=29.86%, ctx=5989610, majf=0, minf=177 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240000,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 4KiB-Q8-Write: (groupid=3, jobs=8): err= 0: pid=152521: Mon May 4 19:59:09 2020 write: IOPS=1400k, BW=5469MiB/s (5735MB/s)(39.1GiB/7314msec); 0 zone resets slat (usec): min=2, max=6681, avg= 4.16, stdev= 7.43 clat (nsec): min=1020, max=10186k, avg=38309.07, stdev=80059.95 lat (usec): min=9, max=10188, avg=42.55, stdev=80.56 clat percentiles (usec): | 1.00th=[ 21], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 33], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 49], 95.00th=[ 59], | 99.00th=[ 94], 99.50th=[ 135], 99.90th=[ 441], 99.95th=[ 2737], | 99.99th=[ 2966] bw ( MiB/s): min= 4578, max= 6979, per=100.00%, avg=5820.14, stdev=98.27, samples=106 iops : min=1172017, max=1786792, avg=1489955.57, stdev=25158.07, samples=106 lat (usec) : 2=0.01%, 10=0.03%, 20=0.93%, 50=89.82%, 100=8.35% lat (usec) : 250=0.72%, 500=0.07%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.06%, 10=0.01%, 20=0.01% cpu : usr=12.76%, sys=73.25%, ctx=1495680, majf=0, minf=125 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240000,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 4KiB-Q32-Read: (groupid=4, jobs=1): err= 0: pid=152543: Mon May 4 19:59:09 2020 read: IOPS=304k, BW=1189MiB/s (1247MB/s)(5000MiB/4205msec) slat (nsec): min=1890, max=62518, avg=2524.61, stdev=728.21 clat (usec): min=58, max=6518, avg=102.21, stdev=36.00 lat (usec): min=61, max=6523, avg=104.80, stdev=36.05 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 85], 10.00th=[ 88], 20.00th=[ 91], | 30.00th=[ 95], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 104], | 70.00th=[ 106], 80.00th=[ 111], 90.00th=[ 117], 95.00th=[ 127], | 99.00th=[ 149], 99.50th=[ 161], 99.90th=[ 289], 99.95th=[ 388], | 99.99th=[ 619] bw ( MiB/s): min= 1106, max= 1281, per=100.00%, avg=1195.44, stdev=74.58, samples=7 iops : min=283374, max=327936, avg=306032.57, stdev=19092.48, samples=7 lat (usec) : 100=48.13%, 250=51.72%, 500=0.11%, 750=0.03%, 1000=0.01% lat (msec) : 10=0.01% cpu : usr=18.79%, sys=73.53%, ctx=10748, majf=0, minf=44 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=1280000,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 4KiB-Q32-Write: (groupid=5, jobs=1): err= 0: pid=152553: Mon May 4 19:59:09 2020 write: IOPS=260k, BW=1014MiB/s (1064MB/s)(5000MiB/4929msec); 0 zone resets slat (usec): min=2, max=1292, avg= 3.07, stdev= 3.99 clat (nsec): min=1169, max=3440.3k, avg=119726.11, stdev=38202.94 lat (usec): min=9, max=3444, avg=122.86, stdev=38.61 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 99], 10.00th=[ 101], 20.00th=[ 103], | 30.00th=[ 105], 40.00th=[ 108], 50.00th=[ 110], 60.00th=[ 115], | 70.00th=[ 127], 80.00th=[ 135], 90.00th=[ 141], 95.00th=[ 153], | 99.00th=[ 241], 99.50th=[ 285], 99.90th=[ 494], 99.95th=[ 685], | 99.99th=[ 1287] bw ( KiB/s): min=972616, max=1133568, per=99.61%, avg=1034652.00, stdev=55377.49, samples=9 iops : min=243154, max=283392, avg=258663.22, stdev=13844.19, samples=9 lat (usec) : 2=0.01%, 10=0.01%, 20=0.06%, 50=0.18%, 100=6.51% lat (usec) : 250=92.41%, 500=0.74%, 750=0.06%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=16.66%, sys=75.67%, ctx=144022, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,1280000,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 4KiB-Q1-Read: (groupid=6, jobs=1): err= 0: pid=152562: Mon May 4 19:59:09 2020 read: IOPS=15.4k, BW=60.1MiB/s (63.0MB/s)(5000MiB/83174msec) slat (nsec): min=2080, max=26179, avg=2459.69, stdev=530.83 clat (usec): min=5, max=70982, avg=62.10, stdev=63.77 lat (usec): min=8, max=70985, avg=64.62, stdev=63.77 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 60], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 64], 90.00th=[ 64], 95.00th=[ 65], | 99.00th=[ 67], 99.50th=[ 68], 99.90th=[ 139], 99.95th=[ 229], | 99.99th=[ 461] bw ( KiB/s): min=59920, max=62672, per=100.00%, avg=61700.32, stdev=396.46, samples=165 iops : min=14980, max=15668, avg=15425.08, stdev=99.12, samples=165 lat (usec) : 10=0.04%, 20=0.02%, 50=0.01%, 100=99.78%, 250=0.11% lat (usec) : 500=0.04%, 750=0.01% lat (msec) : 4=0.01%, 10=0.01%, 100=0.01% cpu : usr=1.26%, sys=4.78%, ctx=1280023, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1280000,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 4KiB-Q1-Write: (groupid=7, jobs=1): err= 0: pid=152734: Mon May 4 19:59:09 2020 write: IOPS=78.8k, BW=308MiB/s (323MB/s)(5000MiB/16249msec); 0 zone resets slat (nsec): min=2210, max=21549, avg=2459.24, stdev=272.82 clat (nsec): min=360, max=4257.7k, avg=9833.56, stdev=6903.88 lat (usec): min=10, max=4260, avg=12.35, stdev= 6.92 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 11], 60.00th=[ 11], | 70.00th=[ 11], 80.00th=[ 11], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 13], 99.50th=[ 14], 99.90th=[ 17], 99.95th=[ 20], | 99.99th=[ 133] bw ( KiB/s): min=310086, max=321632, per=100.00%, avg=315613.19, stdev=2609.60, samples=32 iops : min=77521, max=80408, avg=78903.28, stdev=652.43, samples=32 lat (nsec) : 500=0.01%, 750=0.01% lat (usec) : 4=0.01%, 10=32.92%, 20=67.03%, 50=0.02%, 100=0.01% lat (usec) : 250=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.50%, sys=25.87%, ctx=1280022, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1280000,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=8143MiB/s (8539MB/s), 8143MiB/s-8143MiB/s (8539MB/s-8539MB/s), io=5000MiB (5243MB), run=614-614msec Run status group 1 (all jobs): WRITE: bw=8562MiB/s (8978MB/s), 8562MiB/s-8562MiB/s (8978MB/s-8978MB/s), io=5000MiB (5243MB), run=584-584msec Run status group 2 (all jobs): READ: bw=2689MiB/s (2819MB/s), 2689MiB/s-2689MiB/s (2819MB/s-2819MB/s), io=39.1GiB (41.9GB), run=14877-14877msec Run status group 3 (all jobs): WRITE: bw=5469MiB/s (5735MB/s), 5469MiB/s-5469MiB/s (5735MB/s-5735MB/s), io=39.1GiB (41.9GB), run=7314-7314msec Run status group 4 (all jobs): READ: bw=1189MiB/s (1247MB/s), 1189MiB/s-1189MiB/s (1247MB/s-1247MB/s), io=5000MiB (5243MB), run=4205-4205msec Run status group 5 (all jobs): WRITE: bw=1014MiB/s (1064MB/s), 1014MiB/s-1014MiB/s (1064MB/s-1064MB/s), io=5000MiB (5243MB), run=4929-4929msec Run status group 6 (all jobs): READ: bw=60.1MiB/s (63.0MB/s), 60.1MiB/s-60.1MiB/s (63.0MB/s-63.0MB/s), io=5000MiB (5243MB), run=83174-83174msec Run status group 7 (all jobs): WRITE: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=5000MiB (5243MB), run=16249-16249msec Disk stats (read/write): dm-4: ios=12805000/12795016, merge=0/0, ticks=1073203/182228, in_queue=1255557, util=98.33%, aggrios=12840923/12850350, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md0: ios=12840923/12850350, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=4280230/4283484, aggrmerge=77/475, aggrticks=398850/90297, aggrin_queue=45665, aggrutil=98.31% nvme1n1: ios=4280757/4283635, merge=82/428, ticks=398406/100042, in_queue=49090, util=98.30% nvme0n1: ios=4274346/4278171, merge=80/489, ticks=395498/78447, in_queue=31827, util=98.31% nvme2n1: ios=4285589/4288647, merge=69/510, ticks=402648/92403, in_queue=56080, util=98.30%
But I am not sure if all the options I'm using are equivalent to the settings CrystalDiskMark uses. In particular I have no idea if --jobs
is equivalent to the thread count in CrystalDiskMark.
The point of this exercise (for me) is to try to figure out which virtual disk technology will have the least performance impact, but to do that I need to be able to run the same benchmark on both Windows and Linux.