Two hard drives: same specs, different speeds. Need to find the configuration issue

Question

I have two nodes (with 3 dedicated data drives each) that are showing drastically different write speeds. Their 'hdparm' output looks identical and their 'hdparm -t -T' output is comparable but running a 'dd' command on a mounted file system yields drastically different write speeds. Using 'dd' to test read speeds again yields similar results.

The servers and hard drives are the exact same models, both are running the same software packages (we use chef to push packages out on our cluster).

I'm looking for ideas of parameters to check or other tests to run that can help me sort out the performance discrepancy. It looks like it's at the OS / FS level but I'm not sure what else to look at. Both mounted file systems are EXT4 with noatime and user_xattr enabled.

Fast server:

hdparm -t -T output:

/dev/sdb1:
 Timing cached reads:   2138 MB in  2.00 seconds = 1070.08 MB/sec
 Timing buffered disk reads:  232 MB in  3.02 seconds =  76.84 MB/sec

writing out a 4GB test file

$ dd bs=4K if=/dev/zero of=/mnt/vol1/test.file count=1M
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 40.1102 s, 107 MB/s
0.20user 10.91system 0:40.14elapsed 27%CPU (0avgtext+0avgdata 3472maxresident)k
16inputs+8388608outputs (1major+263minor)pagefaults 0swaps

Reading that file back off disk (and to /dev/null)

$ dd bs=4K of=/dev/null if=/mnt/vol1/test.file count=1M
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 53.3914 s, 80.4 MB/s
0.19user 5.80system 0:53.53elapsed 11%CPU (0avgtext+0avgdata 3488maxresident)k
8389872inputs+0outputs (2major+264minor)pagefaults 0swaps

Slow node:

hdparm -t -T output

/dev/sdc1:
 Timing cached reads:   1982 MB in  2.00 seconds = 991.27 MB/sec
 Timing buffered disk reads:  224 MB in  3.02 seconds =  74.16 MB/sec

$ dd bs=4K if=/dev/zero of=/mnt/vol1/test.file count=1M
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 98.1583 s, 43.8 MB/s
0.35user 17.58system 1:38.17elapsed 18%CPU (0avgtext+0avgdata 3456maxresident)k
8inputs+8388608outputs (0major+263minor)pagefaults 0swaps


$ dd bs=4k of=/dev/null if=/mnt/vol1/test.file count=1M
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 54.7789 s, 78.4 MB/s
0.25user 10.84system 0:54.92elapsed 20%CPU (0avgtext+0avgdata 3488maxresident)k
8389864inputs+0outputs (2major+263minor)pagefaults 0swaps

Answer 1

Try

hdparm -i -I /dev/sda

For both drives and diff the output, that should show you if there's a dma or lookahead setting that's different for either.

Depending on your distro, there should be a place to put the hdparm setting to make sure they are the same.

I would also double check the cables. It could just be one drive is just bad too, you might want to check the smart ECC rates and such.

/usr/sbin/smartctl -A -H /dev/sda
/usr/sbin/smartctl -a /dev/sda

Is what I use to check smart on my drives.

Answer 2

1. Maybe a disk is bad. Check for errors. (also examine the output to see if they are really the same: model, firmware, size, and sector size)

   smartctl -a /dev/sdb smartctl -a /dev/sdc

   If you have errors, run a short test (takes 2 minutes):

   smartctl -t short /dev/sdb

   If the test passes with no errors, then run again with "long" instead of "short" (takes hours).

   And then when it is done, check with "-a" again, and write zeros to your disk at that sector to relocate them (This destroys data! Be very careful about what you put in of= because that is what is overwritten at the raw level with zeros).

   eg. if your sector size is 512 and LBA 555 is bad, then type this command (destroys data!)

   dd if=/dev/zero of=/dev/sdb bs=512 count=1 seek=555

   I would do a higher count, so you don't need to repeat testing and writing zeros as often, as bad sectors are usually next to each other. (destroys much more data!)

   dd if=/dev/zero of=/dev/sdb bs=512 count=500 seek=555

2. Maybe your alignment is wrong. Make sure all partitions start on or after 63 and if your logical sector size is smaller than your physical sector size, make sure your alignment is divisible by physicalsize/logicalsize. This should greatly affect write speed, but not change read speed very much / not at all.

   eg. if physical is 4096 and logical is 512, then your start sector must be divisible by 8 (4096/512). And on some disks, the start must be much higher than 63. On those disks, I use 252 as the first partition start.

   And if you are using an SSD, you must also align to the erase block. A safe number to align to is multiples of 129024 (which meets the 63 requirement on old disks, 4096 byte sectors [advanced format disks, such as most Seagate and WD Green disks], 1024 MB on most SSDs and 2048 MB on the rare SSD)

   Also with SSDs, if they seem slow, you should erase them with vendor supplied tools before using them, or use TRIM.

3. Use proper benchmarking.

   You can not do write tests with dd unless you use conv=fdatasync or another method. David Schwartz suggested using "time dd ...", but if you use conv=fdatasync, it will tell you the correct time and speed in dd without needing to recalculate it yourself. If you have lots of RAM or write cache, you are measuring your RAM plus disk if you do not use an option like conv=fdatasync. http://romanrm.ru/en/dd-benchmark

   eg.

   dd bs=4K if=/dev/zero of=/mnt/vol1/test.file count=1M conv=fdatasync

4. Use proper benchmarking. (part2)

   Many file systems or disks will perform VERY differently when writing zeros. You need to use random files for best results.

   eg.

   first copy to RAM

   cp /somewhere/with/big/files/bigfile.iso /dev/shm

   run test

   dd bs=4K if=/dev/shm/bigfile.iso of=/mnt/vol1/test.file count=1M conv=fdatasync

   or

   prepare random file

   dd if=/dev/random of=/dev/shm/randfile bs=1M count=500

   run test

   dd bs=4K if=/dev/shm/randfile of=/mnt/vol1/test.file count=1M conv=fdatasync

5. Also, if your disks are not really the same, or have different file systems, they will perform very differently with 4k block size. Also test 128k and 1M.