I have two servers with identical hardware, both running CentOS 5.8. They both have 40 TB external RAID arrays attached (via eSATA) to LSI MegaRAID SAS 9280 cards. The array has a GUID partition table and there is one big XFS partition. Disk performance on Machine1 is much better than Machine2, and I am trying to figure out why. It's evident from this hdparm output:
[root@Machine1]# hdparm -t /dev/sdc1 /dev/sdc1: Timing buffered disk reads: 1150 MB in 3.00 seconds = 383.27 MB/sec [root@Machine2]# hdparm -t /dev/sdc1 /dev/sdc1: Timing buffered disk reads: 286 MB in 3.03 seconds = 94.55 MB/sec
...and it's also evident when I have big jobs running with intensive disk I/O (processes with heavy IO tend to go into uninterruptible sleep (status "D" on top) on Machine2, but not on Machine1.
I noticed that the block size usage reported by ls -ls (first colum) differs for the same files compared across machines:
[root@Machine1]$ ls -ls 8 -rw-rw-r-- 1 user1 user1 600 Feb 18 15:41 file1.txt 48 -rwxrwxr-x 1 user1 user1 43814 Feb 18 15:31 file2.gz 696 -rwxrwxr-x 1 user1 user1 706287 Feb 18 15:41 file3.txt [root@Machine2]$ ls -ls 4 -rw-rw-r-- 1 user1 user1 600 Feb 18 11:57 file1.txt 44 -rwxrwxr-x 1 user1 user1 43814 Feb 18 10:15 file2.gz 692 -rwxrwxr-x 1 user1 user1 706287 Feb 18 10:15 file3.txt
The only other difference I could find was using hdparm -g to show geometry. It reveals differing number of cylinders (first number):
[root@Machine1]# hdparm -g /dev/sdc /dev/sdc: geometry = 40529/255/63, sectors = 85930803200, start = 0 [root@Machine2]# hdparm -g /dev/sdc /dev/sdc: geometry = 5348945/255/63, sectors = 85930803200, start = 0
But fdisk reports the same number of cylinders for both machines (only showing Machine1 below) -- or maybe fdisk can't be trusted due to the warning messages?
[root@Machine1]# fdisk -lu /dev/sdc WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted. WARNING: The size of this disk is 44.0 TB (43996571238400 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Disk /dev/sdc: 43996.5 GB, 43996571238400 bytes 255 heads, 63 sectors/track, 5348945 cylinders, total 85930803200 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 4294967295 2147483647+ ee EFI GPT
xfs_info output is identical for both machines (only showing Machine1 below):
[root@Machine1]# xfs_info /dev/sdc1 meta-data=/dev/sdc1 isize=256 agcount=41, agsize=268435455 blks = sectsz=512 attr=1 data = bsize=4096 blocks=10741350391, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
I'm currently stumped. Can anyone come up with ideas for next steps?