I had created five 1TB HDD partitions (/dev/sda1
, /dev/sdb1
, /dev/sdc1
, /dev/sde1
, and /dev/sdf1
) in a RAID 6 array called /dev/md0
using mdadm
on Ubuntu 14.04 LTS Trusty Tahr.
The command sudo mdadm --detail /dev/md0
used to show all drives in active sync.
Then, for testing, I simulated long I/O blocking on /dev/sdb
by running these commands while /dev/sdb1
was still active in the array:
hdparm --user-master u --security-set-pass deltik /dev/sdb
hdparm --user-master u --security-erase-enhanced deltik /dev/sdb
WARNING
DON'T TRY THIS ON DATA YOU CARE ABOUT!
I ended up corrupting 455681 inodes as a result of this ATA operation. I admit my negligence.
The ATA command for secure erase was expected to run for 188 minutes, blocking all other commands for at least that long.
I expected md
to drop the unresponsive drive like a proper RAID controller, but to my surprise, /dev/md0
became blocked as well.
mdadm --detail /dev/md0
queries the blocked device, so it freezes and won't output.
Here's the layout from /proc/mdstat
while I can't use mdadm --detail /dev/md0
:
root@node51 [~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdf1[5] sda1[0] sdb1[4] sdc1[2] sde1[1]
2929887744 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
unused devices: <none>
I tried mdadm /dev/md0 -f /dev/sdb1
to forcefully fail /dev/sdb1
, but that was also blocked:
root@node51 [~]# ps aux | awk '{if($8~"D"||$8=="STAT"){print $0}}'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3334 1.2 0.0 42564 1800 ? D 03:21 3:37 parted -l
root 4957 0.0 0.0 13272 900 ? D 06:19 0:00 mdadm /dev/md0 -f /dev/sdb1
root 5706 0.0 0.0 13388 1028 ? D 06:19 0:00 mdadm --detail /dev/md0
root 7541 0.5 0.0 0 0 ? D Jul19 6:12 [kworker/u16:2]
root 22420 0.0 0.0 11480 808 ? D 07:48 0:00 lsblk
root 22796 0.0 0.0 4424 360 pts/13 D+ 05:51 0:00 hdparm --user-master u --security-erase-enhanced deltik /dev/sdb
root 23312 0.0 0.0 4292 360 ? D 05:51 0:00 hdparm -I /dev/sdb
root 23594 0.1 0.0 0 0 ? D 06:11 0:07 [kworker/u16:1]
root 25205 0.0 0.0 17980 556 ? D 05:52 0:00 ls --color=auto
root 26008 0.0 0.0 13388 1032 pts/23 D+ 06:32 0:00 mdadm --detail /dev/md0
dtkms 29271 0.0 0.2 58336 10412 ? DN 05:55 0:00 python /usr/share/backintime/common/backintime.py --backup-job
root 32303 0.0 0.0 0 0 ? D 06:16 0:00 [kworker/u16:0]
UPDATE (21 July 2015): After I waited the full 188 minutes for the I/O block to be cleared, surprise turned to horror when I saw that md
treated the completely blanked out /dev/sdb
as if it were completely in tact.
I thought that md
would have at least seen that parity was mismatched and then would have dropped /dev/sdb1
.
Panicking, I ran mdadm /dev/md0 -f /dev/sdb1
again, and since the I/O block had been lifted, the command completed quickly.
Filesystem corruption was already happening as input/output errors cropped up. Still panicking, I did a lazy unmount of the data partition in the RAID array and a reboot -nf
since I figured it couldn't get any worse.
After a nail-biting e2fsck
on the partition, 455681 inodes made it into lost+found
.
I've since reassembled the array, and the array itself looks fine now:
root@node51 [~]# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Feb 16 14:34:26 2015
Raid Level : raid6
Array Size : 2929887744 (2794.16 GiB 3000.21 GB)
Used Dev Size : 976629248 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Tue Jul 21 00:00:30 2015
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : box51:0
UUID : 6b8a654d:59deede9:c66bd472:0ceffc61
Events : 643541
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 97 1 active sync /dev/sdg1
2 8 33 2 active sync /dev/sdc1
6 8 17 3 active sync /dev/sdb1
5 8 113 4 active sync /dev/sdh1
It's still quite a shock to me that md
doesn't have two lines of protection that I expected:
- Failing a device when it locks up
- Failing a device when the data it returns are garbage
Questions
- Why doesn't
md
fail the unresponsive drive/partition? - Can I drop the drive/partition from the array while the drive is blocked?
- Can a timeout be configured so that
md
automatically fails a drive that isn't responding to ATA commands? - Why does
md
continue to use a device with invalid data?