Yesterday, my hosting provider changed a SATA cable of one of my HDDs. When my server came back up again, cat /proc/mdstat
showed this:
Personalities : [raid1]
md124 : active raid1 sda1[0]
4193268 blocks super 1.2 [2/1] [U_]
md125 : active (auto-read-only) raid1 sda2[0]
524276 blocks super 1.2 [2/1] [U_]
md126 : active (auto-read-only) raid1 sda3[0]
268434296 blocks super 1.2 [2/1] [U_]
md127 : active raid1 sda4[0]
2657109311 blocks super 1.2 [2/1] [U_]
md3 : active (auto-read-only) raid1 sdb4[1]
2657109311 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sdb3[1]
268434296 blocks super 1.2 [2/1] [_U]
md1 : active (auto-read-only) raid1 sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active (auto-read-only) raid1 sdb1[1]
4193268 blocks super 1.2 [2/1] [_U]
I booted into the rescue console, seeing that all the arrays are degraded.
md3 : active (auto-read-only) raid1 sdb4[1]
2657109311 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sdb3[1]
268434296 blocks super 1.2 [2/1] [_U]
md1 : active (auto-read-only) raid1 sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active (auto-read-only) raid1 sdb1[1]
4193268 blocks super 1.2 [2/1] [_U]
Then, I added the missing drives to each array:
mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3
mdadm /dev/md3 -a /dev/sda4
Next, the arrays started to recover. Once complete, I rebooted into the normal system and recovery began again.
This time /dev/sdb
is marked as missing:
Personalities : [raid1]
md3 : active raid1 sda4[2] sdb4[3]
2657109311 blocks super 1.2 [2/1] [U_]
[===>.................] recovery = 17.1% (456317824/2657109311) finish=288.2min speed=127254K/sec
Recovery stopped after 3 hours and now the drive is marked as spare:
md3 : active raid1 sda4[2] sdb4[3](S)
2657109311 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sda3[2] sdb3[1]
268434296 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sda2[2] sdb2[1]
524276 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sda1[2] sdb1[1]
4193268 blocks super 1.2 [2/2] [UU]
So far I've lost no data—I checked my own email account and every email I've got before the server was shut down was still there, where the hard drive was failing 3 days ago.
How can I add the spare disk to my RAID array /dev/md3
again?
I found another question/answer which looks similar to my problem here. Would this be safe to do or could I suffer data loss?:
mdadm --grow /dev/md3 --raid-devices=3
mdadm /dev/md3 --fail /dev/{failed drive}
mdadm /dev/md3 --remove /dev/{failed drive}
mdadm --grow /dev/md3 --raid-devices=2
Of course I have backups but if I can avoid to use them I would like to do so.
EDIT: I just spotted a read error in dmesg
which was probably from before the drive failed and was marked as a spare:
[17699.328298] ata1.00: irq_stat 0x40000008
[17699.328324] ata1.00: failed command: READ FPDMA QUEUED
[17699.328356] ata1.00: cmd 60/08:00:80:d8:05/00:00:ff:00:00/40 tag 0 ncq 4096 in
[17699.328358] res 51/40:08:80:d8:05/00:00:ff:00:00/40 Emask 0x409 (media error) <F>
[17699.328446] ata1.00: status: { DRDY ERR }
[17699.328471] ata1.00: error: { UNC }
[17699.332240] ata1.00: configured for UDMA/133
[17699.332281] sd 0:0:0:0: [sda] Unhandled sense code
[17699.332308] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[17699.332342] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[17699.332384] Descriptor sense data with sense descriptors (in hex):
[17699.332415] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[17699.332491] ff 05 d8 80
[17699.332528] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[17699.332581] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 ff 05 d8 80 00 00 08 00
[17699.332648] end_request: I/O error, dev sda, sector 4278573184
[17699.332689] ata1: EH complete
[17699.332737] raid1: sda: unrecoverable I/O read error for block 3732258944
[17699.377132] md: md3: recovery done.
I tested the drive before with smartctl
:
smartctl -l selftest /dev/sda
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3444 -
[code]
[code]
smartctl -l selftest /dev/sdb
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3444
But munin
shows a smartctl
exit code of 64 and smartctl -l error /dev/sda
shows:
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 552 (device log contains only the most recent five errors)
......
Error 552 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 80 d8 05 0f
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 00 80 d8 05 40 00 20:56:57.342 READ FPDMA QUEUED
ef 10 02 00 00 00 a0 00 20:56:57.342 SET FEATURES [Reserved for Serial ATA]
27 00 00 00 00 00 e0 00 20:56:57.342 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 00 20:56:57.340 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 00 20:56:57.340 SET FEATURES [Set transfer mode]
Error 551 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
When the command that caused the error occurred, the device was active or idle.
....
EDIT #2:
mdadm --examine /dev/sdb4
/dev/sdb4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 38dec3bf:770fb774:6e9a28d0:ff3eac4a
Name : rescue:3
Creation Time : Tue Feb 26 21:21:56 2013
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 5314218895 (2534.02 GiB 2720.88 GB)
Array Size : 5314218622 (2534.02 GiB 2720.88 GB)
Used Dev Size : 5314218622 (2534.02 GiB 2720.88 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 83caa70a:6fe627f8:5a9a22d4:54a457f8
Update Time : Tue Jul 9 23:08:37 2013
Checksum : 7a729887 - correct
Events : 3478472
Device Role : spare
Array State : A. ('A' == active, '.' == missing)
My hard drive was just replaced.
Personalities : [raid1]
md2 : active raid1 sdb3[1]
268434296 blocks super 1.2 [2/1] [_U]
md1 : active raid1 sdb2[1]
524276 blocks super 1.2 [2/1] [_U]
md0 : active (auto-read-only) raid1 sdb1[1]
4193268 blocks super 1.2 [2/1] [_U]
I didn't use a tool to recover data because I was pretty sure that the data on /dev/sdb
was up to date until my server rebooted and my array broke, so I just copied the partition table from /dev/sdb
to /dev/sda
and rebuilt the array.
copy partitions
sgdisk -R /dev/sda /dev/sdb
mix ids
sgdisk -G /dev/sda
recreate array
--create /dev/md3 --level=1 --raid-devices=2 /dev/sdb4 missing
mdadm /dev/md3 -a /dev/sda3
Well, I hope this time the rebuild finishes.
sudo mdadm --examine /dev/sdb4
?/dev/sda
is having uncorrectable issues. Time to replace the disk. See the edits to my post. If it fails at different sectors each time in the logs, you might try shutting the system down and replacing the SATA cable, but if that doesn't fix it then you need to replace the drive