3

Yesterday, my hosting provider changed a SATA cable of one of my HDDs. When my server came back up again, cat /proc/mdstat showed this:

Personalities : [raid1]
md124 : active raid1 sda1[0]
      4193268 blocks super 1.2 [2/1] [U_]

md125 : active (auto-read-only) raid1 sda2[0]
      524276 blocks super 1.2 [2/1] [U_]

md126 : active (auto-read-only) raid1 sda3[0]
      268434296 blocks super 1.2 [2/1] [U_]

md127 : active raid1 sda4[0]
      2657109311 blocks super 1.2 [2/1] [U_]

md3 : active (auto-read-only) raid1 sdb4[1]
      2657109311 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

I booted into the rescue console, seeing that all the arrays are degraded.

md3 : active (auto-read-only) raid1 sdb4[1]
      2657109311 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

Then, I added the missing drives to each array:

mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3
mdadm /dev/md3 -a /dev/sda4

Next, the arrays started to recover. Once complete, I rebooted into the normal system and recovery began again.

This time /dev/sdb is marked as missing:

Personalities : [raid1]
md3 : active raid1 sda4[2] sdb4[3]
      2657109311 blocks super 1.2 [2/1] [U_]
      [===>.................]  recovery = 17.1% (456317824/2657109311) finish=288.2min speed=127254K/sec

Recovery stopped after 3 hours and now the drive is marked as spare:

md3 : active raid1 sda4[2] sdb4[3](S)
      2657109311 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[2] sdb3[1]
      268434296 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[2] sdb2[1]
      524276 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[2] sdb1[1]
      4193268 blocks super 1.2 [2/2] [UU]

So far I've lost no data—I checked my own email account and every email I've got before the server was shut down was still there, where the hard drive was failing 3 days ago.

How can I add the spare disk to my RAID array /dev/md3 again?

I found another question/answer which looks similar to my problem here. Would this be safe to do or could I suffer data loss?:

mdadm --grow /dev/md3 --raid-devices=3
mdadm /dev/md3 --fail /dev/{failed drive}
mdadm /dev/md3 --remove /dev/{failed drive}
mdadm --grow /dev/md3 --raid-devices=2

Of course I have backups but if I can avoid to use them I would like to do so.


EDIT: I just spotted a read error in dmesg which was probably from before the drive failed and was marked as a spare:

[17699.328298] ata1.00: irq_stat 0x40000008
[17699.328324] ata1.00: failed command: READ FPDMA QUEUED
[17699.328356] ata1.00: cmd 60/08:00:80:d8:05/00:00:ff:00:00/40 tag 0 ncq 4096 in
[17699.328358]          res 51/40:08:80:d8:05/00:00:ff:00:00/40 Emask 0x409 (media error) <F>
[17699.328446] ata1.00: status: { DRDY ERR }
[17699.328471] ata1.00: error: { UNC }
[17699.332240] ata1.00: configured for UDMA/133
[17699.332281] sd 0:0:0:0: [sda] Unhandled sense code
[17699.332308] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[17699.332342] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[17699.332384] Descriptor sense data with sense descriptors (in hex):
[17699.332415]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[17699.332491]         ff 05 d8 80
[17699.332528] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[17699.332581] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 ff 05 d8 80 00 00 08 00
[17699.332648] end_request: I/O error, dev sda, sector 4278573184
[17699.332689] ata1: EH complete
[17699.332737] raid1: sda: unrecoverable I/O read error for block 3732258944
[17699.377132] md: md3: recovery done.

I tested the drive before with smartctl:

smartctl -l selftest /dev/sda
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3444         -
[code]

[code]
smartctl -l selftest /dev/sdb
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3444   

But munin shows a smartctl exit code of 64 and smartctl -l error /dev/sda shows:

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 552 (device log contains only the most recent five errors)
......
Error 552 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:

  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 80 d8 05 0f

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 00 80 d8 05 40 00      20:56:57.342  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00      20:56:57.342  SET FEATURES [Reserved for Serial ATA]
  27 00 00 00 00 00 e0 00      20:56:57.342  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00      20:56:57.340  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      20:56:57.340  SET FEATURES [Set transfer mode]


Error 551 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.
....

EDIT #2:

mdadm --examine /dev/sdb4
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 38dec3bf:770fb774:6e9a28d0:ff3eac4a
           Name : rescue:3
  Creation Time : Tue Feb 26 21:21:56 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 5314218895 (2534.02 GiB 2720.88 GB)
     Array Size : 5314218622 (2534.02 GiB 2720.88 GB)
  Used Dev Size : 5314218622 (2534.02 GiB 2720.88 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 83caa70a:6fe627f8:5a9a22d4:54a457f8

    Update Time : Tue Jul  9 23:08:37 2013
       Checksum : 7a729887 - correct
         Events : 3478472


   Device Role : spare
   Array State : A. ('A' == active, '.' == missing)

My hard drive was just replaced.

Personalities : [raid1]
md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

I didn't use a tool to recover data because I was pretty sure that the data on /dev/sdb was up to date until my server rebooted and my array broke, so I just copied the partition table from /dev/sdb to /dev/sda and rebuilt the array.

copy partitions
sgdisk -R /dev/sda /dev/sdb
mix ids
sgdisk -G /dev/sda
recreate array
--create /dev/md3 --level=1 --raid-devices=2 /dev/sdb4 missing
mdadm /dev/md3 -a /dev/sda3

Well, I hope this time the rebuild finishes.

4
  • That's not good. Can you add the output of sudo mdadm --examine /dev/sdb4 ? Jul 9, 2013 at 21:05
  • should i start an extended self test and see if it finds any errors or should i just wait and see if recovery works this time?
    – supernova
    Jul 9, 2013 at 21:13
  • Nah, /dev/sda is having uncorrectable issues. Time to replace the disk. See the edits to my post. If it fails at different sectors each time in the logs, you might try shutting the system down and replacing the SATA cable, but if that doesn't fix it then you need to replace the drive Jul 9, 2013 at 21:16
  • okay thanks, i will provide this logs to my hoster and will hopefully get a free replacement of the hdd.
    – supernova
    Jul 9, 2013 at 21:27

1 Answer 1

3

I would be hesitant to grow the array. You don't want a larger array, so that's the wrong operation. It may be a roundabout way to achieve the same thing, but I find it's a good philosophy to stick with the intended operators unless there is no other way.

Try:

sudo mdadm manage /dev/md3 --remove /dev/sdb4
sudo mdadm manage /dev/md3 --re-add /dev/sdb4

And watch dmesg for read/write errors on either /dev/sda or /dev/sdb while it rebuilds.


Looks like /dev/sda has bad sectors in /dev/sda4. You should replace the drive. If /dev/sdb is showing good on the S.M.A.R.T. status, The easiest way would be to

  • Get a new drive (I'm assuming it'll show up as /dev/sdc)
  • Repartition it exactly like /dev/sda
  • And one-by-one fail /dev/sdaX and replace with /dev/sdcX
  • Let the arrays rebuild from /dev/sdb for md0-md2

md3 is going to be special because mdadm doesn't see /dev/sdb4 as an array currently.

You can attempt to use gddrescue to recover /dev/sda4 to /dev/sdc4, and then after that is done try to assemble /dev/md3:

sudo mdadm --assemble /dev/md3 /dev/sdc4 /dev/sdb4

and see if it'll start. If it starts, fsck the filesystem to check for errors, and then remove/re-add sdb4 again to start the resync. You'll have some files which are bad / missing / damage, and will need to recover those from backups.

If you can't get a good copy of /dev/sda4 to /dev/sdc4, then you'll need to just create a new array from /dev/sdc4 and /dev/sdb4 and recover the entire contents from backup.

8
  • thanks i will give it a try, its starting recovery again i hope this works out and won't fail again - i will report back once it's finished.
    – supernova
    Jul 9, 2013 at 20:56
  • Yeah, if the recovery stops, I would expect an error to be in printed from dmesg as to why. Jul 9, 2013 at 20:58
  • hmmhmh i just went through demsg and spotted an read error: : [sda] Add. Sense: Unrecovered read error - auto reallocate failed ..
    – supernova
    Jul 9, 2013 at 21:00
  • Add it to the bottom of your question. That doesn't look good. Jul 9, 2013 at 21:01
  • i've added smartctl test results and error log of the hdd, whats wrong with this disk if smartctl doesn't seem to find any errors but there are clearly errors on the drive?
    – supernova
    Jul 9, 2013 at 21:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .