Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Yesterday my hoster changed a SATA cable of one of my hdds, when my server came back up again cat /proc/mdstat showed this

Personalities : [raid1]
md124 : active raid1 sda1[0]
      4193268 blocks super 1.2 [2/1] [U_]

md125 : active (auto-read-only) raid1 sda2[0]
      524276 blocks super 1.2 [2/1] [U_]

md126 : active (auto-read-only) raid1 sda3[0]
      268434296 blocks super 1.2 [2/1] [U_]

md127 : active raid1 sda4[0]
      2657109311 blocks super 1.2 [2/1] [U_]

md3 : active (auto-read-only) raid1 sdb4[1]
      2657109311 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

i booted into rescue console seeing md124 ,md125,.. arrays are gone.

md3 : active (auto-read-only) raid1 sdb4[1]
      2657109311 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

so i added missing drives to the array

mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3
mdadm /dev/md3 -a /dev/sda4

and it started to recover. once complete I rebooted into normal system and it started to recover again.

but this time sdb is marked as missing?

Personalities : [raid1]
md3 : active raid1 sda4[2] sdb4[3]
      2657109311 blocks super 1.2 [2/1] [U_]
      [===>.................]  recovery = 17.1% (456317824/2657109311) finish=288.2min speed=127254K/sec

and recovery stopped after 3 hours and now the drive is marked as spare:

md3 : active raid1 sda4[2] sdb4[3](S)
      2657109311 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[2] sdb3[1]
      268434296 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[2] sdb2[1]
      524276 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[2] sdb1[1]
      4193268 blocks super 1.2 [2/2] [UU]

so far I've lost no data, I checked my own email account and every mail I've got before the server was shut down is still there, where the harddrive was failing 3 days ago.

HOW can I add the spare disk to my raid again?

I found another question/answer which looks similar to my problem here - would this be save to do or could I suffer any data loss?

mdadm --grow /dev/md3 --raid-devices=3
mdadm /dev/md3 --fail /dev/{failed drive}
mdadm /dev/md3 --remove /dev/{failed drive}
mdadm --grow /dev/md3 --raid-devices=2

of course I have backups but if I can avoid to use them I would like to do so :-)


i just spotted an read error in demsg which was probably from before when it failed and went into spare

[17699.328298] ata1.00: irq_stat 0x40000008
[17699.328324] ata1.00: failed command: READ FPDMA QUEUED
[17699.328356] ata1.00: cmd 60/08:00:80:d8:05/00:00:ff:00:00/40 tag 0 ncq 4096 in
[17699.328358]          res 51/40:08:80:d8:05/00:00:ff:00:00/40 Emask 0x409 (media error) <F>
[17699.328446] ata1.00: status: { DRDY ERR }
[17699.328471] ata1.00: error: { UNC }
[17699.332240] ata1.00: configured for UDMA/133
[17699.332281] sd 0:0:0:0: [sda] Unhandled sense code
[17699.332308] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[17699.332342] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[17699.332384] Descriptor sense data with sense descriptors (in hex):
[17699.332415]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[17699.332491]         ff 05 d8 80
[17699.332528] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[17699.332581] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 ff 05 d8 80 00 00 08 00
[17699.332648] end_request: I/O error, dev sda, sector 4278573184
[17699.332689] ata1: EH complete
[17699.332737] raid1: sda: unrecoverable I/O read error for block 3732258944
[17699.377132] md: md3: recovery done.

i tested the drive before with smartctl

smartctl -l selftest /dev/sda
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3444         -
[code]

[code]
smartctl -l selftest /dev/sdb
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3444   

but munin shows a smartctl exit code of 64 and smartctl -l error /dev/sda shows

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 552 (device log contains only the most recent five errors)
......
Error 552 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:

  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 80 d8 05 0f

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 00 80 d8 05 40 00      20:56:57.342  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00      20:56:57.342  SET FEATURES [Reserved for Serial ATA]
  27 00 00 00 00 00 e0 00      20:56:57.342  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 00      20:56:57.340  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      20:56:57.340  SET FEATURES [Set transfer mode]


Error 551 occurred at disk power-on lifetime: 3444 hours (143 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.
....

edit(2)

mdadm --examine /dev/sdb4
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 38dec3bf:770fb774:6e9a28d0:ff3eac4a
           Name : rescue:3
  Creation Time : Tue Feb 26 21:21:56 2013
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 5314218895 (2534.02 GiB 2720.88 GB)
     Array Size : 5314218622 (2534.02 GiB 2720.88 GB)
  Used Dev Size : 5314218622 (2534.02 GiB 2720.88 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 83caa70a:6fe627f8:5a9a22d4:54a457f8

    Update Time : Tue Jul  9 23:08:37 2013
       Checksum : 7a729887 - correct
         Events : 3478472


   Device Role : spare
   Array State : A. ('A' == active, '.' == missing)

my harddrive was just replaced.

Personalities : [raid1]
md2 : active raid1 sdb3[1]
      268434296 blocks super 1.2 [2/1] [_U]

md1 : active raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      4193268 blocks super 1.2 [2/1] [_U]

i didnt use a tool to recover data because i was pretty sure that the data on sdb was up to date UNTIL my server rebooted and my array broke, so i just copied the partition table from sdb to sda and rebuilt the array.

copy partitions
sgdisk -R /dev/sda /dev/sdb
mix ids
sgdisk -G /dev/sda
recreate array
--create /dev/md3 --level=1 --raid-devices=2 /dev/sdb4 missing
mdadm /dev/md3 -a /dev/sda3

well i hope this time the rebuild finishes.

share|improve this question
    
That's not good. Can you add the output of sudo mdadm --examine /dev/sdb4 ? –  Darth Android Jul 9 '13 at 21:05
    
should i start an extended self test and see if it finds any errors or should i just wait and see if recovery works this time? –  supernova Jul 9 '13 at 21:13
    
Nah, /dev/sda is having uncorrectable issues. Time to replace the disk. See the edits to my post. If it fails at different sectors each time in the logs, you might try shutting the system down and replacing the SATA cable, but if that doesn't fix it then you need to replace the drive –  Darth Android Jul 9 '13 at 21:16
    
okay thanks, i will provide this logs to my hoster and will hopefully get a free replacement of the hdd. –  supernova Jul 9 '13 at 21:27
add comment

1 Answer

up vote 3 down vote accepted

I would be hesitant to grow the array. You don't want a larger array, so that's the wrong operation. It may be a roundabout way to achieve the same thing, but I find it's a good philosophy to stick with the intended operators unless there is no other way.

Try:

sudo mdadm manage /dev/md3 --remove /dev/sdb4
sudo mdadm manage /dev/md3 --re-add /dev/sdb4

And watch dmesg for read/write errors on either /dev/sda or /dev/sdb while it rebuilds.


Looks like /dev/sda has bad sectors in /dev/sda4. You should replace the drive. If /dev/sdb is showing good on the S.M.A.R.T. status, The easiest way would be to

  • Get a new drive (I'm assuming it'll show up as /dev/sdc)
  • Repartition it exactly like /dev/sda
  • And one-by-one fail /dev/sdaX and replace with /dev/sdcX
  • Let the arrays rebuild from /dev/sdb for md0-md2

md3 is going to be special because mdadm doesn't see /dev/sdb4 as an array currently.

You can attempt to use gddrescue to recover /dev/sda4 to /dev/sdc4, and then after that is done try to assemble /dev/md3:

sudo mdadm --assemble /dev/md3 /dev/sdc4 /dev/sdb4

and see if it'll start. If it starts, fsck the filesystem to check for errors, and then remove/re-add sdb4 again to start the resync. You'll have some files which are bad / missing / damage, and will need to recover those from backups.

If you can't get a good copy of /dev/sda4 to /dev/sdc4, then you'll need to just create a new array from /dev/sdc4 and /dev/sdb4 and recover the entire contents from backup.

share|improve this answer
    
thanks i will give it a try, its starting recovery again i hope this works out and won't fail again - i will report back once it's finished. –  supernova Jul 9 '13 at 20:56
    
Yeah, if the recovery stops, I would expect an error to be in printed from dmesg as to why. –  Darth Android Jul 9 '13 at 20:58
    
hmmhmh i just went through demsg and spotted an read error: : [sda] Add. Sense: Unrecovered read error - auto reallocate failed .. –  supernova Jul 9 '13 at 21:00
    
Add it to the bottom of your question. That doesn't look good. –  Darth Android Jul 9 '13 at 21:01
    
i've added smartctl test results and error log of the hdd, whats wrong with this disk if smartctl doesn't seem to find any errors but there are clearly errors on the drive? –  supernova Jul 9 '13 at 21:05
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.