I have a software raid5 setup existing of 4 disks. sda, sdb, sdc, sdd Since everything was up and running I wanted to test the array. To do this I pulled the plug on one of the disks (sdd). Everything went fine so I tried re-adding the disk with mdadm --add /dev/md0 /dev/sdd, mdadm was rebuilding the array so after a couple of hours I checked and something went wrong, sdc failed.

Long story short: sdc has some bad sectors, when mdadm reads from these sectors it fails.

In /var/log/messages I can see:

read error not correctable (sector 753682864 on sdc).

I can not --assemble the array as it sees sdc and sdd as faulty.

I can however mdadm /dev/md0 --create --assume-clean -l5 -n4 /dev/sd[abc] missing. After this I tried to add sdd but without success.

When mdadm fails and it reports the sector in /var/log/messages I can verify this by executing hdparm --read-sector [sector] /dev/sdc. When I overwrite the sector using hdparm --write-sector it gets replaced and then it functions again. But this is a very time consuming process.

Are there any ways how I can recover from this mess? Would mirroring sdc to sdd (with dd) and then adding sdd as the missing drive help?

  • How many bad sectors are there? If there are only a few, then fix them with hdparm. If there are many, then toss the drive and restore from backup. – psusi Feb 9 '12 at 2:55
  • Like any good administrator I don't have a backup. I assumed raid 5 would provide enough redundancy. I don't know how many bad sectors there are but it looks like it's quite random. I'm going to replace the sata cable first, then try again. – siebz0r Feb 9 '12 at 6:50
  • 3
    RAID5 is not a substitute for backup, they serve completely different purposes. If you accidentally delete a file or a software bug issues a file delete or wipe command, the RAID 5 array will dutifully destroy the data on all drives. – David Schwartz Feb 9 '12 at 10:08
  • I know, but raid 5 suits my needs – siebz0r Feb 9 '12 at 12:56
  • 1
    Re: "RAID is not backup because it won't help you with rm -rf". This is not an insurmountable problem: use LVM to create a snapshot alongside the live volume on the same md array. If you delete the wrong thing, or everything, or the filesystem gets corrupted somehow, just restore to the snapshot. Sure, things can go wrong there too but it's much less likely and there are risks of failure in ANY backup or redundancy strategy. It's all about managing risk. – Chris Smith Jul 17 '12 at 3:24

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.