up vote 1 down vote favorite
share [g+] share [fb]

I have a software raid5 setup existing of 4 disks. sda, sdb, sdc, sdd Since everything was up and running I wanted to test the array. To do this I pulled the plug on one of the disks (sdd). Everything went fine so I tried re-adding the disk with mdadm --add /dev/md0 /dev/sdd, mdadm was rebuilding the array so after a couple of hours I checked and something went wrong, sdc failed.

Long story short: sdc has some bad sectors, when mdadm reads from these sectors it fails.

In /var/log/messages I can see:

read error not correctable (sector 753682864 on sdc).

I can not --assemble the array as it sees sdc and sdd as faulty.

I can however mdadm /dev/md0 --create --assume-clean -l5 -n4 /dev/sd[abc] missing. After this I tried to add sdd but without success.

When mdadm fails and it reports the sector in /var/log/messages I can verify this by executing hdparm --read-sector [sector] /dev/sdc. When I overwrite the sector using hdparm --write-sector it gets replaced and then it functions again. But this is a very time consuming process.

Are there any ways how I can recover from this mess? Would mirroring sdc to sdd (with dd) and then adding sdd as the missing drive help?

link|improve this question
How many bad sectors are there? If there are only a few, then fix them with hdparm. If there are many, then toss the drive and restore from backup. – psusi Feb 9 at 2:55
Like any good administrator I don't have a backup. I assumed raid 5 would provide enough redundancy. I don't know how many bad sectors there are but it looks like it's quite random. I'm going to replace the sata cable first, then try again. – siebz0r Feb 9 at 6:50
2  
RAID5 is not a substitute for backup, they serve completely different purposes. If you accidentally delete a file or a software bug issues a file delete or wipe command, the RAID 5 array will dutifully destroy the data on all drives. – David Schwartz Feb 9 at 10:08
I know, but raid 5 suits my needs – siebz0r Feb 9 at 12:56
If you don't have a backup, then your data is already gone. Lesson learned. – psusi Feb 9 at 16:04
show 1 more comment
feedback

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
or
required, but never shown
discard

Browse other questions tagged or ask your own question.