I had created two 2TB HDD partitions (
/dev/sdc1) in a RAID 1 array called
mdadm on Ubuntu 12.04 LTS Precise Pangolin.
sudo mdadm --detail /dev/md0 used to indicate both drives as active sync.
Then, for testing, I failed
/dev/sdb1, removed it, then added it again with the command
sudo mdadm /dev/md0 --add /dev/sdb1
watch cat /proc/mdstat showed a progress bar of the array rebuilding, but I wouldn't spend hours watching it, so I assumed that the software knew what it was doing.
After the progress bar was no longer showing,
cat /proc/mdstat displays:
md0 : active raid1 sdb1(S) sdc1 1953511288 blocks super 1.2 [2/1] [U_]
sudo mdadm --detail /dev/md0 shows:
/dev/md0: Version : 1.2 Creation Time : Sun May 27 11:26:05 2012 Raid Level : raid1 Array Size : 1953511288 (1863.01 GiB 2000.40 GB) Used Dev Size : 1953511288 (1863.01 GiB 2000.40 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Mon May 28 11:16:49 2012 State : clean, degraded Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Name : Deltique:0 (local to host Deltique) UUID : 49733c26:dd5f67b5:13741fb7:c568bd04 Events : 32365 Number Major Minor RaidDevice State 1 8 33 0 active sync /dev/sdc1 1 0 0 1 removed 2 8 17 - spare /dev/sdb1
I've been told that mdadm automatically replaces removed drives with spares, but
/dev/sdb1 isn't being moved into the expected position, RaidDevice
UPDATE (30 May 2012): A
badblocks destructive read-write test of the entire
/dev/sdb yielded no errors as expected; both HDDs are new.
As of the latest edit, I assembled the array with this command:
sudo mdadm --assemble --force --no-degraded /dev/md0 /dev/sdb1 /dev/sdc1
The output was:
mdadm: /dev/md0 has been started with 1 drive (out of 2) and 1 rebuilding.
Rebuilding looks like it's progressing normally:
md0 : active raid1 sdc1 sdb1 1953511288 blocks super 1.2 [2/1] [U_] [>....................] recovery = 0.6% (13261504/1953511288) finish=2299.7min speed=14060K/sec unused devices: <none>
I'm now waiting on this rebuild, but I'm expecting
/dev/sdb1 to become a spare just like the five or six times that I've tried rebuilding before.
UPDATE (31 May 2012): Yeah, it's still a spare. Ugh!
UPDATE (01 June 2012): I'm trying Adrian Kelly's suggested command:
sudo mdadm --assemble --update=resync /dev/md0 /dev/sdb1 /dev/sdc1
Waiting on the rebuild now...
UPDATE (02 June 2012): Nope, still a spare...
UPDATE (04 June 2012): P.B. brought up a concern that I overlooked: perhaps
/dev/sdc1 is encountering I/O errors. I hadn't bothered to check
/dev/sdc1 because it appeared to be working just fine and it was brand new, but I/O errors towards the end of the drive is a rational possibility.
I bought these HDDs on sale, so it would be no surprise that one of them is already failing. Plus, neither of them have support for S.M.A.R.T., so no wonder they were so cheap...
Here is the data recovery procedure I just made up and am following:
sudo mdadm /dev/md0 --fail /dev/sdb1so that I can take out
sudo mdadm /dev/md0 --remove /dev/sdb1to remove
/dev/sdb1from the array.
/dev/sdc1is mounted at
cd /mediato work in that area.
sudo chown deltik DtkBkTempto give me (username
deltik) rights to the partition.
- Do copy of all files and directories:
sudo rsync -avzHXShP DtkBk/* DtkBkTemp
UPDATE (06 June 2012): I did a
badblocks destructive write-mode test of
/dev/sdc, following the following procedures:
sudo umount /media/DtkBkto allow tearing down of the array.
sudo mdadm --stop /dev/md0to stop the array.
sudo badblocks -w -p 1 /dev/sdc -s -vto wipe the suspect hard drive, and in the process, check for I/O errors. If there are I/O errors, that is not a good sign. Hopefully, I can get a refund...
I have now confirmed that there are no input/output issues on either HDD.
From all this investigating, my two original questions still stand.
My questions are:
- Why isn't the spare drive becoming active sync?
- How can I make the spare drive become active?