I have RAID 5 setup with a 3ware 9650se controller in my HTPC. I recently lost power to the system, and upon boot up, the RAID 5 reported that it was degraded.
Using the 3ware RAID interface (the one that appears after the POST screen), I added the "missing" disk back to the RAID array, told the array to rebuild, and booted into Linux. I then left the system to rebuild, and headed off to work.
The rebuild process is currently at 47%, and won't be completed for another couple of hours, but from what I've read, the system should be usable, if a little slow, during the rebuild process. However, I now have over 500 media files on my RAID 5 drive that are zero bytes.
Does anyone have any suggestions / experience with zero byte files during a RAID 5 rebuild? Am I going to have to restore from backup, or will the data in my media files magically re-appear after the rebuild has finished?
System Specs:
- Ubuntu 11.04
- Gigabyte GA-MA785G-UD3H Motherboard
- 4GB RAM
- 3ware 9650se RAID controller
- 4x2TB SATA II hard drives
/etc/fstab contents:
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=XXXX / xfs defaults 0 1
/dev/sda3 /boot ext4 defaults 0 2
UUID=XXXX none swap sw 0 0
Output from tw_cli /c0 show
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-5 REBUILDING 47 - 256K 5587.9 ON ON
Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 DEGRADED u0 1.82 TB 3907029168 WD-XXXXXXXXXXXX
p1 OK u0 1.82 TB 3907029168 WD-XXXXXXXXXXXX
p2 OK u0 1.82 TB 3907029168 WD-XXXXXXXXXXXX
p3 OK u0 1.82 TB 3907029168 WD-XXXXXXXXXXXX
p4 NOT-PRESENT - - - -
p5 NOT-PRESENT - - - -
p6 NOT-PRESENT - - - -
p7 NOT-PRESENT - - - -
Update
Reading through the XFS FAQ, it looks like I should have disabled the write cache on the hard drives when I installed the RAID array, as "the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents in that case.".
I wasn't writing to all 500+ files when the power was turned off, so this isn't likely the root cause of the issue I'm seeing, but it looks like it' something I should have done previously.
Update 2
The RAID has finished rebuilding and verifying, and I still have 470 zero byte files. Some of the files re-appeared after the rebuild, but most did not.
In order to help prevent further issues, I've disabled write caching as recommended for the 3ware RAID controller in the XFS FAQ.
As for the actual cause of the issue, I'm not sure. All I know is that a chunk of my data's gone, and I'll need to resurrect what I can from backups.
