For the first time since I've owned a PC (30 years) I have experienced an undetected, uncorrected single-bit disk error. In a RAID array. The sequence of events was
- Upload a collection of digital images (Camera Raw files) from a CF card
- Do some editing in Lightroom (which does not update the original file)
- Back up everything to an external archive disk (using Retrospect)
...time passes (about 1 week)...
- Open the file again in Lightroom -- it's corrupted (a big square magenta blotch)
- Restore a copy from the archive disk -- the restored copy is NOT corrupted
- Compare the two files. There is only a single bit difference... a byte that was originally 0x34 is now 0xB4
The online device is a pair of 2TB drives in RAID-1 on a hardware RAID card (3WARE 9560SE-4LPML).
Given the above sequence, the error was clearly introduced sometime after step 3 since the archived copy was not corrupted, so it couldn't have occurred during the original write. The file is a Canon CR2 raw file, and Lightroom never updates original RAW files, they are considered "digital negatives" and never updated. Instead it saves all edits as sidecar XMP files containing the sequence of edits applied. The file date/time are unmodified from the original.
Clearly somehow the bit error occurred and was propagated by the RAID hardware without producing a warning. I've checked the RAID error logs and there's nothing noteworthy for the last 18 months (since I last upgraded the software and firmware).
- The data was originally written correctly
- It was then read correctly when it was copied to the backup.
- Sometime after that the bit got flipped on the disk (since nothing rewrote the file).
- The RAID hardware is set to run a "verify" once a week. It did not detect the error.
That's just freaky. I would expect a miscompare error in the RAID hardware logs.
Also I can rule out a failing disk since the SMART data on both drives shows zero for all the applicable error attributes:
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
and everything else is nominal as well.
Anyone have a scenario under which this would happen undetected?