6

For the first time since I've owned a PC (30 years) I have experienced an undetected, uncorrected single-bit disk error. In a RAID array. The sequence of events was

  1. Upload a collection of digital images (Camera Raw files) from a CF card
  2. Do some editing in Lightroom (which does not update the original file)
  3. Back up everything to an external archive disk (using Retrospect)
    ...time passes (about 1 week)...
  4. Open the file again in Lightroom -- it's corrupted (a big square magenta blotch)
  5. Restore a copy from the archive disk -- the restored copy is NOT corrupted
  6. Compare the two files. There is only a single bit difference... a byte that was originally 0x34 is now 0xB4

The online device is a pair of 2TB drives in RAID-1 on a hardware RAID card (3WARE 9560SE-4LPML).

Given the above sequence, the error was clearly introduced sometime after step 3 since the archived copy was not corrupted, so it couldn't have occurred during the original write. The file is a Canon CR2 raw file, and Lightroom never updates original RAW files, they are considered "digital negatives" and never updated. Instead it saves all edits as sidecar XMP files containing the sequence of edits applied. The file date/time are unmodified from the original.

Clearly somehow the bit error occurred and was propagated by the RAID hardware without producing a warning. I've checked the RAID error logs and there's nothing noteworthy for the last 18 months (since I last upgraded the software and firmware).

To summarize:

  1. The data was originally written correctly
  2. It was then read correctly when it was copied to the backup.
  3. Sometime after that the bit got flipped on the disk (since nothing rewrote the file).
  4. The RAID hardware is set to run a "verify" once a week. It did not detect the error.

That's just freaky. I would expect a miscompare error in the RAID hardware logs.

Also I can rule out a failing disk since the SMART data on both drives shows zero for all the applicable error attributes:

196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

and everything else is nominal as well.

Anyone have a scenario under which this would happen undetected?

  • Good thing you had a real backup not just a hardware mirror(backup). The Raid1 mirroring the data blindly does not seem to be anything odd at all, that is its job. Even if the one had a virus outbreak the mirror should to? That leaves how in heck did you get a single bit error while just sitting there parked on the disk platter? It happens more often that there is a ram error, because of how the ram is set, timed, and cooled and all, then that error is stored. – Psycogeek Aug 10 '14 at 4:39
  • You mentioned that the file was corrupt when you attempted to open it. You then implied that you read the file a second time, when you verified it against the back up. Did you power cycle your system between these two reads (i.e. are you certain that the corrupt data was read twice from the disks, rather than the original corrupt read being cached)? If not, just for grins, try shutting everything down, rebooting, and reading the corrupt file again. Does the error persist? – Jason C Aug 11 '14 at 0:28
  • @ExUmbris If it persists: One scenario I could imagine is perhaps a random RAM (or more likely on the RAID controller, based on your description) bit flip on write in step 1; but you were operating on a good cached copy in steps 2 and 3 despite a corrupt copy existing on disk. A week later when you accessed the data again, you of course re-read it from the disk, and ended up with the corrupt data that had been written originally. This makes many assumptions and relies on a bit of bad luck, but it's all I can really think of. – Jason C Aug 11 '14 at 0:33
  • @JasonC Hmmm.... that sounds like it might actually be possible. I still have both copies so I will try a full poweroff reboot to make sure the disk and RAID controller cache are cleared. – Ex Umbris Aug 11 '14 at 0:36
  • 2
    @JasonC You are correct. After a full power-off reboot the file is no longer corrupt, so the problem has to be in either the disk or RAID controller cache. It appears the file on disk is just fine. If you post this as an answer I'll accept it. – Ex Umbris Aug 11 '14 at 1:17
2

One possibility is a random bit flip in RAM or the controller on read in step 4. If data was corrupted on read then you would see it in step 4, then if it was still cached you'd also see it in step 6 when comparing files, since the corrupt cached data might still be used.

To test this case, power cycle all of your hardware to ensure the caches are cleared and try opening the file (and running the comparison with the backup) again. If all is well then this was the problem (there's no way to know at what stage of the read the bit flip occurred in so you'll just have to chalk it off as an unsolved mystery).

Failing this, a second, even unluckier possibility is perhaps a random RAM (or more likely on the RAID controller, based on your description) bit flip on write in step 1; but you were operating on a good cached copy in steps 2 and 3 despite a corrupt copy existing on disk. A week later when you accessed the data again, you of course re-read it from the disk, and ended up with the corrupt data that had been written originally. This makes many assumptions and relies on a bit of bad luck. If this is the case you'll just have to restore the backup file and move on.

Those are the only two things I can think of, really. It doesn't sound like an issue with the drives themselves. In any case since there's no way to tell where in the hardware the error occurred, I recommend running a full memory diagnostic just to be safe, although more likely the cause was unfortunate EMI or cosmic rays. As Canadian Luke mentioned in his answer, ECC RAM, if your motherboard supports it, will protect against this type of event, at least on the RAM side. It's actually not uncommon at all.


This case ended up being the OP's problem, rather than the second possibility.

  • 1
    Your original comment suggested the possible involvement of cache and poweroff reboot, after which the file was no longer corrupt. Therefore the problem was never on the disk copy of the data, only in a cache, maybe on the RAID controller, but I can't rule out an error in main memory either since a file block could stay cached for quite a while given that I have 12GB memory on this system. Thanks for pointing me in the right direction. I guess it's time to run memtest overnight :-) – Ex Umbris Aug 11 '14 at 3:08
  • Glad it worked out. 12GB is a decent amount of high density RAM, my hunch is an unlucky neutron at the right place and wrong time (an awkward guest at the transistor party); but memtest is a good idea. – Jason C Aug 11 '14 at 3:12
3

Check your ram. File systems like zfs recommend ecc memory (error checking and correcting memory) to prevent issues like this.

Your raid controller simply wrote the information it was given, and assumed it got correct information. It doesn't check that the information it receives is correct, just that it was written properly.

Your motherboard may not support ecc memory, but that should correct this issue. It's also possible that it was a solar flare that changed that byte on you, as the more ram you have, the greater the chance of an anomaly happening.

I'm out of town on my cell, but I can cite sources on Monday

  • 1
    Solar flares and zinc fibers are always getting into my data – Keltari Aug 10 '14 at 5:16
  • I've improved my original description. Remember that the error occurred on the disk at some point after the good backup was taken. A RAM error wouldn't fit with my actual scenario. – Ex Umbris Aug 11 '14 at 0:16
  • @Keltari An old IBM study estimates 1 bit error per month per 256MB RAM due solely to cosmic radiation (~1 bit error per day for 8GB RAM). Solar radiation is the most common cause of memory errors. – Jason C Aug 11 '14 at 0:46
  • @ExUmbris Actually, a RAM error would fit the description; it happened on one copy of the data, but not the other? The computer sees the data as what's written, and the RAID controller and hard drives report no issues, so those are very likely just writing whatever the RAM is telling it to write. Two different versions, two different devices? Yea, it's a memory error. – Canadian Luke Aug 11 '14 at 16:07

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.