I'm trying to track down a data corruption issue that occurs on reads in Win7x64 and am out of ideas on what to try next.
This originally surfaced when migrating ~2TB of data to a new home NAS; when I validated the copy a small percentage of the files were corrupted. This was back in December and I've been chasing the problem off and on ever since.
Each "corruption" is a single bit is flipped from 0 to 1 or 1 to 0, in one or more locations in the affected files. When multiple files are affected in a single "run", the direction of the flip is identical every case.
It doesn't appear to be a RAM issue; my system doesn't support ECC RAM but I've performed multiple memory tests with no errors found. The system also runs for weeks on end without a reboot or any unexpected crashes (caveat because I expect Firefox to crash periodically) but I'm intentionally avoiding anything that involves moving large amounts of data around until I can resolve this. No CD rips, video transcodes, etc.
In every single case I have seen, the flipped bit is the low-order bit in the seventh byte of a sixteen-byte block of data. In most if not all of the cases I've seen, the byte with the flipped bit byte block is at offset …2x7 or …5X7. I only started getting really diligent about tracking the issue at that level after a month or so.
Here are the first few examples from a multi-GB video file that had 121 bit-flips:
vv
000A46B2D0 1: d2 04 29 dc d9 bf 15 01 f9 34 50 b6 08 11 63 d4
2: d2 04 29 dc d9 bf 15 00 f9 34 50 b6 08 11 63 d4
000EC6B2D0 1: 32 51 26 4f ae a0 42 29 93 5d 64 43 a6 e2 ee ba
2: 32 51 26 4f ae a0 42 28 93 5d 64 43 a6 e2 ee ba
000F46B2D0 1: e8 67 bd 18 08 00 62 59 21 37 94 00 d0 34 67 cf
2: e8 67 bd 18 08 00 62 58 21 37 94 00 d0 34 67 cf
0018C6B2D0 1: b3 6e 0b 97 4e 7d 77 ab f9 74 38 6a 30 ee 9c 44
2: b3 6e 0b 97 4e 7d 77 aa f9 74 38 6a 30 ee 9c 44
001F46B2D0 1: 7e 87 0a c3 17 50 7c 55 39 b9 95 20 b8 6d 75 1a
2: 7e 87 0a c3 17 50 7c 54 39 b9 95 20 b8 6d 75 1a
^^
The ten-digit hex value on the left is a byte position in the two files being compared and the sixteen hex pairs are the sixteen bytes in the file starting at that position. Note the value differences in the eighth byte of each hex-pair column. The correlation occurs at a higher level as well; note the fact that the first byte of each 16-byte row is at an offset ending in 46B2D0 or C6B2D0. This was true for all 121 of the errors in this particular file.
While this issue first manifested during large file copies I don't think it's a "copying" problem per se, but rather something occurs when the file is read, because I don't actually have to write the data back to see the problem:
- In comparing two known-identical sets of files the first check pass will return errors, but when rechecking the specific files that showed problems they will test identical.
- In one case I had a comparison where the bit-flip occurred in the SOURCE file of a known-identical pair, and re-reading that same file multiple times with different tools (Beyond Compare, md5sum, my own byte-level file comparison script that gave the output above) gave the same result, but after a reboot the files bit-compared as identical again. From this result, I'm assuming that all the re-reads were pulling the same bad data from cache.
It also doesn't appear to be tied to a specific storage medium or which application I use to copy the files or compare them afterwards:
- I've seen the behavior on file copies and on read-only comparisons involving the consumer NAS, a USB3-connected backup drive and two different internal spinning-platter drives, in all possible combinations and directions. (All filesystems are NTFS.)
- I've seen the behavior when validating copies made using drag-and-drop in Windows Explorer, Robocopy, Beyond Compare and Teracopy (the latter of which reported checksum mismatches during the verification phase), using comparison tools that include Beyond Compare (Hex View), my own byte-level comparison script, and comparisons to previously-generated md5sum values.
I've tried to determine if it's tied to the presence of any specific process on the system but given how long each comparison pass takes and how rare the error is that's been difficult. Having said that, there was at least one pass where I was getting a lot of comparison errors, and I killed a process that was consuming a lot of memory (CrashPlan backup service, which is an insane resource hog) after which the rest of the comparison pass was error-free. I'm able to reproduce the problem whether or not that specific service is running, though.
(Stuff gets a little more handwavey from this point forward in the question as I don't know the lay of the land very well.)
Since this looked like it might be related to the Windows file cache I checked consumption with Sysinternals CacheSet. Typical values when I see the issue are a Current cache size of ~1Gb and a Peak cache size of 1.5-1.7 GB. This seems reasonable for an 8GB system; I run a lot of RAM-greedy processes but the system never feels particularly memory-constrained. The working set range shows up as 1MB - 1TB(!!), though, which is definitely NOT reasonable. (I also used CacheSet to experimentally set the max working set size at 1Gb but the setting didn't survive a reboot -- not sure if that's expected or not.)
During the period mentioned above where an apparent bad read was sitting in the cache, I also used CacheSet to clear the cache working set, but subsequent reads of the "bad" file still showed the problem until I rebooted.
Size of the working set at the time of the bad read doesn't seem to matter. Though I've only started checking this recently, Sysinternals RamMap has shown Metafile Active/Total values as low as 200MB/250MB at the time of occurrence, which is way way under the maximum.
This could be the same behavior seen in this question, assuming Windows shows the current date for a corrupted XMP date field.
I think that covers everything I know about the problem so far.