6

I'm trying to track down a data corruption issue that occurs on reads in Win7x64 and am out of ideas on what to try next.

This originally surfaced when migrating ~2TB of data to a new home NAS; when I validated the copy a small percentage of the files were corrupted. This was back in December and I've been chasing the problem off and on ever since.

Each "corruption" is a single bit is flipped from 0 to 1 or 1 to 0, in one or more locations in the affected files. When multiple files are affected in a single "run", the direction of the flip is identical every case.

It doesn't appear to be a RAM issue; my system doesn't support ECC RAM but I've performed multiple memory tests with no errors found. The system also runs for weeks on end without a reboot or any unexpected crashes (caveat because I expect Firefox to crash periodically) but I'm intentionally avoiding anything that involves moving large amounts of data around until I can resolve this. No CD rips, video transcodes, etc.

In every single case I have seen, the flipped bit is the low-order bit in the seventh byte of a sixteen-byte block of data. In most if not all of the cases I've seen, the byte with the flipped bit byte block is at offset …2x7 or …5X7. I only started getting really diligent about tracking the issue at that level after a month or so.

Here are the first few examples from a multi-GB video file that had 121 bit-flips:

                                     vv
000A46B2D0  1:  d2 04 29 dc d9 bf 15 01 f9 34 50 b6 08 11 63 d4
            2:  d2 04 29 dc d9 bf 15 00 f9 34 50 b6 08 11 63 d4
000EC6B2D0  1:  32 51 26 4f ae a0 42 29 93 5d 64 43 a6 e2 ee ba
            2:  32 51 26 4f ae a0 42 28 93 5d 64 43 a6 e2 ee ba
000F46B2D0  1:  e8 67 bd 18 08 00 62 59 21 37 94 00 d0 34 67 cf
            2:  e8 67 bd 18 08 00 62 58 21 37 94 00 d0 34 67 cf
0018C6B2D0  1:  b3 6e 0b 97 4e 7d 77 ab f9 74 38 6a 30 ee 9c 44
            2:  b3 6e 0b 97 4e 7d 77 aa f9 74 38 6a 30 ee 9c 44
001F46B2D0  1:  7e 87 0a c3 17 50 7c 55 39 b9 95 20 b8 6d 75 1a
            2:  7e 87 0a c3 17 50 7c 54 39 b9 95 20 b8 6d 75 1a
                                     ^^

The ten-digit hex value on the left is a byte position in the two files being compared and the sixteen hex pairs are the sixteen bytes in the file starting at that position. Note the value differences in the eighth byte of each hex-pair column. The correlation occurs at a higher level as well; note the fact that the first byte of each 16-byte row is at an offset ending in 46B2D0 or C6B2D0. This was true for all 121 of the errors in this particular file.

While this issue first manifested during large file copies I don't think it's a "copying" problem per se, but rather something occurs when the file is read, because I don't actually have to write the data back to see the problem:

  • In comparing two known-identical sets of files the first check pass will return errors, but when rechecking the specific files that showed problems they will test identical.
  • In one case I had a comparison where the bit-flip occurred in the SOURCE file of a known-identical pair, and re-reading that same file multiple times with different tools (Beyond Compare, md5sum, my own byte-level file comparison script that gave the output above) gave the same result, but after a reboot the files bit-compared as identical again. From this result, I'm assuming that all the re-reads were pulling the same bad data from cache.

It also doesn't appear to be tied to a specific storage medium or which application I use to copy the files or compare them afterwards:

  • I've seen the behavior on file copies and on read-only comparisons involving the consumer NAS, a USB3-connected backup drive and two different internal spinning-platter drives, in all possible combinations and directions. (All filesystems are NTFS.)
  • I've seen the behavior when validating copies made using drag-and-drop in Windows Explorer, Robocopy, Beyond Compare and Teracopy (the latter of which reported checksum mismatches during the verification phase), using comparison tools that include Beyond Compare (Hex View), my own byte-level comparison script, and comparisons to previously-generated md5sum values.

I've tried to determine if it's tied to the presence of any specific process on the system but given how long each comparison pass takes and how rare the error is that's been difficult. Having said that, there was at least one pass where I was getting a lot of comparison errors, and I killed a process that was consuming a lot of memory (CrashPlan backup service, which is an insane resource hog) after which the rest of the comparison pass was error-free. I'm able to reproduce the problem whether or not that specific service is running, though.

(Stuff gets a little more handwavey from this point forward in the question as I don't know the lay of the land very well.)

Since this looked like it might be related to the Windows file cache I checked consumption with Sysinternals CacheSet. Typical values when I see the issue are a Current cache size of ~1Gb and a Peak cache size of 1.5-1.7 GB. This seems reasonable for an 8GB system; I run a lot of RAM-greedy processes but the system never feels particularly memory-constrained. The working set range shows up as 1MB - 1TB(!!), though, which is definitely NOT reasonable. (I also used CacheSet to experimentally set the max working set size at 1Gb but the setting didn't survive a reboot -- not sure if that's expected or not.)

During the period mentioned above where an apparent bad read was sitting in the cache, I also used CacheSet to clear the cache working set, but subsequent reads of the "bad" file still showed the problem until I rebooted.

Size of the working set at the time of the bad read doesn't seem to matter. Though I've only started checking this recently, Sysinternals RamMap has shown Metafile Active/Total values as low as 200MB/250MB at the time of occurrence, which is way way under the maximum.

This could be the same behavior seen in this question, assuming Windows shows the current date for a corrupted XMP date field.

I think that covers everything I know about the problem so far.

4
  • 5
    This is interesting. Did you run the memory tests for long enough? I normally recommend a minimum of 10-15 passes in memtest86+, perhaps repeat testing over several days. That translates to say least 12 hours for 8 GB. I've seen errors be detected on the 13th or so pass.
    – Bob
    May 13, 2014 at 1:29
  • if the ram is on the edge of its speed and timings, copying data from to a hard drive can corrupt every once in a while tiny ammounts of data, although they would rarely be in the same location. At first I wondered if this was an archive bit or something specific put on specific files (media) like Ratings or alternate stream, being in the same location, and the compare missing it sometimes. If you know how to control your ram, you could reduce the rate it operates or adjust timings up , and test if it still happens. Super cool it , or raise the voltage a tiny bit and retest. (agrees with bob)
    – Psycogeek
    May 13, 2014 at 3:43
  • 1
    Bob: I've kept it to two or three passes previously, the logic being that this problem happens often enough to show up pretty quickly in a RAM test. But I'll do a more extensive run tonight. Psycogeek: Sounds plausible. I found and disabled a dummy-level "TURBO MODE" overclock setting in the BIOS yesterday that could theoretically have been causing a timing issue. Will see how it does without that. (Bad sign though: it BSOD'd overnight.) May 13, 2014 at 14:57
  • 1
    @LazloNibble you might also want to check the CPU with Prime95 (which verifies both the CPU and how it accesses memory). If it's related to timings/overclocking as the others have said already, try giving that component a slight voltage boost to see if that's what's causing it. May 13, 2014 at 23:25

1 Answer 1

2

@Bob: Looks like it was definitely a bad stick of memory, though it took almost 20 passes with memtest86+ to prove it. Lessons learned: 1) Validate all the hardware before diving down the software/config rabbit-hole, and 2) Never trust the built-in self-tests. Thanks to everyone for their help.

1
  • Just remember that memtest86+ actually relies on correct operation of the RAM, CPU, mobo and PSU. And rarely some other components in a memory-mapped space could interfere (e.g. PCI[e] cards). While failure is usually the RAM, you need to try testing with known good RAM next and make sure that passes.
    – Bob
    Jun 15, 2014 at 9:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .