Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have had in the recent weeks several cases of data corruption that occurred while copying files from one disk to another. Question is: what can be the cause, and how do I pinpoint it?

Some clues:

  • problems (9 cases) occurred on two different machines (one AMD 5050e with ECC RAM, the other some netbook), both running Win7x64 SP1 with no crash or other apparent problem;
  • problems occurred while copying large amount of data (total about 3TB) from one disk to another;
  • copy was with the standard GUI (Windows Explorer) that reported no error;
  • original file and copy have the same size and modification date;
  • data corruption was detected using MD5 hash (md5sum or/and Microsoft's FCIV), which was wrong on the copy (the MD5 of both original and copy is repeatable);
  • fc /B repeatably reports the differences, that always have been on contiguous blocks with exactly 4 kiB boundary (10 cases: one file was hit twice);
  • blocks in error are of varying size, from 4 kiB to 52 kiB, at seemingly random location, in large files (typically some GB);
  • corrupted blocks show no apparent relation to the original; in about half the cases, the corrupted data was all-zero;
  • all disks involved are NTFS, and given a clean bill of health by chkdsk /f (no bad block, no error reported);
  • the two affected destination disks are USB (the HD happens to be from the same manufacturer, but I can't say this is significant)
    • one is a 2.5" 2 TB housed in a self-powered USB 3 (Super-Speed, used in Hi-Speed) enclosure bearing the HD manufacturer's brand;
    • one is a 3.5" 1.5 TB in a Linux-based multimedia enclosure (PCH A-200) with USB 2 (Hi-Speed) slave port;
  • in more than half the cases the corruption was detected like an hour after the copy, with no disconnection or reboot involved; in most or all others the destination disks have been properly ejected;
  • I have no reason to suspect the various source disks (mostly SATA, some SSD).

Addition: I'm really concerned by finding the root cause and pinpointing the culprit(s), more than by working around the issue.

I reason that all the technologies involved are supposed to have a very low rate of undetected errors compared to reported ones (and I have no report of error). Therefore

  • if the error trigger was the magnetic media (an hypothesis that very well matches the observed 4kiB alignment, which I believe matches the internal physical sector size of the disks), it doubles with a disastrous bug somewhere preventing the error to be reported, as it would be (I know from experience) on a at least a read error in a SATA disk of my (different) favorite brand;
  • if the error trigger was poor electrical contact of USB cabling and undetected by CRC (as suggested by an answer); and given that the USB 2 maximum data packet size is 1kiB according to this source, not 4kiB as the alignment of all my errors; there must be some additional bug in the handling of errors (or a gaping hole in the USB specs or how they handle hard disks).
share|improve this question
    
This could well be a bug in the OS. That seems to be about the only common factor. (Though it's worth checking for some 3rd party software, especially anti-virus, that's common to both.) –  Daniel R Hicks Jan 8 at 22:45
    
@Daniel R Hicks: now that you mention it, Microsoft Security Essentials is running on both platforms with real-time protection on. However that would not, by itself, explain that corruption is seen only on USB targets. I have moved several time more data between SATA disks of the first machine without ever noticing corruption (I hope the operative word is not noticing). –  fgrieu Jan 9 at 6:52
    
It may be that MSE treats IDE drives specially. (The problem smells like a multi-threaded buffering issue -- almost certainly one thread reads into a buffer while the other writes from it, and very touchy synchronization is required between the two. There could simply be an inherent bug that timing provokes, or MSE, thrown in the middle, could muck it up.) –  Daniel R Hicks Jan 9 at 12:17

1 Answer 1

differences, that always have been on contiguous blocks with exactly 4 kiB boundary… blocks in error are of varying size, from 4 kiB to 52 kiB, at seemingly random location, in large files

Because drives write in chunks, you will usually see exactly the sort of block-sized errors you see with connection errors as opposed to drive errors.

I was going to ask if the drive is a flash-disk (which I have had the misfortune to experience silent, undetected corruption with), but then I saw this:

the two affected destinations disks are USB

This is another source of corruption I have unfortunately experienced. The problem is that a USB drive goes through a cable, so there can be problems with the electrical connection which can lead to corruption. Whenever you have this sort of problem, the first thing to do is to clean the pins on the connectors of the drive, the cable, and the USB port. You can try first breathing on it with some moist air from your lungs because the humidity helps conductivity. If that seems to have any effect, then you can brush the pins with a toothbrush or something (I usually use an emery board to lightly scrub them).

share|improve this answer
    
I understand that errors happen on both spinning media and cables, but in both cases these are supposed to be detected with very high likelihood. This works well at least for SATA disks of my (different) favorite brand: they have at occasions failed on me, but always reported it (in more other cases, I was warned by the sound of repeated re-seek). What I experience is like two decimal order of magnitudes above the quoted non-recoverable read errors rate (less than one in 10^14), and undetected read errors are supposed to be far rarer. –  fgrieu Jan 8 at 21:13
1  
Unfortunately that’s not the case. Only some errors are detected. If the drive has a bad sector or something, then you will get a CRC error during the copy, but if there is a transmission error, you will probably not notice anything. Like I said, I have experienced exactly what you described with external media. I got no errors at all, so when I (thankfully) did a comparison, I was surprised to find a few, random files were different and comparing those had randomly distributed blocks of differences. Cleaning the pins helped each time. –  Synetech Jan 8 at 21:19
    
If your explanation is the correct one, then at least one of USB (2 Hi-Speed), USB to SATA bridging, or Window's USB disk stack is a badly flawed technology that misses its design objectives entirely. Failing with error is bad, but unavoidable and acceptable. Failing silently at this high rate is not. I want to pinpoint and shame the culprit. –  fgrieu Jan 8 at 21:24
    
Tell me about it! When I noticed the error I was terrified because I wanted to use a removable drive to store all my personal files in a nice, portable way, but if files can be corrupted without any indication, then you may not notice it for some time and the corrupted version can even make it into a backup! ఠ_ఠ Thankfully I always make sure to compare files after copying, but you must make sure to flush the drive caches before doing so. –  Synetech Jan 8 at 21:48
    
The USB 2 maximum data packet size is 1kiB according to this source, not 4kiB as all my errors; thus errors undetected by CRC can not alone explain what I see, by any rational mechanism that I can think of; if that's the problem trigger (which I'm entirely ready to accept), in addition something must fail to properly report or handle an error that is actually detected, and should be reported by design. –  fgrieu Jan 8 at 21:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.