Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Having moved quite large amount of data (~1TB) over the network from one storage to another, I noticed the file on target system differs from the original.

Setup: PC (Windows 7 64) with Windows Sharing -> 1000BaseT network 2x 1G switch -> PC (Windows XP) as Windows Sharing client or NAS with Windows Sharing (probably Samba?) -> 1000BaseT network 1x 1G switch -> PC (Windows 7 64) as Windows Sharing client

Procedure: Copy from share with Total Commander - no error reported -> Synchronize dirs in Total Commanded (compare by content) - some files differ -> Total Commander's Diff (double click in Synchronize dirs output) - some of files marked as different do show difference, some of them are reported as being the same this time. I've tried PC-PC and PC-NAS and both is the same.

I have examined one of the files (~60GB one) and it seems the differences are always single byte having value 0 on original and 128 on target. They are randomly spread all over the file, about 10 of them. Re-running the diff shows some of them persist and some of them are changed, but there's about the same amount of them.

EDIT: To answer syneticon-dj's suspicion about TC, I have to note I have written a simple C# application which reads two files using .NET API and compares them byte by byte. This is how I got the diff info in previous paragraph.

It seems the network transfer fails one bit in every 6 gigabytes or so. How is it possible? Is it normal behaviour? How come it passes the checksums at TCP level? How can I tell what's wrong and what should be replaced?

EDIT: If network transfer is unlike to be the cause of errors, what could be the real cause?

share|improve this question
    
Looks like this is getting migrated to SU. But I can answer this much, it's absolutely not normal to lose a single bit of data let alone whole bytes. –  Chris S Dec 18 '11 at 15:07
    
I would consider it sufficiently network-related to stay here. I would not expect SU's folks to have much insight into protocol details or checksum algorithms either. –  syneticon-dj Dec 18 '11 at 15:56
    
Actually I don't think the inner workings of network protocols are my main interest, I'm more interested in solving the issue and be able to copy data without loses. So maybe it is really rather a user question and I have really posted to wrong forum. Am I able to move it, or has some admin to do it? –  Jan Svab Dec 18 '11 at 17:47
add comment

migrated from serverfault.com Dec 18 '11 at 18:23

This question came from our site for professional system and network administrators.

3 Answers

up vote 3 down vote accepted

I rather would suspect a bug in TCs "compare by content" feature and double-check using local checksum generation for the files on both sides using something like MD5 or SHA-1 hashes and comparing the checksums visually.

Some reasoning

An Ethernet frame has a CRC checksum of 32 bits per frame. TCP adds a 16-bit CRC checksum. A faulty packet with a single-bit-error passing both checksums would be less likely than for a single 32-bit CRC check (2^-32) but more likely than the product of both probabilities (2^-16 * 2^-32 = 2^-48). Where exactly, depends on the characteristics of the algorithms.

Assuming a payload size of 1.400 Bytes (that is if you are not using Jumbo Frames) and rounding it up to 2048 bytes for easier calculations it would roughly mean an error every 2^42 to 2^58 Bytes (5,4 Terabytes to 250 Petabytes) if every frame transmitted carries a single-byte error.

This, however, surely would get noticed otherwise - a rate of CRC checksum failures that high would virtually quench your Ethernet transmissions - you would get abysmally low transfer rates for your file copy. And you would be able to see a lot of CRC check failures in your managed switches' RMON port counters.

So, given the nature of your error it seems rather unlikely that it is a transmission error - these would also look considerably more random.

share|improve this answer
    
Thanks for explanation, it seems to be correct. However, this explains why it is NOT caused by wrong checksums, but doesn't explain any other possible causes or how to deal with situation. –  Jan Svab Dec 18 '11 at 17:41
    
I feel it is highly unlikely to be a bug in TC, data corruption during copy would be a bug which is hard to code :) –  Balázs Pozsár Dec 18 '11 at 17:52
    
Got back to this issue after a while and I have to confirm it was really a bug in TC's compare by content in my case. It fails on large files. I'm not sure why, but I think it is simply somthing it is not meant to be used for, perhaps some memory limit or something like that. If you use TC's file compare on two text files with very long lines, it marks all long lines as different (even the matching ones), probably for the same reason. –  Jan Svab Aug 6 '12 at 14:52
add comment

I would suspect bad memory on either of the machines. Please run memtest86 on both of them.

share|improve this answer
    
I don't think it is a possible reason. I can see the same problem on at least four machines (two desktops, laptop and a NAS) in the same network. None of them shows any other signs of faulty memory chip. However, I'll try the memtest, but I don't expect it to show anything. –  Jan Svab Dec 19 '11 at 8:45
    
please do so, i am really interested in the results because I cannot image it being a network problem –  Balázs Pozsár Dec 20 '11 at 13:30
add comment

The IP checksum is a 16-bit CRC. You can expect roughly 1 in 65k corrupted packets to slip through. If you're doing large file transfers, you're probably better off with a dedicated file copier, using cryptographic checksums on smaller blocks. Like, say, running MD5 and SHA1 over 1 MB chunks, re-transmitting a chunk if one or both checksums differ.

The chunk size suggested (1 MB) is more gut feel than determined by solid engineering, if you're seeing an error roughly every 6 GB, that would lead to a total increase in traffic by roughly 0.1%.

share|improve this answer
    
"copier using cryptographic checksums" -- such as, for example, one of the many applications based on rsync. –  David Cary Dec 18 '11 at 14:09
    
So the point is: "it is normal behavior and it can be expected". But how have you determined the number 1 of 65k corrupted packets? –  Jan Svab Dec 18 '11 at 14:10
    
rsync is a bitch for large files if differential compression is switched on. When using rsync on large files (roughly > 5GB), remember switching it off. –  syneticon-dj Dec 18 '11 at 14:47
    
16-bit CRC should catch all single-bit-per-packet errors, not merely all but 1 out of 2^16. –  David Cary Jan 24 '12 at 4:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.