File copied over the network differs from original

Question

Having moved quite large amount of data (~1TB) over the network from one storage to another, I noticed the file on target system differs from the original.

Setup: PC (Windows 7 64) with Windows Sharing -> 1000BaseT network 2x 1G switch -> PC (Windows XP) as Windows Sharing client or NAS with Windows Sharing (probably Samba?) -> 1000BaseT network 1x 1G switch -> PC (Windows 7 64) as Windows Sharing client

Procedure: Copy from share with Total Commander - no error reported -> Synchronize dirs in Total Commanded (compare by content) - some files differ -> Total Commander's Diff (double click in Synchronize dirs output) - some of files marked as different do show difference, some of them are reported as being the same this time. I've tried PC-PC and PC-NAS and both is the same.

I have examined one of the files (~60GB one) and it seems the differences are always single byte having value 0 on original and 128 on target. They are randomly spread all over the file, about 10 of them. Re-running the diff shows some of them persist and some of them are changed, but there's about the same amount of them.

EDIT: To answer syneticon-dj's suspicion about TC, I have to note I have written a simple C# application which reads two files using .NET API and compares them byte by byte. This is how I got the diff info in previous paragraph.

It seems the network transfer fails one bit in every 6 gigabytes or so. How is it possible? Is it normal behaviour? How come it passes the checksums at TCP level? How can I tell what's wrong and what should be replaced?

EDIT: If network transfer is unlike to be the cause of errors, what could be the real cause?

Answer 1

I rather would suspect a bug in TCs "compare by content" feature and double-check using local checksum generation for the files on both sides using something like MD5 or SHA-1 hashes and comparing the checksums visually.

Some reasoning

An Ethernet frame has a CRC checksum of 32 bits per frame. TCP adds a 16-bit CRC checksum. A faulty packet with a single-bit-error passing both checksums would be less likely than for a single 32-bit CRC check (2^-32) but more likely than the product of both probabilities (2^-16 * 2^-32 = 2^-48). Where exactly, depends on the characteristics of the algorithms.

Assuming a payload size of 1.400 Bytes (that is if you are not using Jumbo Frames) and rounding it up to 2048 bytes for easier calculations it would roughly mean an error every 2^42 to 2^58 Bytes (5,4 Terabytes to 250 Petabytes) if every frame transmitted carries a single-byte error.

This, however, surely would get noticed otherwise - a rate of CRC checksum failures that high would virtually quench your Ethernet transmissions - you would get abysmally low transfer rates for your file copy. And you would be able to see a lot of CRC check failures in your managed switches' RMON port counters.

So, given the nature of your error it seems rather unlikely that it is a transmission error - these would also look considerably more random.

Answer 2

I would suspect bad memory on either of the machines. Please run memtest86 on both of them.

Answer 3

The IP checksum is a 16-bit CRC. You can expect roughly 1 in 65k corrupted packets to slip through. If you're doing large file transfers, you're probably better off with a dedicated file copier, using cryptographic checksums on smaller blocks. Like, say, running MD5 and SHA1 over 1 MB chunks, re-transmitting a chunk if one or both checksums differ.

The chunk size suggested (1 MB) is more gut feel than determined by solid engineering, if you're seeing an error roughly every 6 GB, that would lead to a total increase in traffic by roughly 0.1%.