25

I frequently transfer VM images from hypervisors to an archive server for long term storage.

I transfer using netcat since it is faster than scp, rsync, ect..

hypervisor$ cat foo.box | nc <archive IP> 1234

archive$ nc -l -p 1234 > foo.box

When the file has finished transferring, I verify there was no corruption by running md5sum on both the target and source.

Unfortunately, running a md5sum on a large file can take a very long time. How can I more quickly compare the integrity of two large files?

Update:

  • My transmission rarely gets interrupted so restart-ability is not an issue.
  • It generally takes 3-4 hours to transfer via NC and then 40 minutes to get the md5sum.
  • The security of the hash is not an issue in this case.
  • 2
    You could try different checksum: en.wikipedia.org/wiki/Checksum. I don't know about their performance though – tumchaaditya Jan 8 '14 at 23:49
  • How long does the actual transfer take, and how long does the md5sum take? – Keith Thompson Jan 9 '14 at 0:31
  • The transfer generally takes between 3-4 hours, and the md5sums take about 40minutes to compute. – tbenz9 Jan 9 '14 at 0:56
18

You can use tee to do the sum on the fly with something like this (adapt the netcat commands for your needs):

Server:

netcat -l -w 2 1111 | tee >( md5sum > /dev/stderr )

Client:

tee >( md5sum > /dev/stderr ) | netcat 127.0.0.1 1111
  • 1
    Just a thought: md5deep has a "chunk" mode (md5deep.sourceforge.net/md5deep.html) that may be useful for this. – LawrenceC Jan 15 '14 at 22:10
  • @ultrasawblade - That's awesome link, I will have to check that out for other purposes. Thanks for mentioning it! – nerdwaller Jan 15 '14 at 22:55
10

Nerdwaller's answer about using tee to simultaneously transfer and calculate a checksum is a good approach if you're primarily worried about corruption over the network. It won't protect you against corruption on the way to disk, etc., though, as its taking the checksum before it hits disk.

But I'd like to add something:

1 TiB / 40 minutes ≈ 437 MiB/sec1.

That's pretty fast, actually. Remember that unless you have a lot of RAM, that's got to come back from storage. So the first thing to check is to watch iostat -kx 10 as you run your checksums; in particular you want to pay attention to the %util column. If you're pegging the disks (near 100%), then the answer is to buy faster storage.

Otherwise, as other posters mentioned, you can try different checksum algorithms. MD4, MD5, and SHA-1 are all designed to be cryptographic hashes (though none of those should be used for that purpose anymore; all are considered too weak). Speed wise, you can compare them with openssl speed md4 md5 sha1 sha256. I've thrown in SHA256 to have at least one still strong enough hash.

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md4              61716.74k   195224.79k   455472.73k   695089.49k   820035.58k
md5              46317.99k   140508.39k   320853.42k   473215.66k   539563.35k
sha1             43397.21k   126598.91k   283775.15k   392279.04k   473153.54k
sha256           33677.99k    75638.81k   128904.87k   155874.91k   167774.89k

Of the above, you can see that MD4 is the fastest, and SHA256 the slowest. This result is typical on PC-like hardware, at least.

If you want even more performance (at the cost of being trivial to tamper with, and also less likely to detect corruption), you want to look at a CRC or Adler hash. Of the two, Adler is typically faster, but weaker. Unfortunately, I'm not aware of any really fast command line implementations; the programs on my system are all slower than OpenSSL's md4.

So, your best bet speed-wise is openssl md4 -r (the -r makes it look like md5sum output).

If you're willing to do some compiling and/or minimal programming, see Mark Adler's code over on Stack Overflow and also xxhash. If you have SSE 4.2, you will not be able to beat the speed of the hardware CRC instruction.


1 1 TiB = 1024⁴ bytes; 1 MiB = 1024² bytes. Comes to ≈417MB/sec with powers-of-1000 units.

  • It is fast, I'm copying from one big RAID array to a 2nd large RAID array. – tbenz9 Jan 15 '14 at 21:33
  • @tbenz9 I figured, no way that's a single disk! I've added some pointers to some really fast hashes, which unfortunately will require at least compiling them... But they will surely run as fast as your disks (or even your RAM) can provide the data. (And if you're wondering about Mark Adler v. Adler32, yes, that appears to be Adler32's creator) – derobert Jan 15 '14 at 21:54
  • @derobert, Instead of using small files to test, shouldn't you have tested it with a big file like 1TB? – Pacerier Mar 18 '18 at 14:46
  • @derobert, Why don't you use shasum instead? – Pacerier Mar 18 '18 at 14:54
  • @Pacerier that's the output from OpenSSL's built-in benchmark. No doubt with longer blocks, it'll be a bit faster, but the ranking is unlikely to change (it was consistent across all the sizes it did test). Does shasum have a faster implementation than OpenSSL? Though honestly nowadays if you want a fast cryptographic hash , you'd use BLAKE2. – derobert Mar 18 '18 at 15:24
9

The openssl command supports several message digests. Of the ones I was able to try, md4 seems to run in about 65% of the time of md5, and about 54% of the time of sha1 (for the one file I tested with).

There's also an md2 in the documentation, but it seems to give the same results as md5.

Very roughly, speed seems to be inversely related to quality, but since you're (probably) not concerned about an adversary creating a deliberate collision, that shouldn't be much of an issue.

You might look around for older and simpler message digests (was there an md1, for example)?

A minor point: You've got a Useless Use of cat. Rather than:

cat foo.box | nc <archive IP> 1234

you can use:

nc <archive IP> 1234 < foo.box

or even:

< foo.box nc <archive IP> 1234

Doing so saves a process, but probably won't have any significant effect on performance.

  • 1
    Thanks for the tip on cat, not related to the question but a helpful tip nonetheless. Cheers! – tbenz9 Jan 9 '14 at 19:55
  • @tbenz9: readable code is easier to debug and maintain and change. "Useless cat" therefore is not necessarily entirely bad. If there's no performance gain by avoiding it, then it's better to go with whatever you're more comfortable with, assuming you will be the maintainer of this code. – iconoclast Jan 7 '16 at 3:30
  • 1
    @Keith, Link down.. – Pacerier Mar 18 '18 at 14:47
4

Two options:

Use sha1sum

sha1sum foo.box

In some circumstances sha1sum is faster.


Use rsync

It will take longer to transfer, but rsync verifies that the file arrived intact.

From the rsync man page

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred...

  • 1
    Thanks for the tip on sha1sum, rsync takes upwards of 10+ hours to transfer, I can transfer the same file and run the md5sums in about 4 hours using nc and md5sum. I'm trying to get my 4 hours even lower. – tbenz9 Jan 9 '14 at 0:10
3

Science is progressing. It appears that the new BLAKE2 hash function is faster than MD5 (and cryptographically much stronger to boot).

Reference: https://leastauthority.com/blog/BLAKE2-harder-better-faster-stronger-than-MD5.html

From Zooko's slides:

cycles per byte on Intel Core i5-3210M (Ivy Bridge)
function cycles per byte
long msg 4096 B 64 B MD5 5.0 5.2 13.1 SHA1 4.7 4.8 13.7 SHA256 12.8 13.0 30.0 Keccak 8.2 8.5 26.0 BLAKE1 5.8 6.0 14.9 BLAKE2 3.5 3.5 9.3
2

You probably can’t do any better than a good hash.  You might want to check out other hash/checksum functions to see whether any are significantly faster than md5sum.  Note that you might not need something as strong as MD5.  MD5 (and things like SHA1) are designed to be cryptographically strong, so it is infeasible for an attacker/imposter to craft a new file that has the same hash value as an existing value (i.e., to make it hard to tamper with signed e-mails and other documents).  If you’re not concerned about an attack on your communications, but only a run-of-the-mill comms error, something like a cyclic redundancy check (CRC) might be good enough.  (But I don’t know whether it would be any faster.)

Another approach is to try to do the hash in parallel with the transfer.  This might reduce the overall time, and could definitely reduce the irritation factor of needing to wait for the transfer to finish, and then wait again for the MD5 to finish.  I haven’t tested this, but it should be possible to do something like this:

  • On the source machine:

    mkfifo myfifo
    tee myfifo < source_file | nc dest_host port_number & md5sum myfifo
    
  • On the destination machine:

    mkfifo myfifo
    nc -l -p port_number | tee myfifo > dest_file &       md5sum myfifo
    

Of course checking the sizes of the files is a good, quick way to detect if any bytes got dropped.

2

Sending huge files is a pain. Why not try chunking up the files generating a hash for each chunk and then send it over to the destination and then check hash and join up the chunks.

You could also set up a personal BitTorrent network. That would ensure that the whole thing reaches safely.

  • My understanding is since it is one source and one destination a BitTorrent network would not be beneficial. Doesnt that only benefit when its going to lots of destinations from many sources? – tbenz9 Jan 9 '14 at 19:56
  • I considered suggesting this approach (breaking the input file into chunks, sending them separately, and reassembling them on the other end) and I couldn’t figure out how to make it even performance neutral, let alone an improvement. You still have the same amount of network transfer time, but you have a lot more overhead on each end. This essentially entails copying the file from the source machine to the source machine, then copying it to the destination machine, & then copying it from the destination machine to the destination machine. Even with big RAM disks, this is not free. – Scott Jan 9 '14 at 23:03
  • 1
    The only benefit of this approach is restartability, including quicker recovery from a transmission failure. The OP didn’t say how often he gets failures, and didn’t indicate that this was something he wanted optimized. – Scott Jan 9 '14 at 23:04
  • @tben9 Bittorrent is the current tool of choice for single transfer of files. Having the hash information with the file means that the end client can verify the data downloaded and fix it if need be. The multiple sources is for speed. So, yes, in this case it is beneficial to use BT to ensure a file is correctly transferred. – Underverse May 24 '15 at 13:36

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.