Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I tested the speed of md5sum on a few Ubuntu 8.04 servers

  • Pentium III 700 MHz: 52 MB/s
  • Atom 1.6 GHz, 32-bit: 119 MB/s
  • Core 2 (Yorkfield) 2.5GHz, 32-bit: 194 MB/s
  • Core 2 (Yorkfield) 2.5GHz, 64-bit: 222 MB/s

Then I downloaded a tool (by apt-get install) called md5deep and found that it's roughly 20% faster (as tested on the 32-bit Core 2 server). This makes me feel that the "vanilla" md5sum included in Ubuntu isn't the fastest one.

Questions:

  1. Other than md5deep, are you aware of any MD5 calculators that are potentially faster than md5sum? (Answers for software from other OS are also welcomed.)
  2. If I want to compile md5sum myself, what compiler options would you suggest for the Core 2 server? (note: gcc 4.2.4 in Ubuntu 8.04 does not seem to support -march=core2)

Edit: I downloaded and compiled coreutils-8.4 on the 64-bit Core 2 server and the speed increased to 320 MB/s... Unbelievable!

FYI, the one included in Ubuntu 8.04 is coreutils-6.10.

Is it safe to replace coreutils with a newer version?

share|improve this question
    
Usually md5sum is faster than my disk...eg I don't have a HDD that can read at 200MB/sec, so a faster than that md5sum doesn't help any. Do you have a really fast SSD RAID array? Or are you running md5sum on files stored on a ramdisk (tmpfs) or something? – davr Apr 1 '10 at 22:26
    
At first I used a 1GB file of random bytes, loaded completely in the memory cache. Then I found that the md5sum speed is the same if I pipe null bytes into md5sum on the fly. I decided to use the latter method so that servers without 1GB RAM can also be tested. – netvope Apr 1 '10 at 22:35
    
I'd be curious to know how OpenSSL's md5 implementation compares to the others you've benchmarked. – Spiff Apr 2 '10 at 4:33

Just thought I'd chime in to add some updated speed benchmarks.

Using md5sum on Ubuntu 10.10, which uses coreutils 8.5:

  • Core 2 Quad (Kentsfield) 2.66GHz, 64-bit: 404 MB/s
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .