Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for a program, that should be source portable across Windows and Linux ( for eg. ANSI C, etc ) that generates cryptographic hashes like MD5, SHA, etc as well as CRC32 of a file/list of files passed to it.

I will be running this executable on terabytes of files, generating their SHA, MD5 and CRC32 ( and more in the future ) signatures, so speed is important.

What I had at mind is exactly what ReHash is ReHash, ReHash.

Unfortunately, I saw that many users have complained that there are errors in the implementation of the hashes as well as errors in the way padding ( for block based algorithms ) had been implemented.

Being no expert in cryptography, and just looking for a black box solution that simply gives me the hashes that I want without requiring me to, more than, compile some code, I was wondering if there is nothing better?

I could, of course, write a glue program in Python that would use the crypto modules to generate what I want, but I would have preferred a compiled binary from a language like C.

I will be running this executable on terabytes of files, generating their SHA, MD5 and CRC32 ( and more in the future ) signatures and handling all this from Python code, so something compatible with Python would be preferred, but not at the expense of C like speed.

share|improve this question

The python crypto operations are implemented in native code (compiled from C). Since you want the values in a python program, using them will be simpler.

Linux comes with utilities for checksum calculations (cksum, md5sum, sha1sum, ...). So do most other unices. There are several Windows ports of the GNU utilities (which is what you get on Linux): Cygwin, Gnuwin32, Msys, ... You'll need recent enough utilities if you want to get SHA-256 and SHA-512.

There are several ANSI C implementations with very liberal licenses of various cryptographic algorithms floating around, often not collected in a single distribution. You could search for them and test them on small input to check their reliability.

Under Windows, make sure you're correctly treating files as binary or text as desired, since the checksums are defined over byte streams, not line streams. (Normally you'd want to open the files in binary mode, but if you have a text file that got transcoded to Windows line endings, you'll need to open it as text to reverse the effect.) Under any OS, make sure you don't do any encoding translation when opening the file.

Since speed is very important to you, gather all the implementations you can find and benchmark them on moderate-size input (a few megabytes). Different implementations may give better speed on different architectures. 64-bit implementations are likely to be faster where you can run them at all.

share|improve this answer
Although the crypto routines are written in C, I did not seem to notice any function that took a filename as argument. That means I would need to open the file and read it, etc - all from Python. This means that this would not be C like performance. Also, I would be more interested in an existing solution than rolling out my own! – PoorLuzer Oct 11 '10 at 1:00
But the file routines are also in C. – Ignacio Vazquez-Abrams Oct 11 '10 at 15:37

OpenSSL has tools that compute hashes. The cygwin ( project has openssl tools. Though it will be a bit slower than a pure Windows app because of the cygwin layer, you'll also get an environment where you can script your hash generation.

share|improve this answer
And Python has an OpenSSL binding, which makes it an alternative to the standard hashlib. The Cygwin layer shouldn't influence the speed of pure integer computation, but the quality of the compiler might (I'm sure there are better compilers for win32/i386 than Cygwin's gcc, especially if 64-bit is possible.). – Gilles Oct 11 '10 at 17:28
So which one would be faster and correct : OpenSSL or hashlib? – PoorLuzer Oct 12 '10 at 3:33
Both would be correct, they implement same algorithm. Fast, you'd need to test. I'm thinking implementation ease ease in going over your huge dataset are more important than raw speed. – Rich Homolka Oct 12 '10 at 15:05
hashlib would be "easier" ti implement as it's built in. Let me see if I can get some of those file verifiers to work correctly and spew out the hashes I want before I key in some code to do the same thing. – PoorLuzer Oct 13 '10 at 0:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .