Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm interested in storing an indicator of file / directory integrity between two archived copies of directories. It's around 1TB of data stored recursively on hard drives. Is there a way using OpenSSL to generate a single hash for all the files that can be used as a comparison between two copies of the data, or at a later point to verify the data has not changed?

share|improve this question
up vote 2 down vote accepted

You could recursively generate all the hashes, concatenate the hashes into a single file, then generate a hash of that file.

share|improve this answer
For a single command, something like md5 -q <(find . -type f 2>/dev/null | xargs md5 -q | sort) works well in Bash and doesn't require a temp file. Alter if your system uses md5sum instead of md5. Also be aware that sort can behave differently on different platforms which will affect the final checksum if the order is different. Add flags like ! -name ".DS_Store" to the find component to ignore certain files, like the .DS_Store files on Mac OS X that can throw off the checksum since they're generated by the OS. – Justin Mrkva Mar 2 '14 at 22:37

You can't do a cumulative hash of them all to make a single hash, but you can compress them first then compute the hash:

$tar -czpf archive1.tar.gz folder1/
$tar -czpf archive2.tar.gz folder2/
$openssl md5 archive1.tar.gz archive2.tar.gz

to recursively hash each file:

$find . -type f -exec openssl md5 {} +
share|improve this answer
1TB of data - no room to tar them. Is there a way to recursively generate hashes of all files? – Kieveli Nov 19 '09 at 19:00
yes, added it to my answer. – John T Nov 19 '09 at 19:54
nice tar idea, but not always applicable. the 'find' method is better in general. if there is 'no room' for the tarball: % tar -cf - folder | openssl md5 – akira Nov 20 '09 at 7:22

Doing a md5 sum on the tar would never work unless all of the metadata (creation date, etc.) was identical as well, because tar stores that as part of its archive.

I would probably do an md5 sum of the contents of all of the files:

find folder1 -type f | sort | tr '\n' '\0' | xargs -0 cat | openssl md5
find folder2 -type f | sort | tr '\n' '\0' | xargs -0 cat | openssl md5
share|improve this answer

You should be probably interested to output the digest in coreutils format (identical to md5sum -b)

So md5sum command could be :

find . -path '*/.svn' -prune -o -type f -print0 | sort | tr '\n' '\0' | xargs -0 openssl dgst -md5 -r 

or with an output to a file

find . -path '*/.svn' -prune -o -type f -print0 | sort | tr '\n' '\0' | xargs -0 openssl dgst -md5 -r > ../mydigest.md5
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .