I'm interested in storing an indicator of file / directory integrity between two archived copies of directories. It's around 1TB of data stored recursively on hard drives. Is there a way using OpenSSL to generate a single hash for all the files that can be used as a comparison between two copies of the data, or at a later point to verify the data has not changed?


You could recursively generate all the hashes, concatenate the hashes into a single file, then generate a hash of that file.

  • For a single command, something like md5 -q <(find . -type f 2>/dev/null | xargs md5 -q | sort) works well in Bash and doesn't require a temp file. Alter if your system uses md5sum instead of md5. Also be aware that sort can behave differently on different platforms which will affect the final checksum if the order is different. Add flags like ! -name ".DS_Store" to the find component to ignore certain files, like the .DS_Store files on Mac OS X that can throw off the checksum since they're generated by the OS. – Justin Mrkva Mar 2 '14 at 22:37

You can't do a cumulative hash of them all to make a single hash, but you can compress them first then compute the hash:

$tar -czpf archive1.tar.gz folder1/
$tar -czpf archive2.tar.gz folder2/
$openssl md5 archive1.tar.gz archive2.tar.gz

to recursively hash each file:

$find . -type f -exec openssl md5 {} +
  • 1
    1TB of data - no room to tar them. Is there a way to recursively generate hashes of all files? – Kieveli Nov 19 '09 at 19:00
  • yes, added it to my answer. – John T Nov 19 '09 at 19:54
  • nice tar idea, but not always applicable. the 'find' method is better in general. if there is 'no room' for the tarball: % tar -cf - folder | openssl md5 – akira Nov 20 '09 at 7:22

Doing a md5 sum on the tar would never work unless all of the metadata (creation date, etc.) was identical as well, because tar stores that as part of its archive.

I would probably do an md5 sum of the contents of all of the files:

find folder1 -type f | sort | tr '\n' '\0' | xargs -0 cat | openssl md5
find folder2 -type f | sort | tr '\n' '\0' | xargs -0 cat | openssl md5

You should be probably interested to output the digest in coreutils format (identical to md5sum -b)

So md5sum command could be :

find . -path '*/.svn' -prune -o -type f -print0 | sort | tr '\n' '\0' | xargs -0 openssl dgst -md5 -r 

or with an output to a file

find . -path '*/.svn' -prune -o -type f -print0 | sort | tr '\n' '\0' | xargs -0 openssl dgst -md5 -r > ../mydigest.md5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.