Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to compare all the files in two folders via an md5sum in one command. Some like the following (bash) in Debian:

$ cd ~/FOLDER1
$ md5sum ~/FOLDER2/* | md5sum -c -

The idea is that the output of the hashes from the first md5sum will be passed into the second one and used as the input file. However, testing of this shows that it just compares each file in FOLDER2 to itself and returns "OK" for each one. I think the reason this is not working is because the filenames output from the first md5sum include the full path. I've looked at md5deep but have not found anything to help me there. I know that it is possible to do the md5sum for one folder, write the results out to a file, and then use that file as the input for the second md5sum. I was wanting to do it all in one line though a pipe, rather than using two commands and writing out a file.

Edit: The accepted answer here (using diff) might do what I want, but I don't know if diff (correctly) compares binary files.

Edit: The get the output I wanted using md5sum (which shows the filename and "OK"), I've resorted to writing a batch file. Execute with ~/FOLDER1 ~/FOLDER2.

cd "$1"
md5sum * > /tmp/md5sum.cmp
cd "$2"
md5sum -c /tmp/md5sum.cmp
cd $HERE

This script will only compare files which are present in ~/FOLDER. If ~/FOLDER2 has additional files, these will not be compared and no output will indicate that they even exist.

share|improve this question
up vote 2 down vote accepted

You can use process substitution to pass the output of the 2 md5sum's to diff. Diff in this case would be fine because the md5 outputs are plain text. Something like:

diff <(md5 ~/FOLDER1/* | awk '{print $4}') <(md5 ~/FOLDER2/* | awk '{print $4}')

Sorry, I don't have Debian here and can't test this on it. The above is tested on OS X that has md5 which may be slightly different in terms of output. On OS X the 4th column of md5 is the actual md5 sum, that's why I am taking just these columns.

Instead of awk, you could also use cut, but you may need to change the separator to get the 4th column (these are not tab-separated).

share|improve this answer
Thanks. This works quite well, as long as you know how to interpret the output of diff, which took me a while :) In Debian, the command is md5sum and the actual hash is in column 1. – SabreWolfy Jan 21 '12 at 19:37

Kind of long but returns the filename and OK if they match. Instead of using '-c' it's just comparing the two strings output from running md5sum on the file in each folder.

for f in *; do [[ -f $f ]] && if [ $(md5sum "$f" | cut -d" " -f1) == $(md5sum dir2/"$f" | cut -d" " -f1) ]; then echo "$f" "OK"; else echo "$f" "MODIFIED"; fi; done
share|improve this answer

From my .bashrc file.
very old stuff, it should be possible to write in much sorter code. I never got around to re writing it. (like everything else that was meant for a temporary-fix, used forever) I am posting this shamefull piece of code, hopefully someone can do it better and and post the result :-)

Features :

  • Recrusive dir transversal
  • md5sum check for uniqueness/diffrence
  • Lists updated files in full path

The code says it all. arg1 is old dir, arg2 is new dir.

function find-updated-files-between-old-new(){
 [ ! -d "$1" ] || [ ! -d "$2" ] && echo "*** Error: The directory is not found." > /dev/stderr && return 1;
( ( cat <(cd "$1";find . -type f -printf "+%p\n") <(cd "$2";find . -type f -printf "-%p\n")
 )|sort -k1.2|tee 1>/dev/null >(uniq -us1|awk -v B="$2" 'BEGIN{sub("/$",""B)}/^-/{print B substr($0,3);
 }') >(uniq -ds1|awk -vA="$1" -vB="$2" 'BEGIN{B=g(B);A=g(A)}{
 C=substr($0,3);if(f(A)!=f(B))print B C;}function g(y){sub("/$","",y);return y}
 function f(y,z,e){e="md5sum \""y""C"\"";e|getline z;close(e);return substr(z,1,32)}' )
 ) | cat

As the function name implies

function find-files-name-collision-between-dir1dir2(){
 [ ! -d "$1" ] || [ ! -d "$2" ] && echo "*** Error: The directory is not found." > /dev/stderr && return 1;
( cat <(cd "$1";find . -type f -printf "+%p\n") <(cd "$2";find . -type f -printf "-%p\n") )|sort -k 1.2 | uniq -d -s 1

Just for completeness

function mv-mergedir1todir2(){
 [ ! -d "$1" ] || [ ! -d "$2" ] && echo "*** Error: The directory is not found." && return 1;
 ( cd "$1" ; tar cf - . ) | (cd "$2" ; tar --keep-old-files xvf - )
 echo -e "Done. Duplicate filnames are not replaced. \n#Use \n# ( cd \"$1\" ; tar cf - . ) | (cd \"$2\" ; tar --overwrite xvf - ) \n#if you do not like that. "

This awful piece of code should be removed from my bashrc, however it has been around for a long time...

share|improve this answer
Does it work? Is this meant as an answer to the question, or are you wanting feedback on the code? – SabreWolfy Aug 14 '12 at 17:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .