Extract identical files from two directories

Question

I am looking for a linux solution for the following problem:

Given two directories with a large number of files. All file names are random and different in both directories. The contents of some of the files in the two directories are identical however.

I want to copy all files which occur in both directories to a third directories. ("Occur in both" means have the same content not the same name.)

Answer 1

Assuming your filenames don't have any whitespace and there are no subdirectories in either directory, the following will print pairs of filenames with matching MD5 sums:

join -o 1.2,2.2 <(md5sum $D1/* | sort) <(md5sum $D2/* | sort)

To get just one of the filenames, use -o 1.2 or -o 2.2.

If filenames (or paths) might include whitespace, you'll need to be more clever.

If a single directory might have the same file with more than one name, you will also need to be more clever -- and you will need to decide exactly what to do. One possibility would be to filter out the duplicates before doing the join:

join -o 1.2,2.2 <(md5sum $D1/* | sort | uniq -w16) \
                <(md5sum $D2/* | sort | uniq -w16)

---

DO NOT USE sum

sum outputs a 16-bit checksum; if you have even a couple of hundred files in each directory, it is likely that you will get a false positive if you compare 16-bit checksums. md5sum is not absolutely definitively safe, either, but the odds of a collision with 128-bit checksums is tiny. In case of doubt, and if it's really important, cmp the files as well:

join -o 1.2,2.2 <(md5sum $D1/* | sort) <(md5sum $D2/* | sort) |
while read F1 F2; do
  if cmp -s $F1 $F2; then
    cp F1 $D3
  fi
done

(Again, that won't work if the files might have whitespace in their names.)

Answer 2

Using your favorite shell for this pseudo code:

cd D1; sum * | while read l; do echo "D1 $l"; done >/tmp/foo
cd D2; sum * | while read l; do echo "D2 $1"; done >>/tmp/foo

sort -n /tmp/foo | awk '
$1 == prev_cs { echo "cp $3 dest"}
     /prev_cs = $1/
' | shell

You can save the awk output for review before issuing the copies, if you like

Answer 3

this might do the job, as with mpez0's ans for copying the dups.

find {tst1,tst2} -exec sum {} {} \; 2> /dev/null | sort | uniq