0

I am looking for a linux solution for the following problem:

Given two directories with a large number of files. All file names are random and different in both directories. The contents of some of the files in the two directories are identical however.

I want to copy all files which occur in both directories to a third directories. ("Occur in both" means have the same content not the same name.)

1

Assuming your filenames don't have any whitespace and there are no subdirectories in either directory, the following will print pairs of filenames with matching MD5 sums:

join -o 1.2,2.2 <(md5sum $D1/* | sort) <(md5sum $D2/* | sort)

To get just one of the filenames, use -o 1.2 or -o 2.2.

If filenames (or paths) might include whitespace, you'll need to be more clever.

If a single directory might have the same file with more than one name, you will also need to be more clever -- and you will need to decide exactly what to do. One possibility would be to filter out the duplicates before doing the join:

join -o 1.2,2.2 <(md5sum $D1/* | sort | uniq -w16) \
                <(md5sum $D2/* | sort | uniq -w16)

DO NOT USE sum

sum outputs a 16-bit checksum; if you have even a couple of hundred files in each directory, it is likely that you will get a false positive if you compare 16-bit checksums. md5sum is not absolutely definitively safe, either, but the odds of a collision with 128-bit checksums is tiny. In case of doubt, and if it's really important, cmp the files as well:

join -o 1.2,2.2 <(md5sum $D1/* | sort) <(md5sum $D2/* | sort) |
while read F1 F2; do
  if cmp -s $F1 $F2; then
    cp F1 $D3
  fi
done

(Again, that won't work if the files might have whitespace in their names.)

0

Using your favorite shell for this pseudo code:

cd D1; sum * | while read l; do echo "D1 $l"; done >/tmp/foo
cd D2; sum * | while read l; do echo "D2 $1"; done >>/tmp/foo

sort -n /tmp/foo | awk '
$1 == prev_cs { echo "cp $3 dest"}
     /prev_cs = $1/
' | shell

You can save the awk output for review before issuing the copies, if you like

  • sum is really a bad idea, see en.wikipedia.org/wiki/Birthday_problem. If you have 300 different random files, the odds are roughly even that two of them will have the same sum. – rici Mar 15 '14 at 2:25
  • Then pick a different checksum algorithm, like openssl dgst -md5. Same code. – mpez0 Mar 15 '14 at 12:05
0

this might do the job, as with mpez0's ans for copying the dups.

find {tst1,tst2} -exec sum {} {} \; 2> /dev/null | sort | uniq

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.