Messy Filesytem : Duplicate File Removal from the command line

Question

In debian/ubuntu I want to:

a) create a list of all the files in one directory tree
b) do the same for a second directory tree
c) compare the two lists such that, only the file NAMES are compared (i.e. just comparing the "file.txt" part so that "/home/folder/file.txt" == "/home/secondfolder/folder/file.txt)
d) output a list of all the duplicates

can anyone please explain how to do this using scripting languages or regex or something?

Answer 1

Use find /some/dir -printf '%f\t%p\n > files<N>.lst' or some variant thereof to find the files in the trees, then join -j 1 files1.lst files2.lst to combine the file listings into a single output.

Answer 2

Here is an example which would work with [t]csh, sh, etc., employing 'find', 'awk', 'sort', 'uniq', and a /bin/sh one-liner to run 'md5sum' to obtain the signature of each file. With the list of signatures for both directory structures' files, the command sequence will return those files which are the same:

find a/ b/ -type f -exec md5sum {} \; > /tmp/list; awk '{print $1}' ' | awk '{print $2}' | sh -c 'while read s; do awk "/^$s/ { print \$2}" /tmp/list; echo; done'

Essentially, this generates the md5sum for all files in the 'a' directory and the 'b' directory. The hexadecimal strings (first column) are fed to a pipeline filtering out instances of just a single occurrence for a given checksum, passing the remainder on to an /bin/sh iterator who pulls out all the actual files matching the checksum (and inserting the blank line between groups.)

It separates the groupings of duplicates by a blank line. This offers the obvious advantage (above and beyond the original request) of finding duplicate files which have the same content, but differing file-names.