Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

In debian/ubuntu I want to:

a) create a list of all the files in one directory tree
b) do the same for a second directory tree
c) compare the two lists such that, only the file NAMES are compared (i.e. just comparing the "file.txt" part so that "/home/folder/file.txt" == "/home/secondfolder/folder/file.txt)
d) output a list of all the duplicates

can anyone please explain how to do this using scripting languages or regex or something?

share|improve this question

2 Answers 2

Use find /some/dir -printf '%f\t%p\n > files<N>.lst' or some variant thereof to find the files in the trees, then join -j 1 files1.lst files2.lst to combine the file listings into a single output.

share|improve this answer

Here is an example which would work with [t]csh, sh, etc., employing 'find', 'awk', 'sort', 'uniq', and a /bin/sh one-liner to run 'md5sum' to obtain the signature of each file. With the list of signatures for both directory structures' files, the command sequence will return those files which are the same:

find a/ b/ -type f -exec md5sum {} \; > /tmp/list; awk '{print $1}' ' | awk '{print $2}' | sh -c 'while read s; do awk "/^$s/ { print \$2}" /tmp/list; echo; done'

Essentially, this generates the md5sum for all files in the 'a' directory and the 'b' directory. The hexadecimal strings (first column) are fed to a pipeline filtering out instances of just a single occurrence for a given checksum, passing the remainder on to an /bin/sh iterator who pulls out all the actual files matching the checksum (and inserting the blank line between groups.)

It separates the groupings of duplicates by a blank line. This offers the obvious advantage (above and beyond the original request) of finding duplicate files which have the same content, but differing file-names.

share|improve this answer
    
Only the names are to be compared, no checksums. –  Daniel Beck Jan 31 '11 at 2:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.