I have two directories: old/ and new/. Some of the files in old/ are duplicates of those in new/. Some merely have the same name, and a few may even have identical content but different names. I'm not interested in the subdirectory path to the files, so only an unqualified filename needs to be considered in the matching.

I want to remove the duplicate files from old/, so that it only contains files not also found in new/. For cases where the filenames match, but not the contents, or vice-versa, I want to have this listed in some log.

Has anyone encountered a similar problem, or have a better solution than hacking together a (perl or similar) script from scratch? A runtime dominated by calling md5sum on each file is acceptable.


Of course! Check out FSlint:

FSlint is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is duplicate files. It has both GUI and command line modes.

alt text

FSlint is free and open source software.

  • That looks close enough for what I want. Thanks! – P-Nuts Feb 11 '10 at 20:34

fdupes - finds duplicate files in a given set of directories

fdupes new/ old/

-d for delete

-N for don't ask

fdupes -N -d new/ old/
  • Won't this also find (and delete) duplicates within each directory? In other words, if new (or old) happens to contain duplicate files that may or may not be found within old, those would be deleted too? I don't think fdupes is strictly a diffing program as implied by your arguments "new" and "old". – Stephen Niedzielski Sep 1 '13 at 4:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.