I have read the FAQ and I know this comes close to being closed as asking for a product recommendation...

I have looked at at least 40 "duplicate files" remover utilities (Windows, OSX and Linux) and none of them has the particular feature that I am looking for.

I need to now if there is anything out there that can do this or if I will have to write my own tool for it.

Just a "Yes, it exists" answer would be okay with me.
It would mean I just didn't search hard enough.

My required feature: Remove duplicate files across a large folder-structure, but ONLY if the duplicates reside in the SAME folder.
E.g. Say I have files A,B and C which are identical. A and C are in the same folder. B is in another folder. Either A or C needs to be removed (no preference), but B should be left alone.

Is there something out there that can do this ?
(Preferably Windows, but OS-X or Linux is OK too.)

  • First though: find -type d to traverse all folders, using that to -exec a new find with -maxdepth 1 and -exec to act on all files in that directory. Seems very inefficient though. DO you need a once-off solution or do you need to run this frequently? edit A single find with md5hashes storing all in a file and then acting on that file would be more efficient.). – Hennes Sep 5 '13 at 16:21
  • This question seems to be within the scope of allowed questions so long as you don't ask for software to do it, just some way. – Jon Sep 5 '13 at 16:22
  • I assume you want this tool to be automated and able to perform batch operations? – Ramhound Sep 5 '13 at 16:25
  • I'm saying Python has what you need (mainly because I'm working on a similar tool). What you want is "relatively" easy, with the os, the filecmp and shutil modules. Best part Python is portable for those systems. – Doktoro Reichard Sep 5 '13 at 16:32
  • @Hennes, Ramhound I need to run this every 2 weeks on 15000 folders with over 8 million files (and growing), expect approx 0.1% duplicates. And a MD5 hash collision (however unlikely) is not acceptable, need a full binary compare. Luckily most of the files themselves are relatively small (80% < 1MB, 19% < 5 MB, 1% around 100MB). – Tonny Sep 6 '13 at 7:44

You can use fdupes without -r so it doesn't descend to subdirectories. This prints a list of duplicate files:

find . -type d -exec fdupes -n {} \;

-n ignores empty files. Add -dN (--delete --noprompt) to delete all except the first duplicate file.

You can install fdupes on OS X with brew install fdupes.

  • If I understood, then the OP would have to run that command in all subdirectories? – Doktoro Reichard Sep 5 '13 at 18:39
  • No, the command finds all directories under the current directory recursively and then runs fdupes in them. – Lri Sep 6 '13 at 11:08
  • A slightly longer explanation. find starts searching in the current directory (marked with the .) and will find all files and directories. the -type d option limits that to all directories. So not you have a list off all directories. For each of those it will exec the program fdupes -n with the name of the directory appended (that is the {}). – Hennes Sep 6 '13 at 11:38
  • I had to install Homebrew first, but it is chugging away now using -ndN. Running for an hour now. Looks like it's going to take about 5 hours in total, which isn't too bad. (I'm running if from a MacPro against a SMB share on the NetApp NAS. The NAS has 10G LAN and the Mac only 1G so I'm not saturating the NAS. (Could have used a Linux server but that is on 10G too... Other users wouldn't be too happy I think.) – Tonny Sep 6 '13 at 12:44

Well as I said, I worked up a Python script that does just that.

I've hosted it at Google Code and I've open-sourced it as GPL v3, so I assume anyone that wants to improve the program can do that.

I've also debugged it somewhat (created tens of files in Windows, deleted all leaving the originals). The code is highly commented as to inform anyone of what the code actually does.

I've run it on Python 3.3 but I assume it should work with the latest Python 2.

Oh, and best part, it should work on any OS Python supports (Windows, OSX, Linux, ...)

  • Personally I'm not that familiar Python (I'm a C/C++ guy) but I will most certainly look into it. – Tonny Sep 6 '13 at 7:55
  • I was too a C/C++ guy, but Python has a "batteries included" policy that is quite appealing. If you're familiar with C++, then the syntax is slightly different but noting to get worked up about. – Doktoro Reichard Sep 6 '13 at 9:00
  • I just read your comment about the full binary comparison and updated the code in regards. – Doktoro Reichard Sep 6 '13 at 9:40
  • Even though I already used the fdupes method mentioned above I will most certainly look into your Python program. I want to become more familiar with Python anyway and I always find it easier to learn if I have some real world sample programs to study. Samples in books are often very contrived and a big program is usually too complex. This should be manageable in size and still deal with real-world issues. – Tonny Sep 6 '13 at 12:45

This is a slow but sure and very simple approach that should run on both OSX and Linux. I am assuming that you are interested in duplicate files residing in your $HOME but you can change that to suit your needs.

The idea is to first find a list of all directories, then compare the files inside them and delete any that are identical. As I said, this is very simplistic so it will just keep the first of any pair of files and delete the rest with no warning.

This will print out the dupes but will not make any changes to your files:

find $HOME  -mindepth 1 -type d | while read dir; do 
  find $dir -type -f -exec md5sum {} \; | sort > md5sums;
  gawk '{print $1}' md5sums | sort | uniq -d > dupes;
  while read d; do 
    echo "---"; grep -w $d md5sums | cut -d ' ' -f 2-;
  done < dupes
done; rm dupes md5sum 

This one will silently delete the duplicate files, only run it if you are sure that is OK:

find $HOME  -mindepth 1 -type d | 
while read dir; do 
  find $dir -type -f -exec md5sum {} \; | sort > md5sums;
  gawk '{print $1}' md5sums | sort | uniq -d |
  while read d; do grep -w $d md5sums | cut -d ' ' -f 2- | tail -n +2; done |
  | xargs rm ; 
done; rm dupes md5sum 

CAVEATS: This is slow, actually SLOW, will not give warnings and will delete files silently. On the bright side, it will only do so if those files are in the same directory which is what you want.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.