Possible Duplicate:
Duplicate file finder

Is there a tool that can identify/manage duplicates at the directory level? I've tried using duplicate file removal software to weed out duplicates, but these applications are too granular. I am looking to manage my duplicates at the folder level. Most of my duplicate files are also in (nearly) duplicate folder structures, so I want to to be able to identify the root (nearly) duplicate folder and delete/merge/update the root folder, instead of managing duplicates on a file-by-file basis.

One of the ways these duplicate folders was created was because whenever I upgraded to a larger harddrive, I backed up the old harddrive contents to a folder on the new harddrive. I have several directories filled with duplicate files and directories. Some directories are super-sets of other directories (contain the same files/directories plus additional files) Others contain a common subset of files/directories, but each contain additional files.


One command line tool I've used and like very much because duplicates can be turned into hard links is FINDDUPE by Matthias Wandel. http://www.sentex.net/~mwandel/finddupe/

Hardlinks make duplicates appears as if the file still exists in two places, but only one copy exists on the disk. But not that this only works on NTFS file systems, and you must run it with administrator privileges.

Bets of all, it's totally free. :)


I'm not sure what you mean by "granular" when referring to other duplicate removal software, but after trying a couple dozen software programs that do this, I have found that Glary Utilities suits my needs perfectly. It is very fast searching a folder or even a drive or comparing one folder to another for duplicates based on exact size, or time stamp, and or name. You can choose which file extensions to filter in or out of the search. And you can edit the results manually, or let the program delete one of the duplicates in every group. Might I add best of all, it's free and very very fast.

  • Thanks for the suggestion, but this one is too "granular." I am looking to manage my duplicates at the folder level. Most of my duplicate files are also in (nearly) duplicate folder structures, so I want to to be able to identify the root (nearly) duplicate folder and update/delete that one. – Leftium Aug 20 '09 at 21:44

It's hard to find software with the right level of grandularity that does what you want. I'd recommend learning a scripting language ( try MS Shell + VBScript , perl, python ) and write a little command line utility to do this yourself. You can make it as grandular as you like.

Whoever wrote w/e software is suggested above, probably did not have the exact same idea as you, so you will probably have to do some manual looking/deleting.


I'd recommend checking the site commandlinefu. Your on windows so you'd have to use cygwin.

I found this entry at commandlinefu.com that might be usefull

find . -type f -print0|xargs -0 md5sum|sort|perl -ne 'chomp;$ph=$h;($h,$f)=split(/\s+/,$_,2);print "$f"."\x00" if ($h eq $ph)'|xargs -0 rm -v --
  • I prefer UnxUtils and/or MSys instead of Cygwin. I've always found Cygwin to be very slow, due to its emulation layer. – davr Aug 27 '09 at 16:26

If you got suspected folders, you can use a trick.

I'm using the well known xcopy trick to copy everything in the suspected folder to a file. Then, I compare these two files to see if they're intact.


If you have access to the (excellent) Beyond Compare, check out the Folder Sync feature.

alt text


Have you tried AusLogics? They currently supply some free Duplicate File Finding software, here's a link:


Hope this helps :)


i wonder if maybe this would help you, its called winmerge. seems like you can set it up to compare 2 folders and take an action on the folder depending on the results of the comparison.


  • This could be very helpful, but I am looking for a tool that also finds similar/duplicate folders (I may not realize where all the dupes are located, beforehand) – Leftium Aug 25 '09 at 15:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.