Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a lot of duplicate image files on my Windows computer, in different subfolders and with different filenames.

What Python script or freeware program would you recommend for removing the duplictes?

(I've read this similar question, but the poster there is asking about visual duplicates with differing filesizes. Mine are exact duplicates with different filenames.)

share|improve this question

migrated from stackoverflow.com Jun 15 '10 at 2:19

This question came from our site for professional and enthusiast programmers.

2  
Keep in mind that even if all the pixels are the same they still may have different EXIF information (modified by programs that handled the images at some stage) which will pose problems with most currently proposed solutions. –  user12889 Jun 15 '10 at 3:28
add comment

6 Answers

up vote 15 down vote accepted

Don't Rely on MD5 sums.

MD5 sums are not a reliable way to check for duplicates, they are only a way to check for differences.

Use MD5s to find possible *candidate* duplicates, and then for each pair sharing an MD5

  1. Opens both files
  2. Seeks forward in those files until one differs.

Seeing I'm getting downvoted by people doing naïve approaches to file duplicate Identity, If you're going to rely entirely on a hash algorithm, for goodness sake, use something tougher like SHA256 or SHA512, at least you'll reduce the probability to a reasonable degree by having more bits checked. MD5 is Exceedingly weak for collision conditions.

I also advise people read mailing lists here titled 'file check' : http://london.pm.org/pipermail/london.pm/Week-of-Mon-20080714/thread.html

If you say "MD5 can uniquely identify all files uniquely" then you have a logic error.

Given a range of values, of varying lengths from 40,000 bytes in length to 100,000,000,000 bytes in length, the total number of combinations available to that range greatly exceeds the possible number of values represented by MD5, weighing in at a mere 128 bits of length.

Represent 2^100,000,000,000 combinations with only 2^128 combinations? I don't think that likely.

The Least Naïve way

The least naïve way, and the fastest way, to weed out duplicates is as follows.

  1. By size: Files with different size cannot be identical. This takes little time as it does not have to even open the file.
  2. By MD5 : Files with different MD5/Sha values cannot be identical. This takes a little longer because it has to read all bytes in the file and perform math on them, but it makes multiple comparisons quicker.
  3. Failing the above differences: Perform a byte-by-byte comparison of the files. This is a slow test to execute, which is why it is left until after all the other eliminating factors have been considered.

Fdupes does this. And you should use software that uses the same criteria.

share|improve this answer
6  
It is literally more likely that your hard drive will magically destroy an image, than it is MD5 will collide. "Represent 2^100,000,000,000 combinations with only 2^128 combinations" - I agree with you here. If he had 2^100,000,000,000 pictures, MD5 (or almost any hash algorithm) would be bad. –  Greg Dean Jan 2 '09 at 5:57
3  
there is no guarantee, its just unlikely. Its not impossible. Its quite possible to have 10 files that all collide with each other, but are all entirely different. This is unlikely, but it can happen, so you must test for it. –  Kent Fredric Jan 2 '09 at 10:25
2  
file size, then MD5, and only then byte for byte check. –  Brad Gilbert Jan 4 '09 at 4:19
3  
@Kent - I 100% aggree with you. It is laziness to disregard something because it is very unlikely, even as unlikely as we are talking about. I'd be annoyed if some of my data was destroyed just becasue the person who wrote the program thought that something was too unlikely to bother coding for. –  Joe Taylor Nov 22 '10 at 16:06
add comment

It's a one liner on unix like (including linux) OSes or Windows with Cygwin installed:

find . -type f -print0 | xargs -0 shasum | sort |
  perl -ne '$sig=substr($_, 0, 40); $file=substr($_, 42); \
    unlink $file if $sig eq $prev; $prev = $sig'

md5sum (which is about 50% faster) can be used if you know there is no deliberately created collisions (you'd have better chance to win 10 major lotteries than the chance to find one naturally occurring md5 collision.)

If you want to see all the dups you have instead of removing them just change the unlink $file part to print $file, "\n".

share|improve this answer
1  
You can use -print0 and xargs-0 to catch spaces as well, but find also has an -exec option that's useful here: find . -type f -exec shasum {} \; | sort ... Also: You shouldn't use @F (-a) because it won't work with spaces. Try substr instead. –  geocar Jan 2 '09 at 3:42
    
Good call, geocar. Updated the answer with your suggestions. –  obecalp Jan 2 '09 at 3:58
    
"md5sum (which is about 50% faster) can be used if you know there is no deliberately created collisions" - exactly –  Greg Dean Jan 2 '09 at 6:34
add comment

I've used fdupes (written in C) and freedups (Perl) on Unix systems, and they might work on Windows as well; there are also similar ones that are claimed to work on Windows: dupmerge, liten (written in Python), etc.

share|improve this answer
    
Perl and Python software should work identically on Windows and *nix systems, assuming details of the filesystem don't matter. –  CarlF Nov 3 '10 at 12:51
add comment

To remove duplicate images on Windows take a look at DupliFinder. It can compare pictures by a variety of criteria such as name, size, and actual image information.

For other tools to remove duplicate files take a look at this Lifehacker article.

share|improve this answer
add comment

Instead of DupliFinder, try the forked project instead, DeadRinger. We've fixed a ton of bugs in the original project, added a bunch of new features, and dramatically improved performance.

share|improve this answer
add comment

One option can be Dupkiller.

DupKiller is one of the fastest and the most powerful tools for searching and removing duplicate or similar files on your computer. Complicated algorithms, built in its searching mechanism, perform high results — rapid file search. A lot of options allow to flexibly customizing the search.

enter image description here

share|improve this answer
add comment

protected by Community Sep 29 '12 at 7:45

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.