What is the best method to remove duplicate image files from your computer

Question

I have a lot of duplicate image files on my Windows computer, in different subfolders and with different filenames.

What Python script or freeware program would you recommend for removing the duplictes?

(I've read this similar question, but the poster there is asking about visual duplicates with differing filesizes. Mine are exact duplicates with different filenames.)

Answer 1

Don't Rely on MD5 sums.

MD5 sums are not a reliable way to check for duplicates, they are only a way to check for differences.

Use MD5s to find possible candidate duplicates, and then for each pair sharing an MD5

1. Opens both files
2. Seeks forward in those files until one differs.

Seeing I'm getting downvoted by people doing naïve approaches to file duplicate Identity, If you're going to rely entirely on a hash algorithm, for the goodness sake, use something tougher like SHA256 or SHA512, at least you'll reduce the probability to a reasonable degree by having more bits checked. MD5 is Exceedingly weak for collision conditions.

I also advise people read mailing lists here titled 'file check' : http://london.pm.org/pipermail/london.pm/Week-of-Mon-20080714/thread.html

If you say "MD5 can uniquely identify all files uniquely" then you have a logic error.

Given a range of values, of varying lengths from 40,000 bytes in length to 100,000,000,000 bytes in length, the total number of combinations available to that range greatly exceeds the possible number of values represented by MD5, weighing in at a mere 128 bits of length.

Represent 2^100,000,000,000 combinations with only 2^128 combinations? I don't think that likely.

The Least Naïve way

The least Naïve way, and the fastest way, to weed out duplicates is as follows.

1. By size: Files with different size cannot be identical. This takes little time as it does not have to even open the file.
2. By MD5 : Files with different MD5/Sha values cannot be identical. This takes a little longer because it has to read all bytes in the file and perform math on them, but it makes multiple comparisons a quicker.
3. Failing the above differences: Perform a byte-by-byte comparison of the files. This is a slow test to execute, which is why it is left until after all the other eliminating factors have been considered.

Fdupes does this. And you should use software that uses the same criteria.

Answer 2

It's a one liner on unix like (including linux) OSes or Windows with Cygwin installed:

find . -type f -print0 | xargs -0 shasum | sort |
  perl -ne '$sig=substr($_, 0, 40); $file=substr($_, 42); \
    unlink $file if $sig eq $prev; $prev = $sig'

md5sum (which is about 50% faster) can be used if you know there is no deliberately created collisions (you'd have better chance to win 10 major lotteries than the chance to find one naturally occurring md5 collision.)

If you want to see all the dups you have instead of removing them just change the unlink $file part to print $file, "\n".

Answer 3

I've used fdupes (written in C) and freedups (Perl) on Unix systems, and they might work on Windows as well; there are also similar ones that are claimed to work on Windows: dupmerge, liten (written in Python), etc.

Answer 4

To remove duplicate images on Windows take a look at DupliFinder. It can compare pictures by a variety of criteria such as name, size, and actual image information.

For other tools to remove duplicate files take a look at this Lifehacker article.

Answer 5

I believe there are some freewares to handle that, indeed.

I had this problem some years ago, although the issue was to sort out lot of small Gif images (icons) with different names in the same folder.
I wrote a simple Java program computing the MD5 hash of the files, storing that and the names in a hash set or some similar structure eliminating naturally duplicates. Thus I had a list of file names to copy elsewhere to get unique files.

Answer 6

Instead of DupliFinder, try the forked project instead, DeadRinger. We've fixed a ton of bugs in the original project, added a bunch of new features, and dramatically improved performance.

Answer 7

I use an outdated version of More Space (but no experience with current version). It lets you find duplicates by size and timestamp, and optionally calculates CRC for duplicate detection. (free unlimited version with banner ads)

Answer 8

DoubleKiller

Answer 9

I've faced the problem of file duplication in the scads of files I've collected and wrote s little python program to simply run down a set of file lists to look for and eliminate duplicates but there are things I would suggest.

One procedure here is to first compare file sizes, then compare md5sums then do a bytewise comparison to truly decide if the files are duplicates. I wonder if this is really efficient since to calculate an md5sum you have to read the entire file, then if you find you have similar md5sums you start reading the file again. I gues you might say that the file could all be in a cache somewhere by the second read but I doubt it somehow.

I found I got a better overall response by dropping the md5sums and simply doing the bytewise comparisons. The differences between two files of similar size but different contents would show up rather quickly and the comparison wouldn't usually reach all the way to the end of the file to be complete.

The main problem I face now is really comparing and identifying two images to determine whether they are similar or even the same but with more or less artifacts from jpeg compression. You always want to keep the better of the two images even if they seem identical at a cursory glance.

Answer 10

I wrote http://pypi.python.org/pypi/SizeDupe%20Reporter/0.9.1 for this

Answer 11

Personally I would recursively search directories, find file sizes that are equivalent then md5 and check for a duplicate. I would also skip over directories where duplications really would not be likely to occur such as a system folder..

Answer 12

though this post is old, but people may find by googling.

You may try to use NoClone - True duplicate file finder no false duplicate files

Answer 13

One option can be Dupkiller .

Answer 14

Try this Free Duplicate File Finder