Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory with lots of images(100,000+). Many of these are duplicates/identical images but obviously all have different filenames. I need to find the images that have the most duplicates in this directory. For example file1.jpeg has 120 duplicates, file2.jpeg has 90 duplicates, etc.

I was thinking I would get the md5 of each file and do some kind of sort, but I'm fuzzy on the details. Can this be done with a shell script?

To be clear, I don't need to remove duplicates(yet), I need to find which files have the most copies.

I'm on OS X if that helps.

share|improve this question

2 Answers 2

up vote 0 down vote accepted

If the files are exact duplicates, postprocessing the output of shasum * | sort is likely to help. Save it to a file since the computation can take a while and you're likely to need it more than once:

shasum * | sort >/tmp/shasums

For example, to see the spread of identical files (with only checksums, not file names):

</tmp/shasums cut -d ' ' -f 1 | uniq -c

Here's a way to see both file names and duplicate counts:

</tmp/shasums sed 's/ .*//' | uniq -c - | join -1 2 - /tmp/shasums | sort -k 2,1

Without GNU uniq, I have nothing better to offer to show the file names in a nice way than the following Perl script:

</tmp/shasums perl -lne '
    s/^([^ ]*?)  //; # set $1 to the checksum and $2 to the filename
    push @{$names{$1}}, $_; # dispatch file names by checksum
    END {
        # iterate through the checksums, sorted by repeat count
        foreach (sort {@$a <=> @$b} values %names) {
            # print the repeat count and the file names
            printf "%d %s\n", scalar(@$_), join(" ", @$_)
        }
    }'
share|improve this answer

This is a quick and dirty pipeline that will print names of duplicates between lines of hyphens. It only looks in the current directory, but you could use find to do a recursive search.

md5sum *.jpeg | sort | awk '{if ($1 != prev) print "-----"; print $2; prev = $1}'

Example output:

-----
unique1.jpeg
-----
dup1.jpeg
dup2.jpeg
dup3.jpeg
-----
same1.jpeg
same2.jpeg
-----
solo1.jpeg
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.