How to Find Most Common or Frequent Files In Directory?

Question

I have a directory with lots of images(100,000+). Many of these are duplicates/identical images but obviously all have different filenames. I need to find the images that have the most duplicates in this directory. For example file1.jpeg has 120 duplicates, file2.jpeg has 90 duplicates, etc.

I was thinking I would get the md5 of each file and do some kind of sort, but I'm fuzzy on the details. Can this be done with a shell script?

To be clear, I don't need to remove duplicates(yet), I need to find which files have the most copies.

I'm on OS X if that helps.

Answer 1

If the files are exact duplicates, postprocessing the output of shasum * | sort is likely to help. Save it to a file since the computation can take a while and you're likely to need it more than once:

shasum * | sort >/tmp/shasums

For example, to see the spread of identical files (with only checksums, not file names):

</tmp/shasums cut -d ' ' -f 1 | uniq -c

Here's a way to see both file names and duplicate counts:

</tmp/shasums sed 's/ .*//' | uniq -c - | join -1 2 - /tmp/shasums | sort -k 2,1

Without GNU uniq, I have nothing better to offer to show the file names in a nice way than the following Perl script:

</tmp/shasums perl -lne '
    s/^([^ ]*?)  //; # set $1 to the checksum and $2 to the filename
    push @{$names{$1}}, $_; # dispatch file names by checksum
    END {
        # iterate through the checksums, sorted by repeat count
        foreach (sort {@$a <=> @$b} values %names) {
            # print the repeat count and the file names
            printf "%d %s\n", scalar(@$_), join(" ", @$_)
        }
    }'

Answer 2

This is a quick and dirty pipeline that will print names of duplicates between lines of hyphens. It only looks in the current directory, but you could use find to do a recursive search.

md5sum *.jpeg | sort | awk '{if ($1 != prev) print "-----"; print $2; prev = $1}'

Example output:

-----
unique1.jpeg
-----
dup1.jpeg
dup2.jpeg
dup3.jpeg
-----
same1.jpeg
same2.jpeg
-----
solo1.jpeg