Deduplicate files using the command line in OSX Lion

Question

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches.

I have looked up several answers but none of them work for me, I think because of differences in command syntax between terminal versions.

Things I have tried:

http://www.chriswrites.com/2012/02/how-to-find-and-delete-duplicate-files-in-mac-os-x/

Finding and removing duplicate files in osx with a script

and some of these: http://www.commandlinefu.com/commands/view/3555/find-duplicate-files-based-on-size-first-then-md5-hash#comment

This approach feels closest:

find . -size 20 \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif - /tmp/f.tmp > duplicates.txt

But I get an error: grep: -: No such file or directory

Upon inspection, /tmp/f.tmp does exist. The duplicates.txt file is created but is empty.

How do I deduplicate these files?

Answer 1

By default on Mac OS you'll be getting BSD grep, while the command that you've posted is most probably for GNU grep. The two versions of the tools are similar, but not identical. This is true for other tools as well (e.g. the GNU and BSD versions of date behave slightly differently, too).

The issue is the - following the grep command. GNU grep interprets this as stdin (thus, those lines that have been identified as duplicates by uniq -d), while BSD grep is actually looking for a while called -; hence the error message:

>> find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif - /tmp/f.tmp
grep: -: No such file or directory

The same thing happens if you replace the - with another name that doesn't match any files:

>> find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif unknown-file /tmp/f.tmp
grep: unknown-file: No such file or directory

You can bash's process substitution to feed the stdout from everything up to the uniq to grep as a file, e.g. (I've dropped the size criterion in this example, but the rest is the same):

>> grep -hif <(find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d) /tmp/f.tmp
4192268874 275 ./foo/META-INF/leiningen/foo/foo/project.clj
4192268874 275 ./foo/project.clj

Answer 2

Finding duplicates based on hashkey works well and is very fast. I use the following code often. If you are running this on a Mac and face any issues, install GNU tools and use the second version.

Linux

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Mac version

gfind -not -empty -type f -printf "%s\n" | sort -rn | guniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 gmd5sum | sort | guniq -w32 --all-repeated=separate