Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am using OSX 10.8.4 and would like to remove duplicate files in a single flat directory if an md5 hash matches.

I have looked up several answers but none of them work for me, I think because of differences in command syntax between terminal versions.

Things I have tried:

http://www.chriswrites.com/2012/02/how-to-find-and-delete-duplicate-files-in-mac-os-x/

Finding and removing duplicate files in osx with a script

and some of these: http://www.commandlinefu.com/commands/view/3555/find-duplicate-files-based-on-size-first-then-md5-hash#comment

This approach feels closest:

find . -size 20 \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif - /tmp/f.tmp > duplicates.txt

But I get an error: grep: -: No such file or directory

Upon inspection, /tmp/f.tmp does exist. The duplicates.txt file is created but is empty.

How do I deduplicate these files?

share|improve this question
    
If you're not opposed to using a script, checkout dupinator, it is a python script which could accomplish this task. code.activestate.com/recipes/362459 –  spuder Aug 17 '13 at 18:17
    
Tried this. it finds no duplicates in a test folder which has intentional duplicates. Also comment 6 notes a data-losing bug: code.activestate.com/recipes/362459/#c6 –  Simple As Could Be Aug 17 '13 at 18:45

2 Answers 2

up vote 1 down vote accepted

By default on Mac OS you'll be getting BSD grep, while the command that you've posted is most probably for GNU grep. The two versions of the tools are similar, but not identical. This is true for other tools as well (e.g. the GNU and BSD versions of date behave slightly differently, too).

The issue is the - following the grep command. GNU grep interprets this as stdin (thus, those lines that have been identified as duplicates by uniq -d), while BSD grep is actually looking for a while called -; hence the error message:

>> find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif - /tmp/f.tmp
grep: -: No such file or directory

The same thing happens if you replace the - with another name that doesn't match any files:

>> find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d | grep -hif unknown-file /tmp/f.tmp
grep: unknown-file: No such file or directory

You can bash's process substitution to feed the stdout from everything up to the uniq to grep as a file, e.g. (I've dropped the size criterion in this example, but the rest is the same):

>> grep -hif <(find . \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ' ' | uniq -d) /tmp/f.tmp
4192268874 275 ./foo/META-INF/leiningen/foo/foo/project.clj
4192268874 275 ./foo/project.clj
share|improve this answer

Finding duplicates based on hashkey works well and is very fast. I use the following code often. If you are running this on a Mac and face any issues, install GNU tools and use the second version.

Linux

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Mac version

gfind -not -empty -type f -printf "%s\n" | sort -rn | guniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 gmd5sum | sort | guniq -w32 --all-repeated=separate

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.