Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.

share|improve this question
Please define "duplicate" - Binary equivalent, text equivalent, same size, same name? – l0b0 Mar 18 '11 at 12:33
This stuff also looks useful, finding dupes with a single line, even considering a check for same size first.… – deepc Mar 19 '12 at 0:46
up vote 5 down vote accepted

Don't know about Mac compatibility, but this Works For Me(TM):

[ -n "$1" ] || exit 1
exec 9< <( find "$1" -type f -print0 )
while IFS= read -r -d '' -u 9
    file_path="$(readlink -fn -- "$REPLY"; echo x)"
    exec 8< <( find "$1" -type f -not -path "$file_path" -print0 )
    while IFS= read -r -d '' -u 8 OTHER
        cmp --quiet -- "$REPLY" "$OTHER"
        case $? in
                echo -n "cmp -- "
                printf %q "${REPLY}"
                echo -n ' '
                printf %q "${OTHER}"
                echo ""
                echo "\`cmp\` failed!"
                exit 2

The result is a set of commands you can run to check that the result is correct :)

Edit: The last version works with really weird filenames like:

$'/tmp/--$`\\! *@ \a\b\E\E\f\r\t\v\\"\' \n'
share|improve this answer
Brilliant, thank you! – adrian Mar 18 '11 at 13:00
Nice work, but I see three issues: 1) this is O(n^2), meaning it will be really slow on big dirs. 2) it seems it will do both checks 'does a = b' and 'does b = a' 3) your check for same file name means it won't check at all if the file exists in both a dir and subdir. e.g. DIR/fred and DIR/subdir/fred will not be compared.hows u – Rich Homolka Mar 18 '11 at 16:08

This will find files under a dir with dupes. It's pretty raw, but it works.


trap "rm -f $TMPFILE" EXIT INT

if [ ! -d "$1" ]
    echo "usage $0 directory" >2
    exit 1

# dump fingerprints from all target files into a tmpfile
find "$1" -type f 2> /dev/null | xargs $CKSUMPROG  > $TMPFILE 2> /dev/null

# get fingerprints from tmpfile, get the ones with duplicates which means multiple files with same contents
for DUPEMD5 in $(cut -d ' ' -f 1 $TMPFILE | sort  | uniq -c | sort -rn | grep -v '^  *1 ' | sed 's/^ *[1-9][0-9]* //')
    if [ -z "$PRINTBLANK" ]

    grep "^${DUPEMD5} " $TMPFILE | gawk '{print $2}'
share|improve this answer
Clever. Too bad it starts over calculating those checksums everytime. I guess that part is left to the reader ;-) – deepc Mar 19 '12 at 0:29

it works for me on my mac, you will catch the duplicate file by their md5 value:

find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort
share|improve this answer
after sort if you add | uniq -d -f 1 -s 33 it will output just the duplicates – Tim Reddy Jan 31 '14 at 23:16

If only interested in files in the current directory (as indicated by OP), then this is the simpliest. For linux and Windows (msys - tested, or MinGW or Cygwin with any with GnuWin32). This will list all duplicates.

md5sum * | sort | uniq -D -w 32

For BSD/Mac OS X (it will list only the first duplicate)

md5 -r * | sort | uniq -d -w 32
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.