Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.

share|improve this question
3  
Please define "duplicate" - Binary equivalent, text equivalent, same size, same name? – l0b0 Mar 18 '11 at 12:33
    
This stuff also looks useful, finding dupes with a single line, even considering a check for same size first. commandlinefu.com/commands/view/3555/… – deepc Mar 19 '12 at 0:46
up vote 5 down vote accepted

Don't know about Mac compatibility, but this Works For Me(TM):

#!/bin/bash
[ -n "$1" ] || exit 1
exec 9< <( find "$1" -type f -print0 )
while IFS= read -r -d '' -u 9
do
    file_path="$(readlink -fn -- "$REPLY"; echo x)"
    file_path="${file_path%x}"
    exec 8< <( find "$1" -type f -not -path "$file_path" -print0 )
    while IFS= read -r -d '' -u 8 OTHER
    do
        cmp --quiet -- "$REPLY" "$OTHER"
        case $? in
            0)
                echo -n "cmp -- "
                printf %q "${REPLY}"
                echo -n ' '
                printf %q "${OTHER}"
                echo ""
                break
                ;;
            2)
                echo "\`cmp\` failed!"
                exit 2
                ;;
            *)
                :
                ;;
        esac
    done
done

The result is a set of commands you can run to check that the result is correct :)

Edit: The last version works with really weird filenames like:

$'/tmp/--$`\\! *@ \a\b\E\E\f\r\t\v\\"\' \n'
share|improve this answer
    
Brilliant, thank you! – adrian Mar 18 '11 at 13:00
    
Nice work, but I see three issues: 1) this is O(n^2), meaning it will be really slow on big dirs. 2) it seems it will do both checks 'does a = b' and 'does b = a' 3) your check for same file name means it won't check at all if the file exists in both a dir and subdir. e.g. DIR/fred and DIR/subdir/fred will not be compared.hows u – Rich Homolka Mar 18 '11 at 16:08

This will find files under a dir with dupes. It's pretty raw, but it works.

#!/bin/bash

CKSUMPROG=md5sum
TMPFILE=${TMPDIR:-/tmp}/duplicate.$$
trap "rm -f $TMPFILE" EXIT INT

if [ ! -d "$1" ]
then
    echo "usage $0 directory" >2
    exit 1
fi

PRINTBLANK=
# dump fingerprints from all target files into a tmpfile
find "$1" -type f 2> /dev/null | xargs $CKSUMPROG  > $TMPFILE 2> /dev/null

# get fingerprints from tmpfile, get the ones with duplicates which means multiple files with same contents
for DUPEMD5 in $(cut -d ' ' -f 1 $TMPFILE | sort  | uniq -c | sort -rn | grep -v '^  *1 ' | sed 's/^ *[1-9][0-9]* //')
do
    if [ -z "$PRINTBLANK" ]
    then
        PRINTBLANK=1
    else
        echo
        echo
    fi

    grep "^${DUPEMD5} " $TMPFILE | gawk '{print $2}'
done
share|improve this answer
    
Clever. Too bad it starts over calculating those checksums everytime. I guess that part is left to the reader ;-) – deepc Mar 19 '12 at 0:29

it works for me on my mac, you will catch the duplicate file by their md5 value:

find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort
share|improve this answer
    
after sort if you add | uniq -d -f 1 -s 33 it will output just the duplicates – Tim Reddy Jan 31 '14 at 23:16

If only interested in files in the current directory (as indicated by OP), then this is the simpliest. For linux and Windows (msys - tested, or MinGW or Cygwin with any with GnuWin32). This will list all duplicates.

md5sum * | sort | uniq -D -w 32

For BSD/Mac OS X (it will list only the first duplicate)

md5 -r * | sort | uniq -d -w 32
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .