Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.

share|improve this question
3  
Please define "duplicate" - Binary equivalent, text equivalent, same size, same name? –  l0b0 Mar 18 '11 at 12:33
    
This stuff also looks useful, finding dupes with a single line, even considering a check for same size first. commandlinefu.com/commands/view/3555/… –  deepc Mar 19 '12 at 0:46
add comment

3 Answers

up vote 5 down vote accepted

Don't know about Mac compatibility, but this Works For Me(TM):

#!/bin/bash
[ -n "$1" ] || exit 1
exec 9< <( find "$1" -type f -print0 )
while IFS= read -r -d '' -u 9
do
    file_path="$(readlink -fn -- "$REPLY"; echo x)"
    file_path="${file_path%x}"
    exec 8< <( find "$1" -type f -not -path "$file_path" -print0 )
    while IFS= read -r -d '' -u 8 OTHER
    do
        cmp --quiet -- "$REPLY" "$OTHER"
        case $? in
            0)
                echo -n "cmp -- "
                printf %q "${REPLY}"
                echo -n ' '
                printf %q "${OTHER}"
                echo ""
                break
                ;;
            2)
                echo "\`cmp\` failed!"
                exit 2
                ;;
            *)
                :
                ;;
        esac
    done
done

The result is a set of commands you can run to check that the result is correct :)

Edit: The last version works with really weird filenames like:

$'/tmp/--$`\\! *@ \a\b\E\E\f\r\t\v\\"\' \n'
share|improve this answer
    
Brilliant, thank you! –  adrian Mar 18 '11 at 13:00
    
Nice work, but I see three issues: 1) this is O(n^2), meaning it will be really slow on big dirs. 2) it seems it will do both checks 'does a = b' and 'does b = a' 3) your check for same file name means it won't check at all if the file exists in both a dir and subdir. e.g. DIR/fred and DIR/subdir/fred will not be compared.hows u –  Rich Homolka Mar 18 '11 at 16:08
add comment

This will find files under a dir with dupes. It's pretty raw, but it works.

#!/bin/bash

CKSUMPROG=md5sum
TMPFILE=${TMPDIR:-/tmp}/duplicate.$$
trap "rm -f $TMPFILE" EXIT INT

if [ ! -d "$1" ]
then
    echo "usage $0 directory" >2
    exit 1
fi

PRINTBLANK=
# dump fingerprints from all target files into a tmpfile
find "$1" -type f 2> /dev/null | xargs $CKSUMPROG  > $TMPFILE 2> /dev/null

# get fingerprints from tmpfile, get the ones with duplicates which means multiple files with same contents
for DUPEMD5 in $(cut -d ' ' -f 1 $TMPFILE | sort  | uniq -c | sort -rn | grep -v '^  *1 ' | sed 's/^ *[1-9][0-9]* //')
do
    if [ -z "$PRINTBLANK" ]
    then
        PRINTBLANK=1
    else
        echo
        echo
    fi

    grep "^${DUPEMD5} " $TMPFILE | gawk '{print $2}'
done
share|improve this answer
    
Clever. Too bad it starts over calculating those checksums everytime. I guess that part is left to the reader ;-) –  deepc Mar 19 '12 at 0:29
add comment

it works for me on my mac, you will catch the duplicate file by their md5 value:

find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort
share|improve this answer
    
after sort if you add | uniq -d -f 1 -s 33 it will output just the duplicates –  Tim Reddy Jan 31 at 23:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.