0

I started out writing a simple bash shell script to find duplicate files within a given directory using SHA1 to detect matches. Everything worked until I encountered file names with spaces. The detection still works, but in the output the spaces are converted to linefeeds.

The script...

#!/bin/bash
export TARGET=$1
find $TARGET -type f -exec openssl sha1 \{\} \; > ./dupes.txt
COUNT=-1
for EVALUATION in `cat ./dupes.txt | sed 's/SHA1(\(.*\))\= \(.*\)$/\2 \1/' | awk '{print $1}' | sort | uniq -c | sort -nr`
do
    if [[ $COUNT == -1 ]]
    then
        COUNT=$EVALUATION
    else 
        HASH=$EVALUATION
        if [[ $COUNT == 1 ]]
        then
            break
        fi
        echo "--- duplicate set ---"
        for FILE in `grep $HASH ./dupes.txt | awk -F"[()]+" '{print $2}'`
        do
            echo "$FILE"
        done
        echo "---------------------"
        COUNT=-1
    fi
done

Run the script like...

./dupes.sh /home/dacracot/testDupes

It will create a file dupes.txt that looks something like...

SHA1(/home/dacracot/testDupes/lP3wj.jpg)= 324d91f412745481ed38aa184e5a56bfc3bf43b5
SHA1(/home/dacracot/testDupes/1673.gif)= 9c4029ec2e310f202b413d685209373d234e5465
SHA1(/home/dacracot/testDupes/.DS_Store)= b0ae6631a1412863f958da64091f4050005bf8d6
SHA1(/home/dacracot/testDupes/tae 2.svg)= 3ddc4fd6ae505bd01f370d0a018ef1f84b4d8011
SHA1(/home/dacracot/testDupes/tae.graffle)= 77f1ad6d695d944abacfe3a7f196be77125b6ef6
SHA1(/home/dacracot/testDupes/tae.svg)= 3ddc4fd6ae505bd01f370d0a018ef1f84b4d8011
SHA1(/home/dacracot/testDupes/22402_graph.jpg)= 24e5a25c8abf322d424dd5ce2e5b77381cd001c4
SHA1(/home/dacracot/testDupes/forwardcont.jpg)= 981e75060ae8e3aad2fe741b944d97219c8ccbe5
SHA1(/home/dacracot/testDupes/tae.svg.gz)= 922af5a5adbf7a4e7fd234aac7bcee2986133c4d
SHA1(/home/dacracot/testDupes/Alt2012.pdf)= 97d1fd997df9eb310b30a371c53883f5227cf10a
SHA1(/home/dacracot/testDupes/vcBZ8.jpg)= 7553c19fcb6aa159aada2e38066b5ba84465ee57
SHA1(/home/dacracot/testDupes/derm.graffle)= 0e1c4032f5f1fadc3a1643b2b77f816011c2d67f
SHA1(/home/dacracot/testDupes/WA.png)= 0e2e77624c3a76da4816f116665a041f6bdced2d
SHA1(/home/dacracot/testDupes/DRAW.GIF)= 6a8e4a2bf413e84140a0edeb40b475a5d3e4c255
SHA1(/home/dacracot/testDupes/crazyTalk.gif)= 1d938bbcb8cf09f30492df4504a50348cef7ea9d

And finally output that looks like...

--- duplicate set ---
/home/dacracot/testDupes/tae
2.svg
/home/dacracot/testDupes/tae.svg
---------------------

But as you can see from the first file, the output should be...

--- duplicate set ---
/home/dacracot/testDupes/tae 2.svg
/home/dacracot/testDupes/tae.svg
---------------------

What is changing the space to a linefeed?

3
  • 1
  • @KamilMaciorowski Such arbitrary rules result in reducing the utility of Stack Exchange sites. What exactly is wasted? What is the down side compared to the upside of potentially finding someone who can help?
    – dacracot
    May 23, 2019 at 14:39
  • 1
    My attitude: Stack Exchange gives you the ability to find someone who can help but this is not its main purpose. The main purpose is to build a knowledge base. Answers should be helpful for many users (including future ones) who happen to have similar problems. By posting on two sites you created a situation where there is one answer here and another distinct one there. Fellow users will have to find two questions to know possible answers. Which answer is (probably) the best? There is no single community able to read both, compare and favor one by voting. May 23, 2019 at 20:42

1 Answer 1

1

May I make some modifications to your script to simplify it and also eliminate your problem?

I see you are using OpenSSL to calculate the SHA1 hash, and then parsing out some unnecessary output and proceeding to sort and iterate over the list of hashes. You don't specify an OS tag in your post, but your use of /bin/bash suggests Linux, so why not use sha1sum instead? It produces simpler output which is easier for your script to process. (Readers using BSD can use sha1 -r to get equivalent output.)

#!/bin/bash

find "$1" -type f -exec sha1sum \{\} \; > dupes.txt

awk '{print $1}' < dupes.txt |
  sort | uniq -c | sort -nr |
  while read COUNT HASH; do
    if [[ $COUNT == 1 ]]; then
      break
    else
      echo "--- duplicate set ---"
      grep "^$HASH " dupes.txt | sed -e "s/[^ ]* //"
      echo "---------------------"
    fi
  done

There are further optimizations that could be done, such as using find's -print0 option and xargs, but hopefully the revisions above will get you started.

1
  • Nicely done. You reduced the complexity and fixed the issue.
    – dacracot
    May 26, 2019 at 14:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .