1

I found the following command that makes use of md5 file hashes to find duplicate files and list them in a text file.

find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sort | tee duplicates.txt

However, since every file with the same hash is listed, I am unable to use the following command to remove the duplicate files as every line in the text file would be deleted including the original.

while read file; do rm "$file"; done < duplicates.txt

Is there a way to remove all but the first file with a duplicate md5 hash in the duplicates.txt file?

1

1 Answer 1

0

fdupes may be your solution. It can be installed via homebrew with

brew install fdupes

If you don’t have homebrew, you can install homebrew with

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

You can use the -d option to be prompted which files to keep (deleting all others), and the -N option deletes all but the first without prompting. -r recursively searches the specified directory(ies).

So fdupes -rdN <directory> should delete all but the first in each set of duplicates. Be careful when deleting without prompting, of course. Maybe copy the directory first if the files are important.

I’m sorry to suggest a third-party tool; I usually try to find a way to do things without additional software, but fdupes is the only way I know. I’ll think of a way to use your md5 list for the same purpose.

1
  • Thanks for suggesting this tool. For photos and videos it is useful to use fdupes -rdiN <directory> because sometimes the filename contains a space and a number extra and you want to preserve the original file that's actually sorted last.
    – adamkonrad
    Commented May 19 at 4:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .