find duplicate files by size and MDF

Question

I am writing script to search for dups files and group them by size and then by their MD5 check sum.

here is my script:

find . -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 > ./dups.txt

I want to make sure my script generate my dups files by their size and MD5 Correctly.

is there anyway that I can make sure my result is correct or will you help me to check my script and let me know, please ?

Answer 1

Here's a similar duplicate-finding tool, in Perl, under the GPL:

https://github.com/omps/myperlscripts/blob/master/dup-files.pl

You may also wish to explore the -exec option to find(1).

Answer 2

I recommend you to use fdupes, it's designed for this purpose.

Description :

a program for identifying or deleting duplicate files residing within specified directories