From Finding and removing duplicate files in osx with a script, can this be extended to find duplicates across the whole hard drive instead of per folder?

I know there are duplicates but they aren't necessarily in the same folder.

  • 1
    Unless you have a tiny hard drive then the answer is probably no. In fact, even the solution you've posted is quite likely to fail quite often. It is possible for a checksum on two entirely different files to be identical. A checksum is meant to be a quick check for data integrity when reading a file from a drive; it is not meant to be a method for comparing file data. The user of this would quickly find out why those are different concepts when they try this script on an entire hard drive.
    – krowe
    Sep 27 '13 at 18:32

Try using fdupes instead:

brew install fdupes
fdupes -rn .

-r descends to subdirectories recursively and -n ignores empty files. Add -d to be prompted for which files to keep, or add -dN to always keep the first file and delete other files.

When I tried running fdupes -rn /, it took about four hours, but I have an HDD with about 1 TB of files.

fdupes first compares file sizes and MD5 checksums, and if they are identical, it performs byte-by-byte comparisons. MD5 collisions are possible, but they do not occur in practice unless a file is created to deliberately have the same MD5 digest as another (non-identical) file. Even then, fdupes does also do a byte-by-byte comparison.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.