Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From Finding and removing duplicate files in osx with a script, can this be extended to find duplicates across the whole hard drive instead of per folder?

I know there are duplicates but they aren't necessarily in the same folder.

share|improve this question
1  
Unless you have a tiny hard drive then the answer is probably no. In fact, even the solution you've posted is quite likely to fail quite often. It is possible for a checksum on two entirely different files to be identical. A checksum is meant to be a quick check for data integrity when reading a file from a drive; it is not meant to be a method for comparing file data. The user of this would quickly find out why those are different concepts when they try this script on an entire hard drive. – krowe Sep 27 '13 at 18:32

Try using fdupes instead:

brew install fdupes
fdupes -rn .

-r descends to subdirectories recursively and -n ignores empty files. Add -d to be prompted for which files to keep, or add -dN to always keep the first file and delete other files.

When I tried running fdupes -rn /, it took about four hours, but I have an HDD with about 1 TB of files.

fdupes first compares file sizes and MD5 checksums, and if they are identical, it performs byte-by-byte comparisons. MD5 collisions are possible, but they do not occur in practice unless a file is created to deliberately have the same MD5 digest as another (non-identical) file. Even then, fdupes does also do a byte-by-byte comparison.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .