Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to find all duplicate files (based on MD5 hash) and ordered by file size. So far I have this:

find . -type f -print0 | xargs -0 -I "{}" sh -c 'md5sum "{}" |  cut -f1 -d " " | tr "\n" " "; du -h "{}"' | sort -h -k2 -r | uniq -w32 --all-repeated=separate

The output of this is:

1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./
1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./
1832348bb0c3b0b8a637a3eaf13d9f22 4.0K   ./picture2.s

d41d8cd98f00b204e9800998ecf8427e 0      ./test(1).log

Is this the most efficient way?

share|improve this question

migrated from Oct 15 '12 at 3:32

This question came from our site for professional and enthusiast programmers.

There are several tools for deduplication, don't reinvent the wheel. – Paulo Scardine Oct 14 '12 at 21:37
Ok, that's a fair point. But looking at this as a learning exercise for linux cmd, can this be improved? For instance, originally I started off with -exec 'md5sum.....' but research found (using google) xargs was more efficient. – Jamie Curran Oct 14 '12 at 22:00
If you want to learn new techniques, I suggest looking how these tools are solving the problem and you will get a lot of clever ideas (the source, Luke, use the source). – Paulo Scardine Oct 14 '12 at 22:06
See also – artfulrobot May 20 '15 at 14:39
up vote 3 down vote accepted

From "man xargs": -I implies -L 1 So this is not most efficient. It would be more efficient, if you just give as many filenames to md5sum as possible, which would be:

find . -type f -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Then you won't have the file size of course. If you really need the file size, create a shell script, which does the md5sum and du -h and merge the lines with join.

share|improve this answer

Sometimes we are working on reduced sets of linux commands, like busybox or other things that comes with NAS and other linux embedded hardwares (IoTs). In these cases we can't use options like -print0, getting troubles with namespaces. So we may prefer instead:

find | while read file; do md5sum "$file"; done > /destination/file

Then, our /destination/file is ready for any kind of process like sort and uniq as usual.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.