I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, there may a few hundred photos that were copied from one computer to the other as an ad-hoc backup. Now that I want to make a common repository of files, I don't want several copies of the same photo.

If I copy all of these files to a single directory, is there a tool that can go thru and recognize duplicate files and give me a list or even delete one of the duplicates?


Create an md5sum of each file, duplicates md5sums suggest (but doesn't guarantee) duplicate files.

  • 1
    they do strongly "suggest", however :) ..and if the files are the same size, the likelihood of the suggestion being correct is exceptionally high – warren Aug 25 '10 at 3:31

You could use dupemerge to turn the identical files into hardlinks. It'll take a very long time on a large file set though. SHA (or MD5) hashes of the files will almost certainly work faster, but you'll have to do more legwork in finding the duplicates. The probability of accidental collision is so low that in reality you can ignore it. (In fact, many deduplication products already do this.)

Your best bet for dealing with photos and music is to get tools tailored to finding duplicates of those items in particular. Especially since you may not have files that are identical at a binary level after things like tagging or cropping or encoding differences come into play. You'll want tools that can find photos that "look" the same and music that "sounds" the same even if minor adjustments have been made to the files.

  • 1
    hardlink is very fast (took 200 seconds for 1.2M files, 320GB) – Marcel Waldvogel Feb 5 '17 at 19:27

Well, if you have the ability, you can set up a deduplicating filesystem and put your backups on that. This will not only deduplicate whole files, but also similar pieces of files. For example, if you have the same JPEG in several places, but with different EXIF tags on each version, a deduplicating filesystem would only store the image data once.

Deduplicating filesystems include lessfs, ZFS, and SDFS.

  • 1
    I can't speak about deduplication on SDFS or lessfs, but Dedupe on ZFS does not work like this. It checks on the block level, so identical files will likely be flagged for dedupe, but files that have a different header will likely not. Additionally, files smaller than your ZFS recordsize (default is 128KB) can be packed in a single block together, so they may not benefit from dedupe either. – notpeter Jan 24 '14 at 0:19
  • Deduplicated filesystem (zfs or btrfs) only saves space on backup host. It doesn't save amount of transferred data. It is some idea to compute some hash for files and send it to backup host. when it says "I have it", backup for that file is done. This is implemented on very expensive (money) backup software. I never heard about it on open source. But maybe it is implemented at now, who knows :) – Znik Jun 24 '14 at 12:54
  • @Znik rsync --link-dest is your friend if you want to save on read/transfer as well. A simple introduction is here: netfuture.ch/2013/08/… – Marcel Waldvogel Feb 5 '17 at 19:30
  • it is working by testing inodes assigned to a file. but when deduplication is hidden, for example in squashfs filesystem, then rsync can't check this and will transfer the same data many times. – Znik Feb 22 '17 at 8:30

When I was doing this kind of thing, I learned that it's a lot more engaging/time-efficient to actually just go through the files yourself in your free time, over the course of a couple weeks. You can tell the difference between things way better than your computer can.

If you don't agree, then I suggest EasyDuplicateFinder. As I mentioned above, though, it'll take a long time, say, about a day for 5GB of files.

And on another note, Crashplan does what you were doing before, but in a much more organized, non-versioning-problem way.


Another possibility, presuming the machines you're backing-up will support it, is to use something like rsync.

If you rsync from A to B, then from C to B, then from D to B, etc, exact duplicates (ie, by filename) will be eliminated (and synchronized between the machines you're backing up).

If you don't want them all synchronized to each other, however, this is not the best way to go.


For image files, use findimagedupes. It's also packaged in debian.

  • findimagedupes is for photos or graphics, maybe converted from one format to another. photos (i home) is only example in question. It may be any other large files. – Znik Jun 24 '14 at 12:50

Hard links only perform deduplication if the entire file is identical. If headers (EXIF, ID3, …) or metadata (owner) differ, they will not be linked.

When you have a chance of using a file system with block deduplication support (ZFS, btrfs, …) use that instead. I am very fond of the offline (aka batch) dedup support of btrfs, which supports extent-level deduplication and does not constantly consume huge amounts of memory (as ZFS online dedup).

Deduplication also has the advantage that files can be modified by the user without the other copy noticing (which might not be applicable in your case, but in others).

See https://btrfs.wiki.kernel.org/index.php/Deduplication for an excellent discussion.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.