Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to simply backup and archive the files on several machines. Unfortunately, the files have some large files that are the same file but stored differently on different machines. For instance, there may a few hundred photos that were copied from one computer to the other as an ad-hoc backup. Now that I want to make a common repository of files, I don't want several copies of the same photo.

If I copy all of these files to a single directory, is there a tool that can go thru and recognize duplicate files and give me a list or even delete one of the duplicates?

share|improve this question
1  
related: unix.stackexchange.com/questions/3037/… –  David Cary Mar 16 '11 at 23:57
add comment

6 Answers

Create an md5sum of each file, duplicates md5sums suggest (but doesn't guarantee) duplicate files.

share|improve this answer
1  
they do strongly "suggest", however :) ..and if the files are the same size, the likelihood of the suggestion being correct is exceptionally high –  warren Aug 25 '10 at 3:31
add comment

You could use dupemerge to turn the identical files into hardlinks. It'll take a very long time on a large file set though. SHA (or MD5) hashes of the files will almost certainly work faster, but you'll have to do more legwork in finding the duplicates. The probability of accidental collision is so low that in reality you can ignore it. (In fact, many deduplication products already do this.)

Your best bet for dealing with photos and music is to get tools tailored to finding duplicates of those items in particular. Especially since you may not have files that are identical at a binary level after things like tagging or cropping or encoding differences come into play. You'll want tools that can find photos that "look" the same and music that "sounds" the same even if minor adjustments have been made to the files.

share|improve this answer
add comment

Well, if you have the ability, you can set up a deduplicating filesystem and put your backups on that. This will not only deduplicate whole files, but also similar pieces of files. For example, if you have the same JPEG in several places, but with different EXIF tags on each version, a deduplicating filesystem would only store the image data once.

Deduplicating filesystems include lessfs, ZFS, and SDFS.

share|improve this answer
    
I can't speak about deduplication on SDFS or lessfs, but Dedupe on ZFS does not work like this. It checks on the block level, so identical files will likely be flagged for dedupe, but files that have a different header will likely not. Additionally, files smaller than your ZFS recordsize (default is 128KB) can be packed in a single block together, so they may not benefit from dedupe either. –  notpeter Jan 24 at 0:19
add comment

When I was doing this kind of thing, I learned that it's a lot more engaging/time-efficient to actually just go through the files yourself in your free time, over the course of a couple weeks. You can tell the difference between things way better than your computer can.

If you don't agree, then I suggest EasyDuplicateFinder. As I mentioned above, though, it'll take a long time, say, about a day for 5GB of files.

And on another note, Crashplan does what you were doing before, but in a much more organized, non-versioning-problem way.

share|improve this answer
add comment

Another possibility, presuming the machines you're backing-up will support it, is to use something like rsync.

If you rsync from A to B, then from C to B, then from D to B, etc, exact duplicates (ie, by filename) will be eliminated (and synchronized between the machines you're backing up).

If you don't want them all synchronized to each other, however, this is not the best way to go.

share|improve this answer
add comment

For image files, use findimagedupes. It's also packaged in debian.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.