Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

If a cloud backup provider (for example) is storing laptop backups, some files will be identical (Windows or Mac OS system libraries, for example).

The best way to store these would be to have a single copy for all of them. So, when a file is uploaded, a digest of the file could be computed and then matched to a stored copy matching that digest.

is there any product out there to do specifically this? (I know key-value stores can achieve it, but they were not designed or optimized for it)

Thanks for any feedback!

share|improve this question
    
Many products do this internally, including Dropbox. They don't usually advertise this because it's invisible to the user. –  David Schwartz Oct 17 '11 at 23:04

2 Answers 2

up vote 2 down vote accepted

What you're looking for is a deduplicating file system or single instance storage. Apparently some server versions of windows support it.

Alternatively you can look at sdfs/opendedup, which does file level deduplication and ZFS which does block level deduplication.

share|improve this answer
    
btrfs also has deduplication patches that will be merged in to the core in the future –  Paul Oct 17 '11 at 23:16

Plan 9 Venti/Fossil implemented this (long ago), here's some more information

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.