Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If a cloud backup provider (for example) is storing laptop backups, some files will be identical (Windows or Mac OS system libraries, for example).

The best way to store these would be to have a single copy for all of them. So, when a file is uploaded, a digest of the file could be computed and then matched to a stored copy matching that digest.

is there any product out there to do specifically this? (I know key-value stores can achieve it, but they were not designed or optimized for it)

Thanks for any feedback!

share|improve this question
Many products do this internally, including Dropbox. They don't usually advertise this because it's invisible to the user. – David Schwartz Oct 17 '11 at 23:04
up vote 2 down vote accepted

What you're looking for is a deduplicating file system or single instance storage. Apparently some server versions of windows support it.

Alternatively you can look at sdfs/opendedup, which does file level deduplication and ZFS which does block level deduplication.

share|improve this answer
btrfs also has deduplication patches that will be merged in to the core in the future – Paul Oct 17 '11 at 23:16

Plan 9 Venti/Fossil implemented this (long ago), here's some more information

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .