2

If a cloud backup provider (for example) is storing laptop backups, some files will be identical (Windows or Mac OS system libraries, for example).

The best way to store these would be to have a single copy for all of them. So, when a file is uploaded, a digest of the file could be computed and then matched to a stored copy matching that digest.

is there any product out there to do specifically this? (I know key-value stores can achieve it, but they were not designed or optimized for it)

Thanks for any feedback!

Improve this question
1
  • Many products do this internally, including Dropbox. They don't usually advertise this because it's invisible to the user.
    – David Schwartz
    Oct 17, 2011 at 23:04

2 Answers 2

Reset to default
2

What you're looking for is a deduplicating file system or single instance storage. Apparently some server versions of windows support it.

Alternatively you can look at sdfs/opendedup, which does file level deduplication and ZFS which does block level deduplication.

Improve this answer
1
  • btrfs also has deduplication patches that will be merged in to the core in the future
    – Paul
    Oct 17, 2011 at 23:16
2

Plan 9 Venti/Fossil implemented this (long ago), here's some more information

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .