Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Background

I have backups of a website which stores all of it's data into a single file. This file is several gigs large and I have many different backups of this file. Most of the data within is mostly the same plus whatever was added or changed to it.

I want to keep all the concurrent backups I've made through the years in case I find a horrible surprise of data corruption along the line. However storing a 10gig file every month gets expensive.

Seeking Solution

I've often thought about different ways of alleviating this problem. One thought that comes up very often combines the idea of a duplicating file system which doesn't require it's own partitioned volume on a hard drive. Something like what truecrypt does, what it calls, "file hosted containers" which when using the truecrypt program allows you to mount and dismount that volume as a regular hard drive.

Question

Is there a virtual hard drive mounter which uses file-based container which uses data deduplicaiton file system?

(This question is a little awkward to put into words, if you have a better idea on how to ask this question please feel free to help out.)

share|improve this question
    
Hmm... Potentially this?: docs.oracle.com/cd/E23120_01/html/821-2854/… –  Mallow Nov 7 '13 at 23:56
    
Can I do that on linux, requires more reading –  Mallow Nov 8 '13 at 0:00
add comment

3 Answers 3

Use ZFS or BTRFS filesystems or OpenDEDUP.

I should also note that you can create "disks" in files on linux and mount them with the loopback (mount -o loop ...) device; thus them being virtual.

You may be better off just loopback mounting a ZFS formatted file; since ZFS is pretty much the defacto when it comes to deduplication. If you don't know how to do this see here.

share|improve this answer
    
its been eons since I looked at it, but opendedup runs on a file image. In addition, zfs needs a load of resources for online deduplication, so it might not be the best option if you're not willing to do some performance tuning and/or to throw some hardware at it, IIRC –  Journeyman Geek Nov 8 '13 at 0:26
    
@JourneymanGeek although true in regards to ZFS it is the most complete and defacto FS used with FreeBSD, Solaris, and Illumos. And, ZFS is the direction that modern filesystems are heading towards (btrfs, and ReFS for example). –  josten Nov 8 '13 at 5:37
add comment

While it dosen't help for the data you have so far, you really ought to be looking for something like rsnapshot , or even simply rsync to make incremental backups. While deduplication is very shiny, and awesome, needing to check through every block and comparing it then deduplicating similar files is heavy. Doing incremental backups at backup time makes much more sense.

share|improve this answer
    
It needs not be heavy, it's a matter of implementation. When data is copied into the container, it can as well be hashed and the hashes then stored as a meta data. The next time data is copied in, the program would get a block, hash it, look up the hash in the meta store, see if it's already there and store the block accordingly. There's a program called Bvckup that employs parallel reading and hashing and it manages to be faster than a straight copy. –  Angstrom Nov 8 '13 at 9:31
add comment

A binary patch utility can produce a patch file which if most of the two files are the same is pretty small. You can pick pairs of files, generate a patch and delete the target and just save the source file plus patch file.

I have used xdelta for this purpose.

xdelta delta JanFile FebFile JanToFebPatch

xdelta delta JanFile MarFile JanToMarPatch

Works well if you do full backup + several incremental based on the full. Various options to speed things up or reduce memory usage.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.