2

I am attempting to make compressed incremental backups of a Windows partition using ntfsclone on Linux. Currently, I am making full ntfsclone images using something like the following command:

sudo ntfsclone -s -t -o - /dev/sda2 | xz -zc > new-backup.ntfsclone.xz

What I would like to do is perform incremental backups by creating a delta file using the previous backup and the ntfsclone output stream rather than create a new ntfsclone image each backup session.

I understand that creating diffs of binaries is not easy, and that several tools (rdiff, xdelta, bsdiff, etc.) exist for this purpose; however, they all require the two source files to be present on disk and cannot compare data directly from output streams.


In other words, are there any solutions for binary streams that do something like the following command?

diff -u <(xz -dc old-backup.ntfsclone.xz) \
<(sudo ntfsclone -s -t -o - /dev/sda2) \
| xz -zvc > new-backup.ntfsclone.delta.xz

Obviously, the above command doesn't work, but I would like to be able to decompress the old image, compare it to an ntfsclone representation of the Windows partition, and create a compressed delta image from the comparison in a single step.


A few conditions are:

  • Process must use ntfsclone
  • Deltas must be patchable directly to disk
  • Preferably, everything takes place in a single step

It is also possible (and may be preferable) to create a delta file by restoring the ntfsclone image to pipe and creating a diff by comparing directly to /dev/sda2, as this would likely create a smaller output file, but doing so would complicate filtering out unallocated disk space.

(That is, we would have to zero out unused disk space or do something like diff -u <(xz -dc old-backup.ntfsclone.xz | ntfsclone -r -t -o - -) <(sudo ntfsclone -t -o - /dev/sda2) | xz -zc > new-backup.img.delta.xz to prevent archiving all of the deleted file data. Yuck!)

Any takers?


NOTE: A tool called ntfsddup by Wei Dong does exist for accomplishing almost exactly this, but I am having a hard time getting the darn thing to work. The software is in VERY early stages and needs a LOT of work.


2 Answers 2

2

I have successfully used a tool called bup (based on git) which can efficiently store large binaries that differ only slightly. It also works very well with nfs, which is convenient for me because I have an nfs server with a lot of space where I keep data:

for example, to backup ntfs images from my laptop (diaz)

export BUP_DIR=/mnt/t3/big1/diaz/ntfsbup
ntfsclone -s -o - /dev/sda2 | bup split -n sda2.img

to restore (to a different drive):

bup join sda2.img | ntfsclone -r -O /dev/sdb9 -

or a previous version:

bup join sda2.img^ | ntfsclone -r -O /dev/sdb9 -

or

bup join sda2.img~2 | ntfsclone -r -O /dev/sdb9 -
1

A while ago, I wrote a program which creates efficient deltas between ntfsclone images.

https://github.com/koljanowak/ntfscloneimgdelta

It was particularly easy because the file format of the ntfsclone images is pretty straight forward. They are in essence just a series of bytes saying either '1: here comes the next ntfs cluster', followed by one ntfs cluster, or '0: the next cluster was unused', followed by nothing. My delta file uses exactly the same file format, adding a third case '2: the next cluster was identical in both images', followed by nothing.

Usage: ntfscloneimgdelta delta OLDFILE [NEWFILE [DELTA]]
       ntfscloneimgdelta patch OLDFILE [DELTA [NEWFILE]]

OLDFILE and NEWFILE do not need to be in ascending chronological order, you can just swap them to create reverse deltas. This enables you to always have the latest backup as a full dump, and keeping older dumps as deltas.

Omitting file names or replacing them with '-' uses stdin or stdout. This way, you can take a new dump of a partition and create a reverse delta between this dump and the previous dump in one go.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .