Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I make regular snapshots of my VM using a nightly script. These backups are compressed using WinRAR and do shrink considerably, but I suspect it's not as efficient as had the file had been deduplicated first (just a hunch which I'm hoping to test).

So instead of compressing the VHD itself, I would like to deduplicate the single file first, and then compress the output of the deduplicator.

Is anyone aware of such a CLI tool?

share|improve this question
How do you mean to deduplicate 1 file? Can you explain? – Notitze Jan 8 '11 at 9:52
@Notitze by removing sections of the file that repeat. You might call this poor man's compression (which remove redundancy in general), but for files that have massive amounts of repetition a general-purpose compressor might not remove as much information as a deduplicator might. – romkyns Jan 8 '11 at 11:09
I doubt that to be the case, really. The point of a compression tool is to eliminate duplication in a file, and trades off execution time in favor of compression ratio compared to a deduping fs. If the VM contains lots of empty space, zeroing out the free space from in the VM and compacting the virtual disk will probably help more than anything else. – afrazier Jan 8 '11 at 13:34
@afrazier the empty space is of course zeroed and the disk gets compacted. I am yet to find a production quality free deduping FS for Windows. Existing compressors like WinRAR are pretty bad at handling long-range duplication. Try it yourself: make 10 copies of the same 5MB mp3 file, and compress at max settings using WinRAR. You will get a 50MB file, alas. – romkyns Jan 8 '11 at 17:25
@romkyns That might have been because my file size was within the dictionary size. Duh me. :-) Perhaps try making a shadow copy of the VHD (suspend VM, snapshot, restart VM) and back that up -- that'd let you extend your backup window and reduce downtime. Adi Oltean's Blog has some useful scripts and information on getting it done. – afrazier Jan 10 '11 at 2:05

You can try OpenDedup's SDFS. It's not exactly what you're looking for, but probably as close as you'll get with Windows.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.