Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a set of backups of mostly photos. The directory looks sort of like this:

backup/Day1/photos/1.jpg
               .../2.jpg
backup/Day2/photos/2.jpg
               .../3.jpg
               .../4.jpg
backup/DayN/photos/2.jpg
               .../3.jpg
               .../9.jpg

Files with same name are identical. There are many duplicates. Due to the way the backup system works, it's not possible to create incremental backup directly. I always get the entire dump each day.

If I want to create a compressed archive for a date range, say Day 5~9, what is the best tool/compression algorithm to do that, which does deduplication well? (I'm not counting on compressing the jpeg itself)

share|improve this question
    
There's no best compression algorithm when you're attempting to compress files that are already compressed. –  slhck Nov 10 '12 at 22:36
    
Which OS? Also, what are you looking to do - skip dupes inside the archive so only a single copy of each file exists, but extraction restores the multiple copies? As for "best" algo, that might keep changing but there are definitely some algos/tools that use specialised models for JPEG compression and consequently can handle them better than others. –  Karan Nov 10 '12 at 22:38
    
I'm looking for algorithm that does deduplication well, not jpeg compression. –  Wei Shi Nov 11 '12 at 4:12

1 Answer 1

up vote 0 down vote accepted

ZPAQ is a little known archiver which does deduplication, and, AFAIK, the only one. Because ZPAQ also does versioning, you can keep every single backup in a single archive, reducing space usage even more.

Or, you could use a version control system, such as Git, which does file deduplication automagically.

Or, if you want to something more complicated, you could tinker with Btrfs or ZFS, which are filesystems with deduplication built-in. Btrfs' deduplication is not as good as ZFS' though.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.