14

I am doing a backup of my work and I'm looking for a program that can more efficiently compress files. The windows default compression program works fine, but it doesn't work as "smart" as i would like.

My files are basically entire snapshots of my work taken over the past few months as I do my regular complete backups. not a whole lot changes between the backups and many files are simply copies of each other.

My entire file set is approximately 15GB uncompressed, while individual snapshot backups (compressed using standard compression programs) are approximately 150 to 200 MB each (they typically increase in size with more recent dates). The total size of all the compressed snapshots is about 3GB. Due to the nature of the files, I expect a "smart" compression program to take the entire volume down to around 400-500 MB, that is, taking into account reasonable inefficiencies and extra space to compress the differences in the files.

Is there a compression program that will take advantage of the fact that many files are identical to achieve a much higher compression ratio than the standard windows compression program?

Thanks -Faken

Edit: Just finished a best compression of the entire volume using standard windows compression. It is only 10MB better than the compressed snapshots combined. In my opinion that is unacceptable knowing the nature of the files.

3
  • On what version of windows?
    – user1931
    Sep 4, 2009 at 6:12
  • 1
    Heh, its vista...
    – Faken
    Sep 4, 2009 at 6:16
  • 4
    Wouldn't switching to version control instead of those backups be better? I'm using git also for binary files and it works really well. I don't think it could compress it as good as 7z, but does it matter if it takes 150 MB or 300 MB? If not, I'd go for the easy of use and flexibility.
    – maaartinus
    Mar 14, 2011 at 5:10

4 Answers 4

19

7-zip supports solid compression if I remember correctly, so it should compress a lot of nearly identical files very well.

5
  • 2
    +1, and consider the default 7z format.
    – nik
    Sep 4, 2009 at 6:12
  • I'll give it a try, I'll get back to you with the results.
    – Faken
    Sep 4, 2009 at 6:12
  • 4
    After 1 hour of compression at the highest settings I got a file size of 133MB. This is very impressive! My original aggressive estimate (which I didn't state) was a file size of about 150MB (this is opposed to my stated conservative estimate of 400-500MB). Looks like I'll be converting to 7z from now on!
    – Faken
    Sep 4, 2009 at 7:17
  • Even if you need to stick to standard Zip format, 7-zip has a better yield than other popular compression tools (WinRAR, WinZip) with the highest efficiency setting. Sep 4, 2009 at 9:06
  • 7 zip is literally one of the first tools I put on a new box. It's just too useful.
    – corsiKa
    Dec 15, 2015 at 22:23
7

I did some testing on the aspect of "identical files", as mentioned in the question, using 7-zip (version 9.20), as no one gave an elaborate answer on that, yet. This gave some interesting results. I tested with 10 copies of the file that this sites uses for its page-not-found message. This file won't compress very well as an individual file, being a jpg-file. So, it will demonstrate the efficiency of compressing multiple identical files. Its file size is 37 KB.

  1. When I compress all ten copies, using to zip-format, the file size is 367 KB, with a compressed size of about 99% of the original total size of all 10 files.
  2. When I compress all ten copies, using to 7z-format, the file size is 37 KB, with a compressed size of about 101% of just one of the original files.
  3. If I first put 5 copies in a 7-z archive, then add 3 and finally 2 copies in separate steps, the file size becomes 111 KB, about three times the size of a single original file.

If I open the 3rd archive, one of the properties is Block. This lists 0, 1 and 2 for 3, 5 and 2 of the files, respectively.

Observations:

  1. The zip-format will compress each file individually, not benefiting from the possible to efficiently compress identical files.
  2. The 7z-format will efficiently compress multiple identical files, as long as they are added to the archive in one step.

Conclusions:

  1. For optimal compressions of files, use 7z rather than zip.
  2. Compression may improve dramatically, if you do not add files to an existing 7z-archive, but first decompress it and that compress it again, including the new files, in one step.
1

Windows Vista comes with Backup and Restore Center. It will do incremental backups of your files to avoid wasting space and having to create multiple backups. From the linked page:

Previously backed-up versions of files use only a bare minimum of disk space. If only a small part of a file changes (such as one slide in a presentation), only that portion gets tracked and saved.

1

7-zip has one of the best compression algorithm around. I don't believe there's currently anything that beats 7-zip in compression (algorithm) so far.

You must log in to answer this question.