1

I'm archiving some projects, let's say each of them has own directory:

projects
 |- project-1
 |- project-2
 |- project-3

I started compressing them as following:

==== SITUATION 1 ====

projects
 |- project-1.zip
 |- project-2.zip
 |- project-3.zip

and then I started wondering if wouldn't it be better to compress all data in one zip file:

==== SITUATION 2 ====

projects.zip
 |- project-1
 |- project-2
 |- project-3

or maybe compress already compressed files?:

==== SITUATION 3 ====

projects.zip
 |- project-1.zip
 |- project-2.zip
 |- project-3.zip

Which situation is the best (occupies the least space)? Why? Does it depend on compression algorithm? I know that compressing one compressed file cannot help much, but let's say 20 of them? For me situation 1 doesn't look like a good idea.

4

I doubt that the different schemes would make a lot of difference to be honest since the compression algorithms typically only look forward a limited amount in order to control memory use.

The exception is S3 which would end up larger most likely since compressing a compressed file adds overheads but cannot compress.

If you want better compression, look for newer archiving tools that have better algorithms. 7-zip for example is generally better than zip.

In terms of the difference between s1 and s2, I would say that it depends on how you are most likely to use the archive in the future and how big they end up.

Really big archives are a pain to handle (moving, opening, etc) and this is likely to be more important than saving a few kB.

Additionally, when thinking of long-term storage, don't ignore "bit-rot". A small error in a large archive can be devastating. Loosing one project is probably much better than loosing them all.

You might however look at something like RAR which allows redundancy and split archives. This is a bit like RAID5. You create multiple archive files each of which has built in redundancy so that you can loose a file and still recreate the original data.

1

First of all, keep the excellent arguments of @Julian Knight in mind. Even the best compression is useless if your archive is either too big to handle or gets corrupted by some flipped bits.

If space is your main concern, it might be worthwhile to do some experiments with your particular data and different compression algorithms.

Also, your 3rd approach can indeed lead to another reduction in size. I remember some discussion (see here) about compressing files multiple times using different algorithms. The author was compressing highly redundant text files and could go from 100GB to a few MB after experimenting enough. Note that his case was a bit special, but the general idea is that iterated compression can actually be worthwhile in some cases.

If you are willing to try different compression algorithms, here are some benchmarks that compare speed and compression ratio:

0

Situation 3 is out because it's pointless re-compressing archives with the same algorithm.

Between Situations 1 and 2 the latter definitely has more chance of resulting in a smaller archive, especially when you use larger dictionary sizes (the dictionary in simple words is the memory area used to find and compress repeated patterns in data). Plain old ZIP can only use a tiny 32KB dictionary, which given the hardware these days is way too small.

The RAR 5.0 format by comparison can use a 1GB dictionary on 64-bit systems. It also supports saving identical files as references:

If this option is enabled, WinRAR analyzes the file contents before starting archiving. If several identical files larger than 64 KB are found, the first file in the set is saved as usual file and all following files are saved as references to this first file. It allows to reduce the archive size, but applies some restrictions to resulting archive. You must not delete or rename the first identical file in archive after the archive was created, because it will make extraction of following files using it as a reference impossible. If you modify the first file, following files will also have the modified contents after extracting. Extraction command must involve the first file to create following files successfully.

Thus if you have many duplicate files among your projects, a large dictionary size combined with solid archiving and the feature above is very likely to lead to significant size reduction with Situation 2. Of course all the general caveats about large archives apply, so including a recovery record would also be recommended.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.