Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to perform a series of recompressions of compressed files to study their properties.

At first, I tried simply recompressing .zip files in a loop:

for f in $(seq 1 100)
do 
  zip $f.zip -9 -v $(($f-1)).zip
done

Starting from a fixed 0.zip file, I noticed that, if I re-ran these lines, all but the first few files had different sizes. After some investigation, I realized this was caused by the timestamps of the compressed files. I then reset the timestamps before recompressing:

for f in $(seq 1 100)
do
  touch $(($f-1)).zip -t 200101010101
  zip $f.zip -9 -v $(($f-1)).zip
done

And this time I got deterministic results for the compression using zip. However, using tar with the gzip algorithm still does not work. Each time I run:

for f in $(seq 1 100)
do
  touch $(($f-1)).tar.gz -t 200101010101
  tar cvfz $f.tar.gz $(($f-1)).tar.gz
done

I get different file sizes, from 4.tar.gz until 100.tar.gz. Why does that happen, and what can I do to prevent it from happening?

share|improve this question

2 Answers 2

To enable using tar for incremental backups, some versions of tar store the time at which a file is added to an archive. This may result in different output every time you run your test and hence, in results that seem non-deterministic.

share|improve this answer
    
I'm using GNU tar. I tried several options related to incremental backups, but couldn't make it work deterministically. I also tried setting mtime but it didn't help. Are you implying that it is maybe impossible to obtain deterministic archives with my version of tar? Or maybe there are options to override this behavior? –  dhekir Apr 29 at 12:08
    
Well, I found a way below. Unfortunately the incremental backup settings did not help me, but they motivated me to try to understand the real reason. I was focusing on tar itself, while the issue was the gzip part... –  dhekir Apr 29 at 12:48
up vote 1 down vote accepted

Ok, so I think I finally found a way to get things the way I wanted, and it involves using tar and gzip separately (I'm using both GNU tools, tar 1.26 and gzip 1.6).

When using the z option in tar, it firsts create the tar file and then immediately afterwards it uses that timestamp to create the gzip file. In other words, I cannot control the timestamp of the intermediate file (I didn't find any options to do so, other than changing the mtime of the original files being archived), so the .tar.gz file ends up being non-deterministic.

However, if I first use tar without gzip, then modify the timestamp of the tar file (using touch for instance), and then I compile it using gzip, I get deterministic results.

Incidentally, the resulting files using the two-step process are much smaller (~400KB in my example, instead of the ~1.2MB I get when using tar with the z option). This must be related to the default settings when using tar+gzip and gzip alone. Since my original file was empty anyway, I'm not sure the size reduction is meaningful.

Anyway, the short answer to my question would then be:

  • It is possible to obtain deterministic results, but using tar and gzip separately;

  • It does not seem possible to obtain deterministic results when using tar -z, due to the timestamp of the intermediate tar file before gzip compression. This timestamp is not under control of the user.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.