Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to perform a series of recompressions of compressed files to study their properties.

At first, I tried simply recompressing .zip files in a loop:

for f in $(seq 1 100)
  zip $ -9 -v $(($f-1)).zip

Starting from a fixed file, I noticed that, if I re-ran these lines, all but the first few files had different sizes. After some investigation, I realized this was caused by the timestamps of the compressed files. I then reset the timestamps before recompressing:

for f in $(seq 1 100)
  touch $(($f-1)).zip -t 200101010101
  zip $ -9 -v $(($f-1)).zip

And this time I got deterministic results for the compression using zip. However, using tar with the gzip algorithm still does not work. Each time I run:

for f in $(seq 1 100)
  touch $(($f-1)).tar.gz -t 200101010101
  tar cvfz $f.tar.gz $(($f-1)).tar.gz

I get different file sizes, from 4.tar.gz until 100.tar.gz. Why does that happen, and what can I do to prevent it from happening?

share|improve this question

To enable using tar for incremental backups, some versions of tar store the time at which a file is added to an archive. This may result in different output every time you run your test and hence, in results that seem non-deterministic.

share|improve this answer
I'm using GNU tar. I tried several options related to incremental backups, but couldn't make it work deterministically. I also tried setting mtime but it didn't help. Are you implying that it is maybe impossible to obtain deterministic archives with my version of tar? Or maybe there are options to override this behavior? – anol Apr 29 '14 at 12:08
Well, I found a way below. Unfortunately the incremental backup settings did not help me, but they motivated me to try to understand the real reason. I was focusing on tar itself, while the issue was the gzip part... – anol Apr 29 '14 at 12:48
up vote 3 down vote accepted

Ok, so I think I finally found a way to get things the way I wanted, and it involves using tar and gzip separately (I'm using both GNU tools, tar 1.26 and gzip 1.6).

When using the z option in tar, it first creates the tar file and then immediately afterwards it uses that timestamp to create the gzip file. In other words, I cannot control the timestamp of the intermediate file (I didn't find any options to do so, other than changing the mtime of the original files being archived), so the .tar.gz file ends up being non-deterministic.

However, if I first use tar without gzip, then modify the timestamp of the tar file (using touch for instance), and then I compile it using gzip, I get deterministic results.

Incidentally, the resulting files using the two-step process are much smaller (~400KB in my example, instead of the ~1.2MB I get when using tar with the z option). This must be related to the default settings when using tar+gzip and gzip alone. Since my original file was empty anyway, I'm not sure the size reduction is meaningful.

Anyway, the short answer to my question would then be:

  • It is possible to obtain deterministic results, but using tar and gzip separately;

  • It does not seem possible to obtain deterministic results when using tar -z, due to the timestamp of the intermediate tar file before gzip compression. This timestamp is not under control of the user.

share|improve this answer

As you mentioned, you will need to do the tar and gzip steps separately, however you do not need an intermediate file, you can simply pipe the input from tar to gzip and specify the -n option in gzip. From the manpage: This option stops the filename and timestamp from being stored in the output file.

tar cvf - /path/to/files | gzip -n > archive.tar.gz

Run the above command twice at different times on the same set of files and check the md5 sum. You should see the two archives are identical.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.