Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

This question already has an answer here:

The way i understand the use of tar + gzip is that tar is normally used to consolidate a grouping of files into a single file, then gzip is used to compress that file.

I recently learned that tar can ALSO compress.

Because I do not fully understand how compression works @ it's core, I have (possibly ridiculous) concerns that sending a pre-compressed .tar to gzip might prevent gzip from compressing as well as it's potential would allow and things of that nature.

My question is essentially: What combination of args/compression methods should i use to create the absolute smallest tar.gz, and what does the command line statement look like for that?

share|improve this question

migrated from stackoverflow.com Dec 3 '12 at 19:59

This question came from our site for professional and enthusiast programmers.

marked as duplicate by Kevin Panko, Matthew Williams, music2myear, Scott, Excellll Oct 22 at 14:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1  
Compressing already compressed files may reduce their size, or it may make the archive bigger. It all depends on the type of data and any compression being used. –  Keltari Jan 31 '13 at 19:02
    
What @Keltari said. Compression rates and ratios are highly dependent on what it is you are compressing, which is also why there are different compression algorithms and methods. –  music2myear Oct 21 at 17:13

4 Answers 4

As you stated- "tar can also compress", implies that - tar does not always compress data by itself.

It does so only when used with the z option. That too not by itself, but - by passing the tarred data through gzip.

However instead, as noted in this answer, you can pipe the two commands: tar & gzip such that you can explicitly specify compression level for the gzip command to achieve smallest output size.

tar cvf - /path/to/directory | gzip -9 - > file.tar.gz

Here 9 specifies maximum possible compression level.

share|improve this answer

Usually neither gzip nor tar can create "the absolute smallest tar.gz". There are many compression utilities that can compress to the gz format. I have written a bash script "gz99" to try gzip, 7z and advdef to get the smallest file. To use this to create the smallest possible file run:

tar c path/to/data | gz99 file.gz

The advdef utility from AdvanceCOMP usually gives the smallest file, but is also buggy (the gz99 utility checks that it hasn't corrupted the file before accepting the output of advdef). To use advdef directly, create file.tar.gz however you feel like. Then run:

advdef -z -4 file.tar.gz

This will create a standard gz file that can be read by gzip and tar as normal, just a tiny bit smaller. This is about the best you can do with the gz format.

Since you only recently learnt that tar can compress, and didn't say why you wanted the the smallest ".tar.gz" file, you may be unaware that there are more efficient formats can be used with tar files, such as xz. Generally, switching to a different format can give a vastly better improvement in compression than fiddling round with gzip options. The main disadvantage of xz is that it isn't as common as gzip so the people you send the file to might have to install a new package. It also tends to be a bit slower, particularly when compressing. If this doesn't matter to you, and you really want the smallest tar file, try:

 tar cv path/to/data | xz -9 > file.tar.xz

Modern versions of tar, for example on Ubuntu 13.10, automatically detect compressed files. So even if you use xz compression you can still decompress as usual:

 tar xvf file.tar.xz

To give a quick idea how these compression utilities compare, consider the effect of compressing patch-3.1.1 from the linux kernel:

utility         cpu    format  size(bytes)
gzip -9         0.02s  gz      105,628
advdef -2       0.07s  gz      102,619
7z -mx=9 -tgzip 0.42s  gz      102,297
advdef -3       0.55s  gz      102,290
advdef -4       0.75s  gz      101,956
xz -9           0.03s  xz       91,064
xz -3e          0.15s  xz       90,996

In this trivial example, we see that to get the smallest gz we need advdef (though 7z -tgzip is almost as good and a lot less buggy). We also see that switching to xz gains us much more space than trying to squeeze the most out of the old gz format, without compression taking too long.

share|improve this answer
    
The OP asked for how to get the most compression for a .tar.gz file, but you suggested creating a .tar.xz file. You are answering a different question than asked. –  ChrisInEdmonton Mar 10 at 16:08
1  
Actually, advdef does create a (.tar.)gz file. Since the OP asked for the smallest possible .tar.gz file, not smallest possible gzip file, technically this is the one answer that didn't answer a different question. After giving the (only) technically correct answer to their question, I then explained why they may actually want an xz file instead. I have edited my answer to make this more clear. –  gmatht Mar 10 at 17:13
    
Ah, I see what you are going for. advdef just crashes on my system (v1.15), so 'advdef -z -4 file.tar.gz' doesn't work, but it at least theoretically could. I can't find evidence that it would shrink the file further than 'gzip -9', but it might, and in any case is enough for me to remove my -1 vote. Thanks for clarifying! –  ChrisInEdmonton Mar 10 at 17:29
    
Hmm, I'm using v1.17. Anyway the pedantic mathematician in me wants to point out that my answer arguably isn't technically correct. After all, if you enumerate all possible gz files from shortest to longest and pick the first one that decompresses to the right file, you could shave yet a few more bytes off. But that'd be way too slow in practice. –  gmatht Mar 10 at 17:46
tar c /path/to/data | gzip --best > file.tar.gz

gzip option --best (equivalent to -9) asks for the highest compression level.

share|improve this answer
2  
Alternatively, use --best flag: -9 is confusing to reader. –  om-nom-nom Feb 20 at 12:04

Or, you can tell tar to user maximum compression this way:

export GZIP=-9
tar cvzf file.tar.gz /path/to/directory

Additionally, to keep your envvars clutter-free, you can do this:

env GZIP=-9 tar cvzf file.tar.gz /path/to/directory
share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.