29

Between xz, gzip, and bzip2, which compression algorithm gives the smallest file size and fastest speed when compressing fairly large tarballs?

5
  • 'the best' as in 'resulting in the smallest filesize' ?
    – Hennes
    Apr 10, 2013 at 18:53
  • I don't know, I was trying to find someway to word the question so I could add my test as an answer. I also have no idea why this thing was closed. @Karan
    – Nathan2055
    Apr 10, 2013 at 22:45
  • 2
    Oh, why it was closed it easy. "Best" is highly subjective and usually leads to discussions or non-constructive answers. Best compression can be smallest file size, fastest compression, least power used to compress (e.g. on a laptop), least influence on the system while compressing (e.g. ancient single treaded programs using only one of the cores), ... or a combination of all of those.
    – Hennes
    Apr 11, 2013 at 5:10
  • An interesting article to read is tomshardware.com/reviews/winrar-winzip-7-zip-magicrar,3436.html (windows based, and focussing on 7zip, magicRAR, WinRAR and WinZip rather than xz, gz or bz, but still interesting and providing background information).
    – Hennes
    Apr 11, 2013 at 5:13
  • @Hennes - I cleaned up the post to replace best with exactly what I was researching. Also, thanks for the article you mentioned, I will read it later today.
    – Nathan2055
    Apr 11, 2013 at 18:45

5 Answers 5

20

In my stress test, I compressed 464 megabytes of data using the three formats listed. Gzip returned a 364 MB file. Bzip2 returned a 315 MB file. Xz returned a 254 MB file. I also did a simple speed test:

Compression:

1: Gzip

2: Xz

3: Bzip2 (my fan was blowing quite a bit while this was going, indicating that my Athlon II was fairly strained)

Decompression:

1: Xz

2: Gzip

3: Bzip2

Please note that all of these tests were done with the latest version of 7-Zip.

Xz is the best format for well-rounded compression, while Gzip is very good for speed. Bzip2 is decent for its compression ratio, although xz should probably be used in its place.

5
  • 2
    Good research. Have you tried the various compression level options offered by (at least) bzip2, e.g. bzip2 -9 <file>? Apr 10, 2013 at 20:54
  • @AaronMiller - No, is it possible to use those via 7-Zip?
    – Nathan2055
    Apr 15, 2013 at 21:35
  • It appears so, though I'm not sure to what extent: see dotnetperls.com/7-zip-examples , section "Switch m". Apr 16, 2013 at 13:35
  • 8
    Out of curiosity, what sort of data was the test file? Dec 31, 2013 at 23:35
  • Note that different data types will result in different compressed sizes. See here for examples.
    – user494585
    Jun 27, 2018 at 17:13
9

I think that this article provides very interesting results.

http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

The most size efficient formats are xz and lzma, both with the -e parameter passed.

The fastest algorithm are by far lzop and lz4 which can produce a compression level not very far from gzip in 1.3 seconds while gzip took 8.1 second. The compression ratio is 2.8 for lz4 and 3.7 for gzip.

Here are a few results I extracted from this article :

  • Gzip : 8.1s @ 3.7

  • lz4 : 1.3s @ 2.8

  • xz : 32.2s @ 5.43

  • xz -e : 6m40 @ 7.063

  • xz : 4m51s @ 7.063

So if you really desperatly need speed, lz4 is awesome and still provides a 2.8 compression ratio.

If you desperatly need to spare the byte, xz at the maximum compression level (9) does the best job for text files like the kernel source. However, it is very long and takes a lot of memory.

An good one where needed to minimize the impact on time AND space is gzip. This is the one i would use to make manual daily backups of a production environment.

7

I did my own benchmark on 1.1GB Linux installation vmdk image:

rar    =260MB   comp= 85s   decomp= 5s
7z(p7z)=269MB   comp= 98s   decomp=15s
tar.xz =288MB   comp=400s   decomp=30s
tar.bz2=382MB   comp= 91s   decomp=70s
tar.gz =421MB   comp=181s   decomp= 5s

all compression levels on max, CPU Intel I7 3740QM, Memory 32GB 1600, source and destination on RAM disk

I Generally use rar or 7z for archiving normal files like documents.
and for archiving system files I use .tar.gz or .tar.xz by file-roller or tar with -z or -J options along with --preserve to compress natively with tar and preserve permissions (also alternatively .tar.7z or .tar.rar can be used)

update: as tar only preserve normal permissions and not ACLs anyway, also plain .7z plus backup and restoring permissions and ACLs manually via getfacl and sefacl can be used which seems to be best option for both file archiving or system files backup because it will full preserve permissions and ACLs, has checksum, integrity test and encryption capability, only downside is that p7zip is not available everywhere

2
  • 1
    Student, what was options of rar? Why not try lrzip by kolivas, it should work good for virtual disk images.
    – osgx
    Mar 14, 2015 at 6:39
  • I'm migrating from RAR to Git and tarballs for my text files and btrfs for everything else; my reason for using RAR is not performance, i'm using it because of features such as recovery record, separate file-level 256bit checksum for every file and ... . Mar 14, 2015 at 13:21
6

The question is from 2014, but in the meantime there have been some trends. bzip2 has been made mostly obsolete by xz, and zstd is likely the best for most workflows.

  • Minimum file size: xz is still the best when it comes to minimal file sizes. Compression is fairly expensive though, so faster compression algorithms are better suited if that is a concern. The pxz implementation allows to use multi-core, which can speed up xz compression a bit.

  • Optimizing for fast compression: When it comes to the best algorithm when optimizing primarily for compression speed, there is no clear winner in my opinion but lz4 is a good candidate.

  • Best trade-off: If you need to pick a good overall algorithm without knowing too much about the scenario, then zstd shines. When configured to run at the same speed as gzip, it will easily beat it for size. With better compression rates, it gets closer to xz but at faster speeds. So, if you need a dependable algorithm for a broad set of use cases, zstd will most likely outperform the others. It also has some advanced features, like being able to build an external dictionary, so it can also be further optimized for specific domains.

  • Maximum compatibility: If you need an algorithm that any application will be able to understand, then gzip is still the best default. Compared to zstd, it is mostly obsolete now, but almost any environment will be able to work with gzip, while support for zstd is still not there (in 2021). It has been released in 2016 while gzip is from 1992.

1

Additionally: there are dramatic! difference in output size, when you want to compress a tarball from a folder structures with a lot duplicate files.

You can tune the compression for YOUR large tarball (in my case with many duplicate files) scenario. Which means either:

  • use a tool which does deduplication upfront the actual compression like squashfs or libwim
  • use a tool which can create archives with solid mechanism like 7z https://en.wikipedia.org/wiki/Solid_compression
  • and play with dictionary-size and solid-chunk-size (if available) of xz/gzip/bzip2/etc. That has a significant influence for large files with repetitive parts.

My sample dataset is 1GB of raw data. Several dll files, where a lot are duplicates. With naive compression i got about 150MB archives with xz/gz/etc.

My best result was 40MB in 9s with wimcapture data out.wim --compress=none --solid --solid-chunk-size=1M.

While researching, i also stumbled upon https://github.com/mhx/dwarfs which might be worth taking a look into. It combines modern compressions with deduplication and speed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .