19

I'm looking for the fastest gzip (or zip) utility. I have a LVM volume which 95% exists out of blank 0's, so compressing that is very easy. I'm looking for the most fastest solution, and don't really care of the compression except the 0's.

I'm aware of gzip -1 (same as gzip --fast) but was wondering if there's any faster method.

Thanks.

Edit: after some tests, I compared gzip -1, lzop -1 and pigz -1 with eachother and came to the following results:

PIGZ:

time dd if=/dev/VPS/snap | pigz -1 | ssh backup-server "dd of=/home/backupvps/snap.pigz"

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 2086.87 seconds, 25.7 MB/s
7093985+266013 records in
7163950+1 records out
3667942715 bytes (3.7 GB) copied, 2085.75 seconds, 1.8 MB/s

real    34m47.147s

LZOP:

time dd if=/dev/VPS/snap | lzop -1 | ssh backup-server "dd of=/home/backupvps/snap.lzop"

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 1829.31 seconds, 29.3 MB/s
7914243+311979 records in
7937728+1 records out
4064117245 bytes (4.1 GB) copied, 1828.08 seconds, 2.2 MB/s

real    30m29.430s

GZIP:

time dd if=/dev/VPS/snap | gzip -1 | ssh backup-server "dd of=/home/backupvps/snap_gzip.img.gz

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 1843.61 seconds, 29.1 MB/s
7176193+42 records in
7176214+1 records out
3674221747 bytes (3.7 GB) copied, 1842.09 seconds, 2.0 MB/s

real    30m43.846s

Edit 2:

This is somewhat unrelated to my initial question, however using time dd if=/dev/VPS/snap | lzop -1 | ssh backup-server "dd of=/home/backupvps/snap.lzop" (block size changed to 16M) the time is reduced to real 18m22.442s!

6
  • 1
    Be careful: it's somewhat unfair to use time in such a manner. The throughput of the dd used for pigz is lower than the other two.
    – Henk
    Mar 14, 2012 at 16:15
  • @Devator: by looking at the timings one might conclude that right now pushing bytes through the encrypted ssh tunnel is the bottleneck. did you try to use ssh with the "-c" (compression) flag and let the pre-compressor out of the equation? you could also switch to a faster encryption algorithm. aside from that: re-benchmark without the ssh-tunnel (eg, using /dev/null as the output sink)
    – akira
    Mar 14, 2012 at 17:40
  • As a sidenote, could you use a sparse file? Then the zeroes would take up no space on disk. Your compression would also be faster because the zeroes would be interpolated by the filesystem driver (and wouldn't have to be read from disk.) Mar 15, 2012 at 10:57
  • @Li-aungYip I don't think so, as the "files" are LVM volumes.
    – Devator
    Mar 15, 2012 at 11:03
  • Ah, I see. Carry on! Mar 15, 2012 at 11:07

5 Answers 5

14

If you don't mind stepping away from DEFLATE, lzop is an implementation of LZO that favors speed over compression ratio.

2
  • 1
    or .. snappy: code.google.com/p/snappy
    – akira
    Mar 14, 2012 at 17:35
  • Thanks, I've found lzop to be the fastest in my scenario. It's faster than pigz somehow (probably due to the lot's of 0's).
    – Devator
    Mar 15, 2012 at 10:27
25

Although I personally have not yet used it, I think using parallel gzip could speed up things a bit:

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

4
  • 1
    I use it routinely, and absolutely recommend pigz if multiple cores are available. Other than changing the compression level, this is by far the most accessible and straightforward means of speeding up compression.
    – jgrundstad
    Mar 14, 2012 at 14:55
  • 3
    The site looks a bit odd. But don't be fooled by that, pigz is written by the one of the developers of gzip and zlib, Mark Adler.
    – so_mv
    Aug 12, 2013 at 5:58
  • Looks like the project is abandoned at this point. Nov 28, 2017 at 21:31
  • I prefer to think of it as "stable". It doesn't update often, but it does update. Feb 6, 2018 at 19:38
7

You can try Parallel Gzip (Pascal linked it in), or Parallel BZIP.
In theory, BZIP is much better for text, so you may want to try pbzip.

2

Your disk is limited at 30MB/s

All compressors do well enough. You can even reduce network transfer using slighty slower but omnipresent bzip2.

$dd if=/dev/zero bs=2M count=512 | pigz -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 9.12679 s, 118 MB/s
8192+7909 records in
9488+1 records out
4857870 bytes (4.9 MB) copied, 9.13024 s, 532 kB/s
$dd if=/dev/zero bs=2M count=512 | bzip2 -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 37.4471 s, 28.7 MB/s
12+1 records in
12+1 records out
6533 bytes (6.5 kB) copied, 37.4981 s, 0.2 kB/s
$dd if=/dev/zero bs=2M count=512 | gzip -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 14.305 s, 75.1 MB/s
9147+1 records in
9147+1 records out
4683762 bytes (4.7 MB) copied, 14.3048 s, 327 kB/s

Have you considered rsync? It does checksumming and then gzipping the difference only.

2
  • 1
    My disk is not limited at 30 MB/s. I've just ran your test: pigz -1: 1073741824 bytes (1.1 GB) copied, 8.6779 seconds, 124 MB/s and gzip -1: 1073741824 bytes (1.1 GB) copied, 11.6724 seconds, 92.0 MB/s . I've thought about rsync but that would check the file differenced and it would probably not help, as most of the time a lot has changed.
    – Devator
    Mar 14, 2012 at 22:03
  • If you are at transferring zeroes look how impressive does bzip2 encoding look in comparison. Just at which side you measure speed.... 4Mbit/s of pigz might be too much for a common DSL line... It grows even worse if your disk is that fast.
    – ZaB
    Mar 18, 2012 at 18:51
2

Re: lzop it is slower at its std config... Tweaking can half the time. But there is an even faster replacement called blosc:

https://github.com/FrancescAlted/blosc

Hmm... The time it took to post this and get replies is probably at least double any time savings you'll get though... Now excuse me while I recompile my kernel to shave off another .1s from my 2s boot time.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .