Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm looking for the fastest gzip (or zip) utility. I have a LVM volume which 95% exists out of blank 0's, so compressing that is very easy. I'm looking for the most fastest solution, and don't really care of the compression except the 0's.

I'm aware of gzip -1 (same as gzip --fast) but was wondering if there's any faster method.

Thanks.

Edit: after some tests, I compared gzip -1, lzop -1 and pigz -1 with eachother and came to the following results:

PIGZ:

time dd if=/dev/VPS/snap | pigz -1 | ssh backup-server "dd of=/home/backupvps/snap.pigz"

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 2086.87 seconds, 25.7 MB/s
7093985+266013 records in
7163950+1 records out
3667942715 bytes (3.7 GB) copied, 2085.75 seconds, 1.8 MB/s

real    34m47.147s

LZOP:

time dd if=/dev/VPS/snap | lzop -1 | ssh backup-server "dd of=/home/backupvps/snap.lzop"

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 1829.31 seconds, 29.3 MB/s
7914243+311979 records in
7937728+1 records out
4064117245 bytes (4.1 GB) copied, 1828.08 seconds, 2.2 MB/s

real    30m29.430s

GZIP:

time dd if=/dev/VPS/snap | gzip -1 | ssh backup-server "dd of=/home/backupvps/snap_gzip.img.gz

104857600+0 records in
104857600+0 records out
53687091200 bytes (54 GB) copied, 1843.61 seconds, 29.1 MB/s
7176193+42 records in
7176214+1 records out
3674221747 bytes (3.7 GB) copied, 1842.09 seconds, 2.0 MB/s

real    30m43.846s

Edit 2:

This is somewhat unrelated to my initial question, however using time dd if=/dev/VPS/snap | lzop -1 | ssh backup-server "dd of=/home/backupvps/snap.lzop" (block size changed to 16M) the time is reduced to real 18m22.442s!

share|improve this question
1  
Be careful: it's somewhat unfair to use time in such a manner. The throughput of the dd used for pigz is lower than the other two. –  Henk Mar 14 '12 at 16:15
    
@Devator: by looking at the timings one might conclude that right now pushing bytes through the encrypted ssh tunnel is the bottleneck. did you try to use ssh with the "-c" (compression) flag and let the pre-compressor out of the equation? you could also switch to a faster encryption algorithm. aside from that: re-benchmark without the ssh-tunnel (eg, using /dev/null as the output sink) –  akira Mar 14 '12 at 17:40
    
As a sidenote, could you use a sparse file? Then the zeroes would take up no space on disk. Your compression would also be faster because the zeroes would be interpolated by the filesystem driver (and wouldn't have to be read from disk.) –  Li-aung Yip Mar 15 '12 at 10:57
    
@Li-aungYip I don't think so, as the "files" are LVM volumes. –  Devator Mar 15 '12 at 11:03
    
Ah, I see. Carry on! –  Li-aung Yip Mar 15 '12 at 11:07

5 Answers 5

up vote 8 down vote accepted

If you don't mind stepping away from DEFLATE, lzop is an implementation of LZO that favors speed over compression ratio.

share|improve this answer
1  
or .. snappy: code.google.com/p/snappy –  akira Mar 14 '12 at 17:35
    
Thanks, I've found lzop to be the fastest in my scenario. It's faster than pigz somehow (probably due to the lot's of 0's). –  Devator Mar 15 '12 at 10:27

Although I personally have not yet used it, I think using parallel gzip could speed up things a bit:

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

share|improve this answer
    
I use it routinely, and absolutely recommend pigz if multiple cores are available. Other than changing the compression level, this is by far the most accessible and straightforward means of speeding up compression. –  jgrundstad Mar 14 '12 at 14:55
2  
The site looks a bit odd. But don't be fooled by that, pigz is written by the one of the developers of gzip and zlib, Mark Adler. –  so_mv Aug 12 '13 at 5:58

You can try Parallel Gzip (Pascal linked it in), or Parallel BZIP.
In theory, BZIP is much better for text, so you may want to try pbzip.

share|improve this answer

Re: lzop it is slower at its std config... Tweaking can half the time. But there is an even faster replacement called blosc:

https://github.com/FrancescAlted/blosc

Hmm... The time it took to post this and get replies is probably at least double any time savings you'll get though... Now excuse me while I recompile my kernel to shave off another .1s from my 2s boot time.

share|improve this answer

Your disk is limited at 30MB/s

All compressors do well enough. You can even reduce network transfer using slighty slower but omnipresent bzip2.

$dd if=/dev/zero bs=2M count=512 | pigz -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 9.12679 s, 118 MB/s
8192+7909 records in
9488+1 records out
4857870 bytes (4.9 MB) copied, 9.13024 s, 532 kB/s
$dd if=/dev/zero bs=2M count=512 | bzip2 -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 37.4471 s, 28.7 MB/s
12+1 records in
12+1 records out
6533 bytes (6.5 kB) copied, 37.4981 s, 0.2 kB/s
$dd if=/dev/zero bs=2M count=512 | gzip -1 | dd > /dev/null
512+0 records in
512+0 records out
1073741824 bytes (1.1 GB) copied, 14.305 s, 75.1 MB/s
9147+1 records in
9147+1 records out
4683762 bytes (4.7 MB) copied, 14.3048 s, 327 kB/s

Have you considered rsync? It does checksumming and then gzipping the difference only.

share|improve this answer
1  
My disk is not limited at 30 MB/s. I've just ran your test: pigz -1: 1073741824 bytes (1.1 GB) copied, 8.6779 seconds, 124 MB/s and gzip -1: 1073741824 bytes (1.1 GB) copied, 11.6724 seconds, 92.0 MB/s . I've thought about rsync but that would check the file differenced and it would probably not help, as most of the time a lot has changed. –  Devator Mar 14 '12 at 22:03
    
If you are at transferring zeroes look how impressive does bzip2 encoding look in comparison. Just at which side you measure speed.... 4Mbit/s of pigz might be too much for a common DSL line... It grows even worse if your disk is that fast. –  ZaB Mar 18 '12 at 18:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.