Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

(I posted this first on serverfault, but then I realized it probably belongs here.)

I'm trying compress a very large text file using 7za (p7zip) 9.20. The -mmt option doesn't seem to have any effect. I've tried both -mmt=on and -mmt=2. This is an 8-core machine. One person suggested adding -m0=lzma2 as an argument, but that just gives me E_INVALIDARG. Does anybody know how to make this work?

This has no effect:

7za a -mx=9 -mmt=2 -p myarchive.zip bigfile.txt

And this fails with an error:

7za a -m0=lzma2 -mx=9 -mmt=2 -p myarchive.zip bigfile.txt


7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)
Scanning

Creating archive release_20120601-1-http.zip



System error:
E_INVALIDARG
share|improve this question
    
I believe the option is simply -mmt, not -mmt=2. Also I believe the proper syntax is -mx9, although both might work. –  Breakthrough Jun 7 '12 at 17:20
    
Thanks, but -mmt without an option still uses only one thread. According to docs.bugaco.com/7zip/MANUAL/switches/method.htm#ZipMultiThread, you can specify the number of threads to use with, -mmt=N. –  Brian L Jun 7 '12 at 17:30
    
I would still recommend using LZMA/Deflate even though it's only single threaded. While you might get an increased compression speed with BZip2, it's less efficient when compressing plain text, and the single-threaded variants are slower than the LZMA/Deflate equivalents. –  Breakthrough Jun 7 '12 at 18:09
    
@Breakthrough: BZip2 usually achieves better compression than DEFLATE, as shown in your link. It's also much, much faster than LZMA (when compressing). –  Dennis Jun 7 '12 at 18:17
add comment

2 Answers

up vote 8 down vote accepted

According to -m (Set compression Method) switch # ZipMultiThread - 7ZIP manual & documentation, mt defaults to on, so there's no need to specify it at all.

However, 7zip's implementation of the DEFLATE algorithm doesn't support multi-threading!

As you have already discovered,

7za a archive.zip bigfile

only uses one core.

But .zip files compress every file individually. When compressing several files, the multi-threading option compresses one file per core at once.

Try it and you'll see that

7za a archive.zip bigfile1 ... bigfileN

will use all available N cores.

If you want to speed up the compression of a single file, you have two choices:

  1. Split up bigfile in chunks.

  2. Use a different compression algorithm.

    For example, 7zip's implementation of the BZip2 algorithm supports multi-threading.

    The syntax is:

    7za a -mm=BZip2 archive.zip bigfile
    

Also, the syntax error is caused by your attempt to use the LZM Algorithm for a .zip container. That's not possible.

The possible algorithms for .zip conatiners are DEFLATE(64), BZip2 and no compression.

If you want to use the LZM Algorithm, use a .7z container. This container also handles the following algorithms: PPMd, BZip2, DEFLATE, BCJ, BCJ2 and no compression.

share|improve this answer
    
Thanks, Dennis! –  Brian L Jun 7 '12 at 18:01
    
@Dennis I thought the OP was using LZMA(2), which from the documentation, "LZMA compression uses only 2 threads." Although I agree, intuitively (due to the way Lempel-Ziv encoding works), it would be very difficult to multithread LZMA or Deflate (which is just LZMA with Huffman encoding). –  Breakthrough Jun 7 '12 at 18:03
1  
@Breakthrough: At first, so did I. (Check out the revisions of my answer.) That's what the syntax error was about. You can't use LZMA compression with a .zip container. –  Dennis Jun 7 '12 at 18:05
    
@Dennis ah, thank you for clearing that up. Didn't see that the OP was using a .ZIP container. –  Breakthrough Jun 7 '12 at 18:06
3  
@BrianL there's a "thanks" button built in. It looks like an arrow facing upwards ;) –  nhinkle Jun 7 '12 at 18:36
show 2 more comments

This is an old question, and not the answer to the specific question, but an answer to the spirit of the question (Using all cores to compress a zip format)

pigz (parallel gzip with .zip option)

pigz -K -k archive.zip bigfile txt

This will give you a zip compatible file 7x faster for same compression level.

A quick comparisons of zip compatible and non-zip compressors using single and multiple cores.

wall times on i7-2600k to compress 1.0gb txt file on fedora 20

67s (120mb) 7za (zip,1 thread)
15s (141mb) 7za -mx=4 (zip,1 thread)
17s (132mb) zip (zip,1 thread)
 5s (131mb) pigz -K -k (zip,8 threads)
 9s (106mb) bsc (libbsc.com) (not zip,8 threads)
 5s (130mb) zhuff -c2 (not zip,8 threads)
 2s (149mb) zhuff (not zip,8 threads)

wall times to decompress

4.2s unzip -t
2.0s pigz -t
5.1s bsc d
0.5s zhuff -d
share|improve this answer
    
why pigz when you can pbzip2 or pixz? –  nod Apr 30 at 21:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.