28

(I posted this first on serverfault, but then I realized it probably belongs here.)

I'm trying compress a very large text file using 7za (p7zip) 9.20. The -mmt option doesn't seem to have any effect. I've tried both -mmt=on and -mmt=2. This is an 8-core machine. One person suggested adding -m0=lzma2 as an argument, but that just gives me E_INVALIDARG. Does anybody know how to make this work?

This has no effect:

7za a -mx=9 -mmt=2 -p myarchive.zip bigfile.txt

And this fails with an error:

7za a -m0=lzma2 -mx=9 -mmt=2 -p myarchive.zip bigfile.txt


7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)
Scanning

Creating archive release_20120601-1-http.zip



System error:
E_INVALIDARG
5
  • I believe the option is simply -mmt, not -mmt=2. Also I believe the proper syntax is -mx9, although both might work. Jun 7, 2012 at 17:20
  • Thanks, but -mmt without an option still uses only one thread. According to docs.bugaco.com/7zip/MANUAL/switches/method.htm#ZipMultiThread, you can specify the number of threads to use with, -mmt=N.
    – Brian L
    Jun 7, 2012 at 17:30
  • I would still recommend using LZMA/Deflate even though it's only single threaded. While you might get an increased compression speed with BZip2, it's less efficient when compressing plain text, and the single-threaded variants are slower than the LZMA/Deflate equivalents. Jun 7, 2012 at 18:09
  • @Breakthrough: BZip2 usually achieves better compression than DEFLATE, as shown in your link. It's also much, much faster than LZMA (when compressing).
    – Dennis
    Jun 7, 2012 at 18:17
  • On Windows 2012 R2, when I use 7zip to compress to a 7z file, it uses all processors by default. I wish I could say the same in Ubuntu, but it uses 1 core.
    – LonnieBest
    Jun 29, 2020 at 9:28

6 Answers 6

35

According to -m (Set compression Method) switch # ZipMultiThread - 7ZIP manual & documentation, mt defaults to on, so there's no need to specify it at all.

However, 7zip's implementation of the DEFLATE algorithm doesn't support multi-threading!

As you have already discovered,

7za a archive.zip bigfile

only uses one core.

But .zip files compress every file individually. When compressing several files, the multi-threading option compresses one file per core at once.

Try it and you'll see that

7za a archive.zip bigfile1 ... bigfileN

will use all available N cores.

If you want to speed up the compression of a single file, you have two choices:

  1. Split up bigfile in chunks.

  2. Use a different compression algorithm.

    For example, 7zip's implementation of the BZip2 algorithm supports multi-threading.

    The syntax is:

    7za a -mm=BZip2 archive.zip bigfile
    

Also, the syntax error is caused by your attempt to use the LZM Algorithm for a .zip container. That's not possible.

The possible algorithms for .zip conatiners are DEFLATE(64), BZip2 and no compression.

If you want to use the LZM Algorithm, use a .7z container. This container also handles the following algorithms: PPMd, BZip2, DEFLATE, BCJ, BCJ2 and no compression.

6
  • @Dennis I thought the OP was using LZMA(2), which from the documentation, "LZMA compression uses only 2 threads." Although I agree, intuitively (due to the way Lempel-Ziv encoding works), it would be very difficult to multithread LZMA or Deflate (which is just LZMA with Huffman encoding). Jun 7, 2012 at 18:03
  • 1
    @Breakthrough: At first, so did I. (Check out the revisions of my answer.) That's what the syntax error was about. You can't use LZMA compression with a .zip container.
    – Dennis
    Jun 7, 2012 at 18:05
  • @Dennis ah, thank you for clearing that up. Didn't see that the OP was using a .ZIP container. Jun 7, 2012 at 18:06
  • Wait, so I'll get a different result if I just change the file extension of the container to .7z?
    – Brian L
    Jun 7, 2012 at 18:08
  • 3
    @BrianL there's a "thanks" button built in. It looks like an arrow facing upwards ;)
    – nhinkle
    Jun 7, 2012 at 18:36
7

This is an old question, and not the answer to the specific question, but an answer to the spirit of the question (Using all cores to compress a zip format)

pigz (parallel gzip with .zip option)

pigz -K -k archive.zip bigfile txt

This will give you a zip compatible file 7x faster for same compression level.

A quick comparisons of zip compatible and non-zip compressors using single and multiple cores.

wall times on i7-2600k to compress 1.0gb txt file on fedora 20

67s (120mb) 7za (zip,1 thread)
15s (141mb) 7za -mx=4 (zip,1 thread)
17s (132mb) zip (zip,1 thread)
 5s (131mb) pigz -K -k (zip,8 threads)
 9s (106mb) bsc (libbsc.com) (not zip,8 threads)
 5s (130mb) zhuff -c2 (not zip,8 threads)
 2s (149mb) zhuff (not zip,8 threads)

wall times to decompress

4.2s unzip -t
2.0s pigz -t
5.1s bsc d
0.5s zhuff -d
2
  • why pigz when you can pbzip2 or pixz?
    – nod
    Apr 30, 2014 at 21:31
  • gzip is much, much faster than bzip2, so the extra compression isn't always worth it.
    – jesjimher
    Mar 13, 2017 at 12:27
3

Another option, to achieve multi-theaded compression on Linux is to use what Facebook uses, Zstandard. On Ubuntu, you install it like this:

sudo apt install zstd

Super fast multi-threaded compression:

tar cf - /folder/you/want/to/compress | zstdmt -o /location/to/output/fileName$(date '+%Y-%m-%d_%H:%M:%S').tar.zst

You can specify compression levels 1-19 (3 is default).

Max compression (slowest):

tar cf - /folder/you/want/to/compress | zstdmt -19 -o /location/to/output/fileName$(date '+%Y-%m-%d_%H:%M:%S').tar.zst

Medium compression (level 10):

tar cf - /folder/you/want/to/compress | zstdmt -10 -o /location/to/output/fileName$(date '+%Y-%m-%d_%H:%M:%S').tar.zst

My overall experience is that Zstandard compression isn't as strong as 7zip, but it is way faster and the zstdmt command tries to use all cores.

BTW, on Window, 7zip uses all processors by default and I'm very disappointed that this is not the case in Linux. Its been this way for several years, at this point, and I wish 7zip was multi-threaded by default in Linux too.

0

Verified and tested: To use multithreading on 7za the parameter must be "-mmt#" not "-mmt=#", putting the equal sign makes it to ignore.

How i had discovered? After i run 7z without any parameter it shows the info about parameters, on switches it say "-mmt[N]", not "-mmt=[N]"

So if i understand well, the parameter you are typing "-mmt=2" may be misswritten and may be "-mmt2", without the equal sign.

Not sure if i understand well, my english is really poor.

By the way, why you use "7za" instead of just "7z"?

So to test the parameter i run a set of commands to do benchkmarks and that confirmed the typo error on some documentation. The correct parameter must be typed without the equal sign.

Command to do a benchmark with 7z with only one thread: 7z b -mmt1

Command to do a benchmark with 7z with only two threads: 7z b -mmt2

Command to do a benchmark with 7za with only two threads: 7za b -mmt2

Command to do a benchmark with 7za with only one thread: 7za b -mmt1

There is no equal sign on the parameter ˋ-mmt#ˋ, neither for 7z, nor 7za.

1
  • 2
    I tried -mmt8 on a 12 core processor, but it still only used one thread.
    – LonnieBest
    Jun 29, 2020 at 9:35
0

Just use -mmt[N+1]

For example: -mmt2 is for one thread, -mmt9 is for eight threads

0

For me the answer using LZMA2 in 7z would be :

7za a -m0=LZMA2:d1024m -mx=9 -mmt8 -mfb=64 -md=32m -ms=on myarchive.7z bigfile.txt

I'm just building an archive of an OVF folder containing 7 files (3* 10gb files, 1* 512g and 3 less than 3m files) with option -mmt=32 for using 32 CPU.

Here using LZMA2 (native 7z algo when using 7z format of 7z command), 7za command is using 3200% of CPU with 32 threads in compressing a single file, so parallelisme is perfectly working when not using ZIP/DEFLATE algo (and yes, according to 7za documentation, for a better compression ratio, it's better to use less threads)

And contrary to what it was said here, LZ based algo are usable with parallelism (you can find cuda / openCL implementation of LZSS which are designed to run on FPGA cards with 1000 parallel workers or more)

Searching google, I found 7z-command-line-writer where you can graphically select options and it write the commnd line and options for you. I did not test, but it could help those who don't want to read 7z documentation

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .