90

I often have to gather log files and upload them to a central server (Owned by another company). The central server has a size limit of the file, so I am trying to create the smallest file possible that is still in the zip format.

What are the best setting to use when compressing a text file to a zip format when my only need is a small file size?

7zip Options

I've done the obvious and chosen ultra compression, and I have noticed that LZMA does a better job than deflate, but there are far too many other permutations of options for me to test them all.

  • 1
    Is splitting the the zip in to multiple files an option? – JaredMcAteer May 10 '11 at 14:21
  • 6
    As soon as you pick anything but the Deflate format, it's not a "normal" .zip file anymore, but an "extended" zip file, pioneered by WinZip. They originally kept the extension as .zip, to much consternation (since most normal zip-handling tools can't deal with them), but most archivers use .zipx now to distinguish them from traditional .zip files. If you can use LZMA, switch to .7z and pick PPMd -- it should compress better (and faster!) for text files. – afrazier May 20 '11 at 16:04
  • 1
    @afrazier: "The .ZIP File Format Specification documents the following compression methods: stored (no compression), Shrunk, Reduced (methods 1-4), Imploded, Tokenizing, Deflated, Deflate64, bzip2, LZMA (EFS), WavPack, PPMd." en.wikipedia.org/wiki/Zip_%28file_format%29#Compression_methods – endolith Dec 13 '13 at 22:26
  • 2
    @endolith: bzip2, lzma, wv, and ppmd are all very recent additions to the file format. It's not even safe to assume that your recipient can handle deflate64, much less anything newer. – afrazier Dec 13 '13 at 22:33
  • 1
    define "normal zip tools". Most "normal zip tools" nowadays like 7z and winrar can extract 7z files. – phuclv May 29 '16 at 11:04
90

To create the smallest standard ZIP file that 7-Zip can create, try:

7z a -mm=Deflate -mfb=258 -mpass=15 -r foo.zip C:\Path\To\Files\*

Source: How can I achieve the best, standard ZIP compression?

Otherwise if you don't care about the ZIP standard, use the following ultra settings:

7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on archive.7z dir1

Which are:

-t7z   7z archive

-m0=lzma
       lzma method

-mx=9  level of compression = 9 (Ultra)

-mfb=64
       number of fast bytes for LZMA = 64
-md=32m
       dictionary size = 32 megabytes

-ms=on solid archive = on
  • 4
    @Tek: Why? It's not a good one. The question was about using the "standard ZIP format", so the answer shouldn't be specifying LZMA. -ms=on is for .7z, not standard zip files. -md is related to BZip2, so I don't expect it to affect ZIP (or even LZMA). -mfb=64 is an unoptimized value: -mfb=258 makes smaller zip files. And this answer doesn't even mention -mpass=15 which can affect zip files. This is a nicely formatted answer which is, unfortunately, wrong in multiple ways. – TOOGAM Nov 8 '15 at 12:32
  • 9
    I would use lzma2 – Lance Badger Jul 7 '16 at 15:22
  • If you look at the 7-zip FAQ, it states that newer versions of 7z may have worse performance than older versions in some circumstances. Read the FAQ for more detail, but in short use the 'qs' in Parameters field in the GUI or use -mqs in the command line version to use the old sort by file extension method. 7-zip.org/faq.html. – drojf May 15 '19 at 14:21
24

After much experimentation, digging into the detailed 7zip documentation, and reading some of the 7z source code regarding the advanced LZMA2 parameters, here is a better method below. It reduced some 1GB real-world test files more than 2 to 4 times better than the previously accepted solutions posted here or even in the 7z manpage.

7z a -t7z -mx=9 -mfb=273 -ms -md=31 -myx=9 -mtm=- -mmt -mmtf -md=1536m -mmf=bt3 -mmc=10000 -mpb=0 -mlc=0 archive.7z inputfileordir

The LZMA2 compression is assumed here, but you might be able to get even better performance in 7zip with passing advanced LZMA2 options like -m0=LZMA2:27, or -m0=LZMA2:d25, or an array of parameters like

-m0=BCJ2 -m1=LZMA:d25 -m2=LZMA:d19 -m3=LZMA:d19 -mb0:1

Such parameters didn't seem to be respected by the 7z versions I tested, but you may want to explore further or patch the 7z code to properly parse them. Or maybe it is supposed to work and is just broken in the builds that were tested.

  • 3
    wow, this made a really big difference. For my archive, I experimented with a lot of other suggestions, including other answers here, and the best result I got was 99MB, vs 85MB using these settings. – user9399 Aug 25 '19 at 23:42
  • How would you call this on Windows 10 in command line? I get "The parameter is incorrect" on version 19.00 2019-02-21 – user1306322 Dec 4 '19 at 9:11
  • To run it on Windows, you must add the 7-Zip installation path to your system environment variable. Then you can use 7z inside Command Prompt. – user1100970 Jun 8 '20 at 16:17
  • This link gives more info on adding system environment variables for 7zip. – Jonas Jun 11 '20 at 15:26
  • 2
    your command uses the incredible amount of 45 GB virtual memory. on my PC this caused the OOM-Killer to just kill it. So, this does not seem to be a solution for people with 16 GB of RAM or less. – JPT Jul 17 '20 at 15:46
17

If you can use .7z format rather than just .zip, I would simply use PPMD with the following options and leave everything else as set by the Compression Level:

  • Archive Format: 7z
  • Compression Method: PPMD
  • Compression Level: Ultra

I regularly compress server/text logs (60MB+) using these options and they usually come out at 1-2% of the original size.

  • For text such as log files, ppmd is definitely the way to go. However, the question mentioned that it needed to stay in the zip format, which may not work with PPMD. – Brian Minton Dec 19 '13 at 16:31
  • Just tried zip with PPMD and Windows Explorer opens the contents up without complaint here on Windows 7 – Umber Ferrule Dec 20 '13 at 16:44
  • 4
    I noticed that too. It opens the contents up just fine. However, when I actually tried to view one of the files inside the zip file, it failed. – Brian Minton Dec 23 '13 at 16:58
  • 4
    Why is PPMD superior compression method for text files? – user598527 Feb 27 '17 at 19:58
  • 3
    LZMA2 gives better results for text files than PPMD. – T3rm1 Nov 22 '18 at 10:38
7

I compare for db.fdb 1,2 GB (1236598784 B) in Ubuntu server 14.04.03 with p7zip [64] 9.20 on VM:

1. 7z a -mx=9 1.7z db.fdb
2. 7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on 2.7z db.fdb
3. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on 3.7z db.fdb
4. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on -pass=15 4.7z db.fdb
5. 7z a -mx=9 -mmt=on 5.7z db.fdb
6. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on -mmt=on 6.7z db.fdb

and have that results:

1.7z 96 MB (100108731 B) with 6' 25"
2.7z 95 MB ( 99520375 B) with 5' 18"
3.7z 93 MB ( 97512311 B) with 9' 19"
4.7z 93 MB ( 97512345 B) with 9' 40"
5.7z 96 MB (100108731 B) with 5' 26"
6.7z 93 MB ( 97512311 B) with 9' 09"

I think second method works fine = (almost) best compress with best time. But for best "view" and easy to remember is first method - with small files and no point of max compress. Between 2 and 3 method we don't get extra smaller 7z but pay almost twoo more time for compression. Anyone decide with his own.

7
+50

I have decided to do some experiments for empirically finding the optimal compression parameters.

The tool I have used was 7-ZIP finetuner. This tool hunts for the optimal parameters by simply repeating the compression with varying parameters looking for the optimal combination. A run for one file may sometimes take more than an hour even on a fast computer.

The parameters that it tries are:

LC : number of Literal Context bits
LP : number of Literal Pos bits
PB : number of Pos Bits
YX : level of file analysis
FB : number of Fast Bytes

I have left the default parameters of dictionary size as 512 MB and solid block size On. The tool uses the LZMA method.

The best combinations of parameters on several types of files were as follows:

enter image description here

I note that the best values were not constant even for files of the same type.

Conclusion: There are no best options, as each file may have its own unique best combination. One may drive all parameters up to their limits, but an improvement is not at all guaranteed.

The most common combination seems to be:

LC : 8
LP : 0
PB : 1
YX : 5
FB : 273

Some 7-Zip references:

0

Set the "split to volume, bytes" field to the server's maximum allowed file size (in bytes, I think, although it looks like it accepts common abbreviations like "KB" and "MB"). If the zip file exceeds that size, 7-zip will split it into multiple files automatically, such as integration_serviceLog.zip.001, integration_serviceLog.zip.002, etc. (Way back when, PK Zip used this to span zip files across multiple floppy disks.) You'll need all the files to be present to unzip them. Use that instead of worrying about the absolute best compression settings to use for any particular set of files, because what's best for one file may be different for another file, and you don't want to have to go through this every time you need to copy logs.

  • 1
    I'm worried about how the people on the other side will uncompress the files. I need it to be as simple as possible for them. Do you know if you can unzip the split volumes using the built-in windows zip, or gzip? – jjnguy May 10 '11 at 14:40
  • Apparently, no, the built-in Windows zip-folder feature doesn't do spanned zip files. That's too bad, since it's been a standard feature of the format since before Windows 3. I'd be very surprised if gzip couldn't do it, though. WinZip definitely can. – Rob Kennedy May 10 '11 at 14:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.