62

I often have to gather log files and upload them to a central server (Owned by another company). The central server has a size limit of the file, so I am trying to create the smallest file possible that is still in the zip format.

What are the best setting to use when compressing a text file to a zip format when my only need is a small file size?

7zip Options

I've done the obvious and chosen ultra compression, and I have noticed that LZMA does a better job than deflate, but there are far too many other permutations of options for me to test them all.

  • 1
    Is splitting the the zip in to multiple files an option? – JaredMcAteer May 10 '11 at 14:21
  • 3
    As soon as you pick anything but the Deflate format, it's not a "normal" .zip file anymore, but an "extended" zip file, pioneered by WinZip. They originally kept the extension as .zip, to much consternation (since most normal zip-handling tools can't deal with them), but most archivers use .zipx now to distinguish them from traditional .zip files. If you can use LZMA, switch to .7z and pick PPMd -- it should compress better (and faster!) for text files. – afrazier May 20 '11 at 16:04
  • 1
    @afrazier: "The .ZIP File Format Specification documents the following compression methods: stored (no compression), Shrunk, Reduced (methods 1-4), Imploded, Tokenizing, Deflated, Deflate64, bzip2, LZMA (EFS), WavPack, PPMd." en.wikipedia.org/wiki/Zip_%28file_format%29#Compression_methods – endolith Dec 13 '13 at 22:26
  • 1
    @endolith: bzip2, lzma, wv, and ppmd are all very recent additions to the file format. It's not even safe to assume that your recipient can handle deflate64, much less anything newer. – afrazier Dec 13 '13 at 22:33
  • 1
    define "normal zip tools". Most "normal zip tools" nowadays like 7z and winrar can extract 7z files. – phuclv May 29 '16 at 11:04
61

To create the smallest standard ZIP file that 7-Zip can create, try:

7z a -mm=Deflate -mfb=258 -mpass=15 -r foo.zip C:\Path\To\Files\*

Source: How can I achieve the best, standard ZIP compression?

Otherwise if you don't care about the ZIP standard, use the following ultra settings:

7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on archive.7z dir1

Which are:

-t7z   7z archive

-m0=lzma
       lzma method

-mx=9  level of compression = 9 (Ultra)

-mfb=64
       number of fast bytes for LZMA = 64
-md=32m
       dictionary size = 32 megabytes

-ms=on solid archive = on
  • 2
    @Tek: Why? It's not a good one. The question was about using the "standard ZIP format", so the answer shouldn't be specifying LZMA. -ms=on is for .7z, not standard zip files. -md is related to BZip2, so I don't expect it to affect ZIP (or even LZMA). -mfb=64 is an unoptimized value: -mfb=258 makes smaller zip files. And this answer doesn't even mention -mpass=15 which can affect zip files. This is a nicely formatted answer which is, unfortunately, wrong in multiple ways. – TOOGAM Nov 8 '15 at 12:32
  • 6
    I would use lzma2 – Lance Badger Jul 7 '16 at 15:22
  • If you look at the 7-zip FAQ, it states that newer versions of 7z may have worse performance than older versions in some circumstances. Read the FAQ for more detail, but in short use the 'qs' in Parameters field in the GUI or use -mqs in the command line version to use the old sort by file extension method. 7-zip.org/faq.html. – drojf May 15 at 14:21
13

If you can use .7z format rather than just .zip, I would simply use PPMD with the following options and leave everything else as set by the Compression Level:

  • Archive Format: 7z
  • Compression Method: PPMD
  • Compression Level: Ultra

I regularly compress server/text logs (60MB+) using these options and they usually come out at 1-2% of the original size.

  • For text such as log files, ppmd is definitely the way to go. However, the question mentioned that it needed to stay in the zip format, which may not work with PPMD. – Brian Minton Dec 19 '13 at 16:31
  • Just tried zip with PPMD and Windows Explorer opens the contents up without complaint here on Windows 7 – Umber Ferrule Dec 20 '13 at 16:44
  • 3
    I noticed that too. It opens the contents up just fine. However, when I actually tried to view one of the files inside the zip file, it failed. – Brian Minton Dec 23 '13 at 16:58
  • 2
    Why is PPMD superior compression method for text files? – user598527 Feb 27 '17 at 19:58
  • 1
    LZMA2 gives better results for text files than PPMD. – T3rm1 Nov 22 '18 at 10:38
7

I compare for db.fdb 1,2 GB (1236598784 B) in Ubuntu server 14.04.03 with p7zip [64] 9.20 on VM:

1. 7z a -mx=9 1.7z db.fdb
2. 7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on 2.7z db.fdb
3. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on 3.7z db.fdb
4. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on -pass=15 4.7z db.fdb
5. 7z a -mx=9 -mmt=on 5.7z db.fdb
6. 7z a -t7z -m0=lzma -mx=9 -mfb=258 -md=32m -ms=on -mmt=on 6.7z db.fdb

and have that results:

1.7z 96 MB (100108731 B) with 6' 25"
2.7z 95 MB ( 99520375 B) with 5' 18"
3.7z 93 MB ( 97512311 B) with 9' 19"
4.7z 93 MB ( 97512345 B) with 9' 40"
5.7z 96 MB (100108731 B) with 5' 26"
6.7z 93 MB ( 97512311 B) with 9' 09"

I think second method works fine = (almost) best compress with best time. But for best "view" and easy to remember is first method - with small files and no point of max compress. Between 2 and 3 method we don't get extra smaller 7z but pay almost twoo more time for compression. Anyone decide with his own.

7

After much experimentation, digging into the detailed 7zip documentation, and reading some of the 7z source code regarding the advanced LZMA2 parameters, here is a better method below. It reduced some 1GB real-world test files more than 2 to 4 times better than the previously accepted solutions posted here or even in the 7z manpage.

7z a -t7z -mx=9 -mfb=273 -ms -md=31 -myx=9 -mtm=- -mmt -mmtf -md=1536m -mmf=bt3 -mmc=10000 -mpb=0 -mlc=0 archive.7z inputfileordir

The LZMA2 compression is assumed here, but you might be able to get even better performance in 7zip with passing advanced LZMA2 options like -m0=LZMA2:27, or -m0=LZMA2:d25, or an array of parameters like

-m0=BCJ2 -m1=LZMA:d25 -m2=LZMA:d19 -m3=LZMA:d19 -mb0:1

Such parameters didn't seem to be respected by the 7z versions I tested, but you may want to explore further or patch the 7z code to properly parse them. Or maybe it is supposed to work and is just broken in the builds that were tested.

  • wow, this made a really big difference. For my archive, I experimented with a lot of other suggestions, including other answers here, and the best result I got was 99MB, vs 85MB using these settings. – user9399 Aug 25 at 23:42
0

Set the "split to volume, bytes" field to the server's maximum allowed file size (in bytes, I think, although it looks like it accepts common abbreviations like "KB" and "MB"). If the zip file exceeds that size, 7-zip will split it into multiple files automatically, such as integration_serviceLog.zip.001, integration_serviceLog.zip.002, etc. (Way back when, PK Zip used this to span zip files across multiple floppy disks.) You'll need all the files to be present to unzip them. Use that instead of worrying about the absolute best compression settings to use for any particular set of files, because what's best for one file may be different for another file, and you don't want to have to go through this every time you need to copy logs.

  • 1
    I'm worried about how the people on the other side will uncompress the files. I need it to be as simple as possible for them. Do you know if you can unzip the split volumes using the built-in windows zip, or gzip? – jjnguy May 10 '11 at 14:40
  • Apparently, no, the built-in Windows zip-folder feature doesn't do spanned zip files. That's too bad, since it's been a standard feature of the format since before Windows 3. I'd be very surprised if gzip couldn't do it, though. WinZip definitely can. – Rob Kennedy May 10 '11 at 14:47

protected by Community Nov 4 at 11:46

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.