124

Suppose that I have 10,000 XML files. Now suppose that I want to send them to a friend. Before sending them, I would like to compress them.

Method 1: Don't compress them

Results:

Resulting Size: 62 MB
Percent of initial size: 100%

Method 2: Zip every file and send him 10,000 xml files

Command:

for x in $(ls -1) ;  do   echo $x ; zip "$x.zip" $x ; done

Results:

Resulting Size: 13 MB
Percent of initial size: 20%

Method 3: Create a single zip containing 10,000 xml files

Command:

zip all.zip $(ls -1)

Results:

Resulting Size: 12 MB
Percent of initial size: 19%

Method 4: Concatenate the files into a single file & zip it

Command:

cat *.xml > oneFile.txt ; zip oneFile.zip oneFile.txt

Results:

Resulting Size: 2 MB
Percent of initial size: 3%

Questions:

  • Why do I get such dramatically better results when I am just zipping a single file?
  • I was expecting to get drastically better results using method 3 than method 2, but don't. Why?
  • Is this behaviour specific to zip? If I tried using gzip would I get different results?

Additional info:

$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 4.4.4 20100525 (Red Hat 4.4.4-5) for Unix (Linux ELF) on Nov 11 2010.

Zip special compilation options:
    USE_EF_UT_TIME       (store Universal Time)
    SYMLINK_SUPPORT      (symbolic links supported)
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)
    UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
    STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
    UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
    [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)

Edit: Meta data

One answer suggests that the difference is the system meta data that is stored in the zip. I don't think that this can be the case. To test, I did the following:

for x in $(seq 10000) ; do touch $x ; done
zip allZip $(ls -1)

The resulting zip is 1.4MB. This means that there is still ~10 MB of unexplained space.

  • 34
    If I'm not mistaken, it's this phenomona that causes people to make .tar.gz as opposed to just zipping the whole directory. – corsiKa Dec 14 '15 at 20:21
  • 18
    A similar question was already asked, tl;dr use solid 7zip archives. – Dmitry Grigoryev Dec 14 '15 at 20:37
  • 3
    @sixtyfootersdude As a test to validate some of the answers, can you try zipping the zip produced in method 3? I suspect this will reduce the file size to something comparable to method 4. – Travis Dec 14 '15 at 21:52
  • 7
    Instead of $(ls -1), just use *: for x in *; zip all.zip * – muru Dec 15 '15 at 1:21
  • 4
    If you want to do solid compression with ZIP, here's a workaround: first, create an uncompressed ZIP containing all your files. Then, put that ZIP inside another compressed ZIP. – user20574 Dec 15 '15 at 23:17
128

Zip treats the contents of each file separately when compressing. Each file will have its own compressed stream. There is support within the compression algorithm (typically DEFLATE) to identify repeated sections. However, there is no support in Zip to find redundancy between files.

That's why there is so much extra space when the content is in multiple files: it's putting the same compressed stream in the file multiple times.

  • 9
    It's also why some compression tools give you the option of compressing the files separately or as a single entity. (Though generally that also means you have to decompress more of the archive than you would otherwise if you want to view just a single file in it.) – JAB Dec 14 '15 at 19:35
  • 28
    @JAB: Compression tools like 7z and rar use the term "solid" archive for packing multiple files head to tail into larger compression streams. With a moderate chunk size like 64MiB, random access to a single file might require decompressing up to 64MiB of data from the start of the compression block that it's in. You can get a decent tradeoff between random-access and finding cross-file redundancy. 7z can use the more effective (but slower to compress) LZMA compression scheme, which is another advantage over zip. – Peter Cordes Dec 15 '15 at 4:29
  • Are you saying that there is no support in Zip to find redundancy between files is in the zip file specification? – sixtyfootersdude Dec 16 '15 at 15:12
  • 6
    @sixtyfootersdude Many compression algorithms, such as DEFLATE, operate as a stream. To recover enough information to decompress a part of the stream, you need to process the entire stream up to that point. If they tried to find redundency between files, you'd have to decompress all 1000 files in order to get to the last one. This is typically how tgz works, actually. However, zip was designed to let you extract individual files. tgz is designed to be more all-or-nothing – Cort Ammon Dec 16 '15 at 21:46
  • 1
    @sixtyfootersdude - that's correct. To paraphrase Cort: The pkzip specs don't support working cross-file. If they did then extracting one file might require the whole archive (and every file) to be extracted. – James Snell Dec 18 '15 at 11:40
48

ZIP compression is based on repetitive patterns in the data to be compressed, and the compression gets better the longer the file is, as more and longer patterns can be found and used.

Simplified, if you compress one file, the dictionary that maps (short) codes to (longer) patterns is necessarily contained in each resulting zip file; if you zip one long file, the dictionary is 'reused' and grows even more effective across all content.

If your files are even a bit similar (as text always is), re-use of the 'dictionary' becomes very efficient, and the result is a much smaller total zip.

  • 3
    ZIP does both archiving and compressing. Does this mean that ZIP compresses each file individually, even if they all end up in the same ZIP-file? – gerrit Dec 14 '15 at 20:00
  • 2
    it kind of has to - imagine you remove a single file, you don't wouldn't want it to spend another half hour re-compressing the rest with a new 'dictionary'. - also, it probably assumes that different files need very different 'dictionaries'. – Aganju Dec 14 '15 at 20:02
  • 2
    I don't see why it has to. With Unix tools, I would first archive a file with tar, then compress it with gzip/bz2/lzma. The compression algorithm doesn't care how many files are encoded in the archive. Also, how common is it really to remove a single file from a compressed archive? I don't think I've ever done that. – gerrit Dec 14 '15 at 20:07
  • 4
    I don't disagree, and that is probably a good way. I didn't design or write ZIP. I just said what it does... – Aganju Dec 14 '15 at 20:10
  • 16
    @gerrit It has its own problems. Zip is designed to allow you to quickly access any file in the archive - try unpacking a single file from a 100 GiB UHA archive and you'll see why they chose this way. It's also designed for appending - you can have your backup zip and just keep adding (or replacing) files as needed. All of this is a huge help when using archives. The trade-off is that if you're compressing files that are very similar (which is not all that common), it can't exploit the similarities to reduce archive size. – Luaan Dec 15 '15 at 8:55
43

In Zip each file is compressed separately. The opposite is 'solid compression', that is files are compressed together. 7-zip and Rar use solid compression by default. Gzip and Bzip2 can't compress multiple files so Tar is used first, having the same effect as solid compression.

As the xml file have similar structure and probably similar content if the files are compressed together the compression will be higher.

For example if a file contains the string "<content><element name=" and the compressor already found that string in another file it will replace it with a small pointer to the previous match, if the compressor doesn't use 'solid compression' the first ocurrence of the string in the file will be recorded as a literal which is larger.

9

Zip doesn't just store the contents of the file, it also stores file metadata like the owning user id, permissions, creation and modification times and so on. If you have one file you have one set of metadata; if you have 10,000 files you have 10,000 sets of metadata.

  • 3
    Good point, but the system meta data is just taking up 1.4MB of space. See my edit. – sixtyfootersdude Dec 14 '15 at 18:02
  • 1
    I'm not familiar with the zip algorithm, but the metadata isn't just the file information, but also things like size and a dictionary, possibly some information on distribution of characters. A dictionary on a non-empty text file will be non-zero. That's probably why you see the metadata being larger in your xml files than your empty files. – Ben Richards Dec 14 '15 at 18:54
  • This was my first thought. Zip-File Header Information – WernerCD Dec 14 '15 at 19:40
  • This only explains the difference between 2 and 3 - not 4. – Luaan Dec 15 '15 at 8:56
  • @Luaan No, in both 2 and 3 the metadata for all 10,000 files is included in the zip file or files, so the total file size is almost the same size. In 4, there's only metadata for one file, and the zip file is much smaller. – Mike Scott Dec 15 '15 at 12:01
7

An option missed by the OP is to zip all of the files together with compression turned off, then zip the resulting zip with compression set to maximum. This roughly emulates the behavior of *nix .tar.Z, .tar.gz, .tar.bz, etc. compressed archives, by allowing the compression to exploit redundancies across file boundaries (which the ZIP algorithm cannot do when run in a single pass). This allows the individual XML files to be extracted later, but maximizes the compression. The downside is that the extraction process requires the extra step, temporarily using much more disk space than would be needed for a normal .zip.

With the ubiquity of free tools like 7-Zip to extend the tar family to Windows, there's really no reason not to use a .tar.gz or .tar.bz, etc., as Linux, OS X, and the BSDs all have native tools to manipulate them.

  • gzip and bzip2 might end up even worse because they are designed with compressing streams in mind, so they will have to start outputting compressed data before all of the data to compress is even known. – rackandboneman Dec 16 '15 at 12:31
  • @rackandboneman: This is the tradeoff you have to make when compressing files larger than the amount of memory you're willing to use at compression time. (And also, the amount of CPU time required to find anything globally optimal would be huge.) A huge compression dictionary can also increase the memory required for decompression. This is an option for LZMA (xz / 7-zip). Anyway, adaptive dictionaries can pick up on patterns once they're visible. It's not like it just builds a static coding system based on the first 32k. This is why gzip doesn't suck. – Peter Cordes Dec 21 '15 at 11:13
  • I really like this "trick" if you need to stay with the zip format. I disagree with your "no reason not to use 7-zip"—if I'm sending a file to a non-technical friend, I want to be sure they will be able to open it easily. If I'm sending to a business-client, even more-so. – Wowfunhappy Mar 14 at 20:45
5

The zip compression format stores and compresses each file separately. It doesn't take advantage of repetition between files, only within a file.

Concatenating the file allows zip to take advantage of repetitions across all of the files, resulting in drastically more compression.

For example, say each XML file has a certain header. That header only occurs once in each file but is repeated almost identically in many other files. In methods 2 and 3 zip couldn't compress for this but in method 4 it could.

  • 3
    How is this different from one of the top 3 answers already posted 5hrs earlier? – Xen2050 Dec 15 '15 at 5:35
  • 1
    @Xen2050 Not much difference, I just thought I could explain it more clearly. – BonsaiOak Dec 15 '15 at 23:35
  • 1
    @BonsaiOak - then add a comment to the correct answer or edit if you have enough rep. If not, but your comment adds clarity, someone else might pick this up and edit the post anyway. – AdamV Dec 17 '15 at 13:09
  • @AdamV I see your point. My answer doesn't currently add any useful information, although it arguably did when I wrote it. There are already appropriate comments under the first answer so I don't see the point in adding them, either. Are you saying that I should just close my answer? What harm is there in leaving it open? – BonsaiOak Dec 18 '15 at 22:10
4

Next to the metadata Mike Scott mentioned there is also overhead in the compression algorithm.

When compressing a bunch of individual small files you’ll have to be very lucky to be able to compress them that it just happens to fill one compression block. When compressing a single monolithic block the system can just continue to stream data to its algorithm, ignoring the ‘boundaries’ (for lack of better word) of the individual files.

Also ASCII is known to have a high compression factor. plus xml is often very repetitive making the metadata a large chunk of the data that can’t be so easily compressed as the xml content.

Lastly, if memory serves right, zip uses something like dictionary encoding , which is especially effective on ascii files and even more so on XML due to their repetitivity

Data Compression Explained : http://mattmahoney.net/dc/dce.html

3

Consider this XML:

<root>
  <element id="1" />
  <element id="2" /> 
  <other id="3" />
  ...
</root>

An XML has a very repetitive structure, Zip takes advantage of those repetitions to build a dictionary of which pattern has more occurrences and then, when compressing, uses less bits to store more repeated patterns and more bits to store less repeated pattern.

When you concatenate those files, the source file (the source for zip) is big but contains much more repeated patterns because de distribution of the boring structures of an XML are amortized in the big whole file, giving the chance to ZIP to store those pattern using less bits.

Now, If you combine different XML into a single file, even when those files have completely different tag names, the compression algorithm will found the best pattern distribution across all files and not file by file.

Ultimately the compression algorithm has found the best repeated pattern distribution.

-1

In addition to the 7-Zip answer there's another approach that's not as good but would be worth testing if for some reason you don't want to use 7-Zip:

Compress the zip file. Now, normally a zip file is incompressible but when it contains a lot of identical files the compressor can find this redundancy and compress it. Note that I have also seen a small gain when dealing with large numbers of files without redundancy. If you really care about size it's worth trying if you have an awful lot of files in your zip.

  • That only works if you do the first zip with compression turned off as I mention above. – Monty Harder Dec 18 '15 at 21:38
  • @MontyHarder I've seen it work with the compression turned on. – Loren Pechtel Dec 18 '15 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.