What is the largest size a gzip (say 10kb for the sake of an example) can be decompressed to?
Add a comment
|
7 Answers
Update 2020-02-06: As mentioned in the comments, I have been unable to reproduce the original result with gzip. Working on the assumption that I accidentally used a different compression format in that original quick test I've repeated with gzip and updated the figures below accordingly. This new result agrees with the theoretical maximum compression stated in other answers/comments.
It very much depends on the data being compressed. A quick test with a 1Gb file full of zeros using a standard version of gzip (with default options or specifying -9) gives a compressed size of ~1018Kb, so your 10Kb file could potentially expand into ~10Mbytes.
If the data has low redundancy to start with, for instance, the archive contains images files in a format that is compressed natively (gif, jpg, png, ...), then gzip may add not further compression at all. For binary files like program executables you might see up to 2:1 compression, for plain text, HTML or other markups 3:1 or 4:1 or more is not unlikely. You might see 10:1 in some cases but the ~1030:1 seen with a file filled with a single symbol is something you are not going to see outside similarly artificial circumstances.
You can check how much data would result from unpacking a gzip file, without actually writing its uncompressed content to disk, with gunzip -c file.gz | wc --bytes - this will uncompress the file but not store the results, instead passing them to wc which will count the number of bytes as they pass then discard them. If compressed content is a tar file containing many many small files you might find that noticeably more disk space is required to unpack the full archive, but in most circumstances, the count returned from piping gunzip output through wc is going to be as accurate as you need.
answered May 9, 2010 at 13:11
-
1I've seen HTML expand to 10x (of course x3 and x4 was the most common!).... perhaps a lot of redundant data for those ones that were exploding +8x. I think the page in question that was doing that was a php info page.– ZombiesMay 10, 2010 at 12:10
-
Repetitive markup, as seen in the output of
phpinfo(), compresses very well. The technical information in that output contains more direct repetition than the average chunk of natural language would too, and the alphabet distribution is probably less smooth which could help the Huffman stage get better results. May 10, 2010 at 12:55 -
This answer doesn't account for intentionally malicious compressed data. One can craft a malicious zip file around 10KB that can expand to a bit over 4GB. Jan 2, 2013 at 2:36
-
Zip bombs of that scale rely on nested archives though, so as a human unpacking the file you would noticed something odd before long. They can be used as an effective DoS attack against automated scanners (on mail services and so forth) though. Jan 2, 2013 at 11:47
-
1@DavidSpillett: Nested zip bombs expand into sizes in the petabyte range. That's not what I'm talking about. Look at even just a single layer of a typical zip bomb. Jan 2, 2013 at 23:48
Quoted verbatim from https://stackoverflow.com/a/16794960/293815
The maximum compression ratio of the deflate format is 1032:1. This is because the longest run that can be encoded is 258 bytes. At least two bits are required for each such run (one bit for the length code and one bit for the distance code), hence 4*258 = 1032 uncompressed bytes can be encoded per one compressed byte.
You can get more compression by gzipping the result of gzip. Normally that doesn't improve compression, but for very long runs it can.
By the way, the LZ77 approach used by deflate is more general than run-length encoding. Instead of just a length, a length/distance pair is used. This allows copying a string from some distance back, or replicating a byte as in run-length for a distance of one, or replicating triples of bytes with a distance of three, etc.
Usually you don't get more than 95% compression (so that 10kB gzipped data would decompress to ~200kB), but there are specially crafted files that expand exponentially. Look for 42.zip, it decompresses to few petabytes of (meaningless) data.
-
5
-
6Indeed, 42.zip is specifically a danger to tools that automatically scan zip files recursively, for example virus scanners. Feb 5, 2014 at 0:42
-
7
The compression ratio of any compression algorithm will be a function of the data being compressed (besides the length of that data).
Here is an analysis at MaximumCompression,
Look at one of the samples like,
Summary of the multiple file compression benchmark tests
File type : Multiple file types (46 in total) # of files to compress in this test : 510 Total File Size (bytes) : 316.355.757 Average File Size (bytes) : 620,305 Largest File (bytes) : 18,403,071 Smallest File (bytes) : 3,554
10 MB of zeros in file, compress with gzip -9 to 10217. So maximal ratio looks to be around 1000x.
A huge file containing only one symbol will compress very well.
The answer to your question, depends the input. To give you an idea how compression is done watch this six minutes videos.
https://www.youtube.com/watch?v=ZdooBTdW5bM
What you should get from it is that the compression rate depends on the frequency of each character, thus there is no generel max rate, it depends on the input, for english text it is about 65 percent.
-
3Welcome to Super User! Please quote the essential parts of the answer from the reference link(s), as the answer can become invalid if the linked page(s) change.– DavidPostill ♦Oct 21, 2016 at 11:03
-
2It would be more accurate to say "frequency of each string" rather than "frequency of each character"– JoelFanFeb 9, 2017 at 19:33