9

I have a set of files I would like to compress that I know to be repetitive and compressible, but 7zip chooses a non-optimal order to compress the files and fails to take advantage of their compressibility. How can I get 7zip to compress the files in another order?

The files I want to compress are the following:

  • A 200MB PDF containing a large number of embedded JPGs
  • 190MB of JPGs, all of which are separately embedded in the PDF
  • About 500MB of miscellaneous other moderately compressible files

I know it is possible for 7zip to take advantage of the repetition between the PDF and the bare JPGs because when I archive just the PDF and the JPGs together, I get a compression ratio of 47%. But when I try to include the 500MB of other files, 7zip compresses the JPGs first, then the miscellaneous other data, and by the time it gets to the PDF, the compression algorithm must have 'forgotten' about the JPGs because the PDF is hardly compressed at all.

With 7-zip 9.32 alpha, using the 7z archive format, ultra compression level, LZMA2 algorithm, 256MB dictionary size, 128 word size, 4GB solid block size, and 2 CPU threads, I get the following compression ratios:

  • PDF only: 93%
  • JPGs only: 95%
  • PDF and JPGs together: 47%
  • Misc. files only: 44%
  • Misc. files and PDF: 55%
  • Misc. files and PDF and JPGs: 63%

Since the misc. files are compressible to 44% of their original size, and the PDFs and JPGs together are compressible to 47%, I would expect everything together to be compressible to somewhere on the lower end of 44-47%, but due to the poor ordering of files by 7zip, I get a significantly worse result.

I have tried to alter the order 7zip compresses files by playing with file creation, modification, and access dates. I have tried moving the files to another folder and copying them back so they are rewritten to disc consecutively. I have even tried archiving all the JPGs in a zip file with store-level compression, so that their filesize will approximately match the PDF. No matter what I do, I can't seem to get 7zip to compress the PDF and the JPGs without the misc. files in between.

Any ideas? I am unable to increase the dictionary size due to memory constraints.

2 Answers 2

6

I managed to solve this problem. The solution was to create an archive containing only the miscellaneous files, and after that select "add to archive" from the explorer context menu while selecting both the PDF and JPGs. In the 7zip "Add to Archive" dialog, I chose the same compression settings and archive name as before.

This compressed the PDF and JPGs together - taking full advantage of their redundancy - then added them to the existing archive. It resulted in an overall 45% compression ratio, exactly what I was looking for.

0

PDFs and JPGs are themselves compressed. It is much harder to compress an already compressed file then an uncompressed one. So no, the compression algorithm isn't being "forgotten" - it is more akin to throwing a zip file into another zip file (very little happens in compression).

8
  • 2
    As I already showed though, 7zip IS able to take advantage of the repetition between the JPGs and the PDF which contains embedded copies of the same JPGs.
    – Sam Page
    Feb 4, 2014 at 5:22
  • PDFs compress their local copies of the JPEGs, which make the two copies (one in and one out) very different in binary form.
    – Claudia
    Feb 4, 2014 at 5:25
  • And yet when I archive the PDF and the JPGs together - two copies of the same image data - 7zip is able to detect the duplication and compress the files down to 47% of their original size, smaller than either the PDF or the JPGs alone.
    – Sam Page
    Feb 4, 2014 at 5:28
  • 1
    Compress only the PDF file first. Then open that newly created archive in 7zip and click-and-drag the JPG files into it. Finally, click-and-drag the unrelated files into it.
    – user294732
    Feb 5, 2014 at 2:48
  • 1
    It seems as though every time you add files to an existing archive, 7zip takes advantage of the redundancy between just the new files and completely ignores the files already in the archive. For that reason, your solution of adding the PDF seperately from the JPGs didn't work, user294732, but it did lead me on to the correct solution which was to create an archive containing only the misc. files, then to add the PDF and JPGs in a single operation. Presumably creating an archive of just the PDF and JPGs, then adding the misc. files would work too.
    – Sam Page
    Feb 5, 2014 at 3:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .