1

Let's say you have a lot of files to archive. They are in several different file formats (txt, docx, pdf, png, jpg, mp3, zip, etc.).

What are the best practices to compress them all, so that they take as little space as possible?

For example, according to that question, it's basically useless to compress one or several already compressed files with the same compression algorithm, but you could achieve better compression rates by using different algorithms.

Another example: is it best to compress files having the same file format together?

I'm not interested in which compression format is best in general, but it would be interesting to know if some of them perform better in some cases.

1
  • "several different file formats." ... for example...? I'd also bear in mind the 'bit rot' part of the linked answer, RAR & PAR are more resilient over time & a far better investment than merely saving a few bytes.
    – Tetsujin
    Jul 20, 2018 at 8:56

1 Answer 1

1

First off, other than text and PDF files, everything you listed is already compressed. DOCX files are gzip (well, really DEFLATE, but they use a gzip compatible header) compressed XML, PNG uses DEFLATE, JPG and MP3 do their own thing (the combination of algorithms used by each is specific to their format), and ZIP files use either DEFLATE, or on occasion BZip2. Of these, only JPEG and MPEG are likely to get any significantly better compression ratios, but that's only if they were minimally compressed to begin with, and even then the gains are likely to be minimal. The PDF files may not compress very well either, as they may be mostly images and not text, which are likely already compressed too (usually using JPEG compression).

Now, that out of the way, on to your primary questions:

Does grouping similar file types help?

Sometimes yes, sometimes no. If the files are all smaller than the block size of the compression algorithm, it may help, possibly quite a lot. If they're all bigger though, it usually won't help much. In the case of text files (either plain text, or files that encapsulate it like PDF), grouping files of the same language can help a lot if the files are smaller than the compression block size, because there will generally be a significant amount of redundancy in the data.

Whether or not this is likely to help in your case is something you're unfortunately going to have to test.

Are certain algorithms better in certain cases than others?

Absolutely. JPEG and MPEG layer 3 are examples of this. Both are optimized for compressing a very specific type of data (either images or audio). Brotli is another good example of one that's better in some circumstances, it's optimized for streaming of textual data. Most of the compression formats you would be likely to use though are general purpose, which usually means they do a great job at compressing things like textual data, and a rather poor job at compressing data which isn't structured into a byte-wise stream (DEFLATE is a general purpose algorithm, a fact which really shows when you compare the size of a PNG image to an otherwise identical JPEG image).

Given that you have a lot of mixed data, you probably shouldn't worry too much about this though.

What's the most efficient way to archive lots of data so it takes up minimal space?

Probably some variant of PAQ. The PAQ algorithms are generally considered to be the best (in terms of compression ratio) general purpose compression algorithms that are widely available. They also take FOREVER to compress any reasonably sized amount of data, so they may not be practical in your particular case. More realistic options in terms of how long they take include:

  • XZ: This uses LZMA compression with some extra pre-processing that lets it do an above average job of compressing machine code. Widely available on every platform except Windows (though you can get it on Windows), and generally gets really good compression ratios (LZMA is one of the current gold standards for a compression algorithm that gets reasonable performance and good compression ratios).
  • zstd: This is a newer one developed by the great software engineers working for Facebook. In most cases, it runs faster than XZ and gets comparable (usually slightly better in my experience) compression ratios. Not as widely available yet, but worth trying if you can get it on your system.

In any case, you need an archive format to group the files together if you want them all easily handled in one place. Tar is the format I usually use, but I mostly deal with Linux. A ZIP file with no compression (you can do this on Windows using the command line) will work too.

If you have a very large number of files, I would suggest grouping them int a number of smaller archives instead of one big one. This will hurt your compression ratios a bit, but will save you a lot of time later when you need to pull stuff out of the archive, and makes it easier to handle recovery of a damaged archive.

If you are going to properly set up recovery data for your archive, do it for whatever the final format you're going to store on-disk is (so, the final, compressed, archive file or files). A single bit change in a compressed data stream can completely change the result of uncompressing that stream, so correcting errors before decompressing is a bit easier than doing it afterwards (because decompressing will amplify the size of the error).

1
  • It seems there is no "solid block size" in 7zip when using zip compression, so how can I make sure that "the files are smaller than the compression block size" in this case?
    – CidTori
    Sep 7, 2018 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.