I have a lot of files, mostly png and pdf, some svg, some various others, that need to be transferred. Multiple TB worth, files individually aized anywhere from a few kb to a few mb. The problem is that I have to transfer them remotely from a site that has poor upload (200kb 0 700kb/sec).

It's important I back up these files asap, but currently over the wire is the only way. This is going to take weeks, which is far too long. Unfortunately, I don't have anyone local I can trust to manually tranfer and send me the hard drive. So I'm left with compressing as much as possible to small files.

I have used 7-Zip on Ultra compression (LZMA2) to archive the files and split into 10mb chunks (in different groups for different categories of files). This worked great.

If I do the same thing to all of those together, compress all the groups (separated by folders) into a single 7-Zip Ultra LZMA2 10mb split archive, will it help at all, or will it be a waste of time? Will it be able to merge data to compress to higher degree and result in less overall 10mb files?

NOTE: The reason I broke it into chunks is because it will take a long time to transfer no matter what, so small chunks will prevent a network failure, computer restart, or any other issue to screw up the whole thing.


Sorry, but this will largely be a waste of time.

The way that data compression works is to identify patterns/assumptions, and represent those in a more efficient way. However, the end result doesn't tend to create compressible patterns.

It is possible to take some data which has been compressed rather lousy, and compress it more aggressively, and get some gain. Usually the gain is less than 3%, with 10% on quite rare instances. However, there is a little bit of overhead. So other results, which are quite common, are approximately 0% savings, with some cost, so you may actually increase file size.

Feel free to try it if you like, but the common results of such efforts are improvements which are negligible or non-existent, or even making things worse.

The fact is that not all data can be compressed. The "counting" argument, also known as the "pigeon hole" principle, explains why. (See: Compression FAQ section 8.) Basically, if compressed data is equal or smaller (using fewer bits), there are fewer possible compressed files than uncompressed files, which proves that not every possible unique uncompressed file can be represented with fewer bits.

In fact, most data is uncompressible. Fortunately for us, most interesting data is compressible. For instance, most images don't look like random black and white pictures (e.g., “snow” on old TV sets that displayed random monochrome pixels of random intensity). Most spreadsheets don't actually have completely random numbers (including numbers in the negative trillions, and fractions of one that have forty three decimal places, and plus signs just before division signs).

Even text has patterns, such as heavy usage of vowels, and occasional usage of punctuation marks followed by spaces and then capital letters.

However, compressed data tends to take the useful patterns, and represent them in efficient ways. So the data compression process removes inefficiencies. The result is that there is typically little inefficiency that we can identify and store more efficiently.

| improve this answer | |
  • 1
    Thanks. This was a great answer. "The way that data compression works is to identify patterns/assumptions, and represent those in a more efficient way. However, the end result doesn't tend to create compressible patterns." This is what I needed to know. Since they were groups of image files, I was mostly wondering if the crossover patterns would match, but if it's rearranged after compression (of course for efficiency) then it makes sense that there wouldn't be many. – Cherokee Oone Jun 18 '16 at 19:30
  • "most data is uncompressible" - what types of uncompressible data are you referring to? – chris Jun 8 '18 at 19:33
  • @ChrisAnderson : What do you mean by "type"? The very word "type" tends to suggest that you're referring to some sort of type, format, rules, and/or patterns. Such things may often be compressible (if designed just for simplicity or speed, and likely less so if already compressed or designed to be space-efficient). However, most possible combinations of random data would not have such patterns, and therefore isn't any specific "type" other than "random". Well-randomized data is pretty uncompressible, and makes up for most possible bit patterns that aren't "interesting"/useful/usable. – TOOGAM Jun 9 '18 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.