Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've read that sometimes files like .mp3's or compressed images aren't worth compressing (with zlib for example) because they are already compressed and they'd waste more CPU resources than save space. Are there any recommended libraries out there that can determine if a file is worth compressing before being transferred?

share|improve this question
if a file can't be compressed it will end up only slightly larger than its original size in the worst case, also to figure out whether it is worth compressing will (naively) require a compression pass – ratchet freak Feb 16 '13 at 19:04
@ratchetfreak To your first point, that's not really what I was asking. Trying to compress an already compressed file wastes CPU cycles. I'm trying to avoid doing so. You mentioned a compression pass though, mind elaborating on that? I assume you mean it needs to run through a pass to determine if it's worth compressing, so it's going to waste resources either way? – Chris Feb 16 '13 at 19:05
then the second part of my comment: assessing compressibility would (for the naive algos) require a compression pass and comparing the sizes, => endresult is that you still had to compress the file – ratchet freak Feb 16 '13 at 19:07
@ratchetfreak So I assume then that, realistically, the only way to save any amount of resources would be to check the file extension and assume it's going to be a waste (or not) based on that alone...? – Chris Feb 16 '13 at 19:11
@ratchetfreak, in most cases file(1) will tell you enough to know if compressing is worthwhile without compressing (depending on the file contents). – vonbrand Feb 17 '13 at 1:35
up vote 1 down vote accepted

Compression in general works by removing redundant constructs out of the data stream in a reversible manner.

One of the measures you can use to find how much of the date can be considered redundant is autocorrelation where the lower it is the less the signal will be compressible (exceptions do exist depending on the algorithm used).

However the lowest order algorithm to find the autocorrelation is of the order of O(n*log(n)). I believe that most compression algorithm are of the order O(n) making it a waste to run it on very long files. (it would be more efficient to just try to compress and check file sizes)

share|improve this answer
Relevant SO post for anyone who's curious, confirming what the OP said above. Huffman and LZW encoding both run in O(n) time. – Breakthrough Feb 16 '13 at 22:40

In StoreBackup, you can define rules. A rule can be a combination of different checks, eg. depending on the file suffix. But there also is a special function, COMPRESSION_CHECK, which decides depending on the contents of a file. It does not make much sense to check every file with COMPRESSION_CHECK, therefore you can define eg. the following:

  1. all files ending .jpg and .mp3 are not compressed without any additional checks
  2. all files ending .doc and .pdf are compressed in all cases without any additional checks
  3. all files smaller than 1k are not compressed without any additional checks
  4. all other files are checked with function COMPRESSION_CHECK

This is a behavior described in the referenced document.

(You could eg. also define rules that additionally never compresses files from a special person or group for easier restore with a file browser, but that's not related your questions)

share|improve this answer
Yeah I saw that, all that does it check the file extensions and size of file though. I was hoping for something more elaborate. – Chris Feb 16 '13 at 19:16
Based on your reply to @ratchetfreak that's the best that can be done, right? – Jasjeev Singh Feb 16 '13 at 19:23
Apparently, I was more or less looking for other options, if any existed. I'd upvote if I could, but I don't have any Rep on SuperUser yet. (Really don't understand why Rep doesn't transfer between Stack sites.) – Chris Feb 16 '13 at 20:40
@Chris once you get over some barrier you get a 100 rep bonus on any new site you sign on on – ratchet freak Feb 17 '13 at 0:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.