Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large file (~20 GB) and I want to quickly make an (approximate) estimate of how well it can compress. Searching for a program which can provide an immediate guess at the compressed filesize, and possibly increasing accuracy of the estimate the longer it is left to analyse the data.

share|improve this question

With the -v option gzip prints the compression ratio, so just as a quick and dirty estimate you could do something like this:

$ dd if=myfile bs=1M count=100 2>/dev/null | gzip -v >/dev/null
 80.5%
$


Of course, while quick and straightforward, just checking the compression ratio of the first x bytes (100 Mbytes in this example) could be very misleading if the file contents are not uniform.
As an example, it would probably not be great for a filesystem image.

share|improve this answer

Without sifting through all the data, there is no way to know how compressible it is. Feel free to look at this page for lists of compression tools and speeds, but what you are asking for is basically impossible.

share|improve this answer
3  
I think it's possible to guess the compressability looking at some random fragments looking for possible patterns or measuring the entropy. Of course it will be coarse, but as the OP says, it's a series of successive estimates. – clabacchio May 8 '12 at 7:29
    
It does not have to be perfect, just a guess - some GUI tools such as winrar provide a running guess of the output filesize as it's compressing. I'm looking for a linux based program. – wim May 8 '12 at 7:53
1  
Winrar does not provide a guess. It tells you the compression ratio for the data that it has compressed already. – soandos May 8 '12 at 15:18
    
It does provide you a guess of the resulting filesize! Of course assumptions are made about the entropy, but an exact prediction is not necessary - just an estimate. – wim Apr 2 at 16:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.