I have a large (80GB) virtual drive file which I want to compress but, before I do, I want to see how compressible it actually is, before embarking on creating another (potentially) big file (if it turns out that the data cannot be compressed much).

Is there a way to do this? i.e. analysing an existing set of data to see whether there is much room for compression without actually writing an output file?

  • "without actually compressing" means "without creating a file" or "without spending CPU cycles"? – Kamil Maciorowski Aug 24 '18 at 7:46
  • Without creating a file, yes. Have clarified the question, thanks for the comment. – mydoghasworms Aug 24 '18 at 7:47
  • Note (regarding your original question): tar does not compress. – Kamil Maciorowski Aug 24 '18 at 7:54
  • Oh yes, sorry 😬 – mydoghasworms Aug 24 '18 at 7:55

Any compressing tool that can write to stdout can be used. Just pipe it to wc -c. Example:

gzip -c -9 < bigfile | wc -c

Note the tool does its job and stresses your CPU regardless of where the output goes.

  • 2
    I would add pv bigfile | gzip | wc -c or even pv -cN raw bigfile | gzip | pv -cN zipped | wc -c if you want a progress output. – user1686 Aug 24 '18 at 8:07
  • So there is no way to just analyse the data to be compressed, to see by how much it would be compressed, without actually compressing it, right? Because if I understand correctly what is going on here (because maybe I'm not) then it is actually compressing the data to figure out how big the result is? – mydoghasworms Aug 24 '18 at 8:45
  • @mydoghasworms I guess some heuristics that work in some cases are possible; but I can't say I've seen a tool that works this way. – Kamil Maciorowski Aug 24 '18 at 8:49
  • Thanks, in that case I will accept your answer. I started the process with @grawity 's recommendation to use pv for the progress output. Will still take in excess of half an hour to find how much this 80GB file can be compressed. In terms of time and energy costs vs storage costs, it's probably more worth it to copy the file as is :-) It's an interesting exercise anyway, so thanks for the answer! – mydoghasworms Aug 24 '18 at 8:52
  • 1
    @mydoghasworms This is what the second pv is for. – Kamil Maciorowski Aug 24 '18 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.