Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've got a set of .tar.gz files, which are duplicity backup files (either full backups or incremental ones). I'd like to compute which directories take the most space on backups. This will most probably be a different figure to calculating which directories take the most space on a live filesystem because I need to account for how often are files changing (and therefore taking space on incremental backups) and how compressible are files.

I know that while many other archive formats store compressed files as different entities inside the archive file, .tar.gz files do not, and therefore it is impossible to get an exact amount of storage taken in the archive by a single file after compression. Are there any tools to calculate at least some estimates?

share|improve this question

2 Answers 2

If you are interested in a particular file size after compression, just compress the file with gzip once. That should be the most straight forward method.

share|improve this answer
    
Well, I've got almost a terabyte of backups, and I'd like to compute sums from every compressed file… that would take quite a lot of time. –  liori Nov 7 '12 at 3:37
    
Take a full backup, dump it to some big empty disk. Then run ** gzip -r < top level dump dir > ** . You can break the process into smaller chunk. It does take time but you only do it once. –  John Siu Nov 7 '12 at 3:58
    
I don't have such free space. –  liori Nov 7 '12 at 8:27
up vote 0 down vote accepted

So, I hacked some C code to find some approximate values. The code shows how many bytes did zlib to read from archive to get to each subsequent file. The code is here: https://github.com/liori/targz-sizes

It seems that I could extract more precise data, but these values shouldn't differ from real ones by more than by few bytes per file, and the error is averaged over all files, so it should be good enough for the purpose described in the question.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.