I am currently compressing a list of files from a directory in the following format:

tar -cvjf test_1.tar.gz -T test_1.lst --no-recursion

The above command will compress only those files mentioned in the list. I am doing this because this list is generated such that it fits a DVD. However, during compression the compression rate decreases the estimated file size and there is abundant space left in the DVD. This is something like a Knapsack algorithm.

I would like to estimate the compressed file size and add some more files to the list. I found that it is possible to estimate file size using the following command:

tar -cjf - Folder/ | wc -c

This command does not take a list parameter. Is there a way to estimate compressed file size? I am also looking into options like perl scripts etc.


I think I should provide more information since I have been doing a lot of web search. I came across a perl script(Link)that sort of emulates the Knapsack algorithm.

The current problem with the above mentioned script is that it splits the files in their original state. When I compress the files after splitting them, there are opportunities for adding more files which I consider to be inefficient.

There are 2 ways I could resolve the inefficiency:

a) Compress individual files and save them in a directory using a script. The compressed file could provide a best estimate. I could generate a script using a folder of compressed files and use them on the uncompressed ones.

b) Check whether the compressed file's size is less than the required size. If so, I should keep adding files until I meet the requirement. However, the addition of new files to the compressed file is an optimization problem by itself.


Edit 2:

I re-read your question, and found a new way to interpret it. It is simple to find the size of a file after compression, using something like bzip2 -k $file | wc -c on every file. This, however will not solve your problem, as you are not interested in the size of each compressed file, but the compressed size of a tarball containing all your files (and how much it increases if you add a specific file). The size difference from compression is not linear but in fact rather unpredictable, so the one is a poor predictor for the second.

An example is two identical files of 10 MiB, where both compress individually to 1 MiB and a tarball containing both also compresses to 1 MiB.

Original answer

I believe the common solution if to just split the compressed data with split, which means you need all the resulting disks available if you want to recover a file from any of them. I think tar has a built-in splitting mechanism, but it only works on the amount of uncompressed data, which gives poor results with compression and non-homogenous files.

A different solution (far from perfect) works if you can have manual intervention and might be sufficient. It works best if your files are large compared to the desired volume size

I take you have a target volume size, in MiB. In the example, I use 700 MiB.

If you run

tout="/tmp/09b00c50a9625deeb1089b3c4358c5e5" #or something else
csize="7" # in MiB
mkfifo "$tout"
tar -cvj ~/ 2>"$tout" | dd bs=1M count="$csize" of=/dev/null iflag=fullblock 2>/dev/null &
tail -20 "$tout" | xargs -n 1 ls -ld
rm "$tout"

tar will print out all the files it tries to compress to the pipe, until dd has received 700 MiB of compressed data. tail reads the last 20 files that tar attempted to compress from the fifo.

The last file printed (at least) won't be able to fit within your volume size. Due to caching in both tar and bzip2, several of the other last files might not fit either, though, so you might have to experiment a bit with how many of them you skip. If all the files are very small you might have to tell tail to print out more than 20 of them to see the one that turns you over the limit.


You could even write the filenames to your file list directly (with a suitable filter to remove directories) and just delete the last few lines in a text editor.

| improve this answer | |
  • I have provided some more information in my edit to the post. I am looking forward to hear your thoughts on the same – Sai Mar 19 '12 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.