Scripting compression (ZIP or TAR) to be done in chunks

Question

I'm trying to find a utility or approach out there that will allow me to compress an entire directory into chunks. I know it's easy to specify, for example, that the archive files created should be exactly X size or smaller, but the archive utilities usually make it so you need all the archive files to open the archive, and that's what I'm trying to avoid. I need to be able to specify a maximum size of an archive file and it adds files into it until it's going to run out of space on the next file so it starts a new archive file. That way the archive files are technically independent of each other.

I'm sorry, it's hard for me to put this into words precisely. Please comment if I'm not being specific enough.

Answer 1

This is a concept that's interested me for a while, although I haven't got around to writing a script for it yet as I've been a little busy!

There is no easy way for software to really know what something will compress to, without running the files through the algorithm first (keep in mind, tar is just an archiver, not a compression utility, but you can use it with gzip). The other (easier) option would be to keep adding files to the archive, then check the size each time. I don't know of any premade solutions out there to achieve this, but it can be implemented in a few lines of code!

The basic logic would be something like this:

This is dependent on which utility you use as well. Some do not have switches which allow you to easily remove a file from the archive, so you may need to add a second input variable for leeway. This second variable could determine your estimated max size of a file in the directory, and if the archive is more than that size away from the preferred size, it would add it to the file. If there wasn't enough room, the archive would close and create a new one. Of course, you could tweak this algorithm into infinity, ensuring the size of the next file is checked and compared with the current archive size. You'd probably also want to use the biggest files first, so as you can see a lot of logic can go into this.

Unfortunately this isn't a full solution, but I hope this is a good starting point!

Answer 2

The only way I can see to do this would be to guess at the size of the compressed files. If the files are all of similar types (text, images, etc) then you can probably guess fairly well how large the final compressed file will be after combining x number of them.

I don't think anything does what you're asking for out of the box as it would require compression->check size->compress again type of behavior.

Answer 3

I'm assuming you are backing up a big directory onto a stack of CDs, and you want to be able to pull a file off a CD by sticking in 1 CD (rather than needing to put in 2 or more CDs out of the multi-CD archive).

Perhaps the simplest way to meet your requirement is to individually compress each file into its own little ".zip" file, and then copy those compressed files to the CDs. (There's a way to store the sub-sub-sub-directory that the original file came from in the ".zip" file, so when you restore that file, it gets put back into the proper location, even though all the ".zip" files on disk are in one long list in a single directory).

Once you have a list of zip files, you could start copying from the top of the list, and when the CD is full, eject and resume copying from that point on the list with the next CD. That leaves a little "wasted space" at the end of each CD. Some people, if they get to a small file further down the list that fits in that space, will go back and put that little file into that otherwise wasted space. A few people obsessively attempt to re-arrange which file goes on which disk in order to pack them all as full as possible.

This approach -- independently compressing each file -- sacrifices some disk space in order to gain a little convenience.

Answer 4

As far as i know, afio supports multi-volume archiving, and allows restoring data from arbitrary archive files. But files can span two volumes so you might end up having to do something like

cat archive-3.afio archive-4.afio | afio -i -k -