tar reverse of '--to-command', or: compressing tar entries individually, or: tar with plenty of large files

Question

I would like to create tar gzip archive, but do it the reverse manner of what is most commonly done -- have the files in the archive be compressed individually rather than compress the entire archive: that way it retains the seekable property it should have. It makes much more sense to me, and I don't know why this has not been favored.

I have some ideas on how to do this:

• use an archive tool with compression, such as 7zip
• write my own script to do this by compressing the files first, then feeding them to tar (already found a handy Python script to write a tar file and the Python gzip library seems easy to use.

However, ideally, I would like to continue to use tar for this, as it is a familiar, de-facto tool for archiving where I work. tar has the --to-command switch, which allows piping extracted files to a program. If I had a symmetric command such as --from-command I would easily implement my wish with:

tar cf my_archive.tar file1 file2 --from-command=gzip
tar xf my_archive.tar --to-command=gunzip

My motivation comes from dealing with archives containing a large number of large files. I currently tar-gzip them, but then extracting any files from the archive takes a long time - it needs to be decompressed before tar can access the file, and it does so in a serial manner!

So here are my questions:

• Is there an evident way to achieve this that I am disregarding?
• Has anyone already written a tool to do with, specifically with tar?
• If one would call tar and gzip and standard methods of archiving and compressing in Linux, what would be the equivalent, popular method for archiving with compression in the manner I mentioned about (i.e. not tar.gz)
• Is there another way I am overlooking to circumvent the large amount of time it takes to extract a file from a large tar-gzipped archive?

Thanks!

EDIT

I realize that I need to re-phrase and refine my question. Especially since, as Robin Hood pointed out, there are existing rather easy solutions to create compressed archives (namely, zip). So here it is:

Is there are way to use tar that allows true random-access to the archive while still keeping it compressed? If not, is there another tar-replacement for Linux (that is built with the same rationale, and, ideally, with support for the same command-line options), that does achieve this?

Right now I can replace tar in a general sense with zip, by changing:

tar c path/to/file1 path/to/file2 | gzip > arc.tar.gz
gunzip < arc.tar.gz | tar x

to:

zip -qr - path/to/file1 path/to/file2 > arc.zip
unzip -qoX test.zip

However, this has the disadvantage that it does not support all the options that tar does for archiving, namely:

1. piping each extracted file individually into a pipe (the --to-command switch)
2. unzip does not accept an archive in standard input. funzip does, however - it only outputs the first file in the archive

So it's rather limiting.

Thanks again!

Answer 1

I've read your question multiple times, it's very difficult to understand, but I think I've got it now. You want to have files put into individual tar archives, and then all stored in one gz archive. This won't work because gz archives only support compression of 1 file which is why people tar the files before compressing with gz. You could do the opposite, put each file into a gz archive, and then put all the gz archives into a single tar archive. Alternatively, you could just stop using formats that require double archiving, and use an archive format that supports multiple files, like zip.

Compressing the files inside a tar will still result in sequential access of the gz archives, because the tar format doesn't support random access. Zip archives use a centralized catelogue so random file access is possible without decompressing, or reading the entire archive. I don't do much archiving under Linux, but on Windows I like to use 7-zip to create zip archives with lzma compression. It's worth noting that either of these methods when used with comparable compression to your tar.gz, will yeild a larger archive due to lack of solid compression, which is why tar.gz is very popular in the linux world compared to zip for distributing software.

Create A Series Of GZ Archives And Store In A Tar Archive:

cp -a -n -v "/home/me/example/inputfiles/." --target-directory="/home/me/example/gzfiles"

This will copy the files you wish to archive to a different folder. Gunzip doesn't allow keeping the original unarchived files, but working from a copy will enable you to avoid this.

gzip -9 "/home/me/example/gzfiles/*"

This will create a seperate gz archive of each file, and use max compression. If your system can't handle that try a lower number; the default number is 6.

tar -cf "/home/me/example/tar/archive.tar" -C "/home/me/example/gzfiles ."

This will create a single tar archive that contains all the gz archives.

Extract A Single File From A GZ Archive In A Tar Archive:

sudo apt-get install archivemount

This will install archive mount, a tool that can mount tar files to a directory.

archivemount -o readonly "/home/me/example/tar/archive.tar" "/home/me/example/mount"

This will mount the tar archive so that you can extract the desired gz archive. I believe it is possible to extract individual files from a tar archive with tar, but I don't know the command, hence why I'm using this approach.

gunzip -c "/home/me/example/mount/example1.txt.gz" > "/home/me/example/extract1/example1.txt"

This will extract the file. Gunzip only supports extracting to the source directory, or to standard output so in this command we've used standard output and then piped the output to a file.

sudo umount "/home/me/example/mount"

This will unmount the tar archive.

Extract All Files From A Series Of GZ Archives In A Tar Archive:

cd "/home/me/example/extractall"

This puts the terminal into the directory you want to extract to since tar extracts to the current directory.

tar -xf /home/me/example/tar/archive.tar

This extracts the gz archives.

gunzip *.gz

This extracts the contents of the gz archives to the current directory /home/me/example/extractall/ and removes the gz archives.

Create A ZIP Archive:

cd "/home/me/example/inputfiles"

This puts the terminal into the inputfiles directory since zip creates an archive from the current directory, and saves to it.

zip -9 -r inputfiles inputfiles.zip *

This will create a zip archive of all the inputfiles directory contents, excluding hidden files, and use max compression. p7-zip would be a better tool to use if you need high compression.

mv "/home/me/example/inputfiles/inputfiles.zip" "/home/me/example/zip/archive.zip"

This will allow you to rename the archive what-ever you wish, and move it where you want.

Extract A ZIP Archive:

cd "/home/me/example/zip"

This puts the terminal into the directory containing the zip.

unzip -n archive.zip

This extracts the zip archives contents to the current directory.

Answer 2

If what you want is individually compressed files in an archive with random access, then dar ("Disk ARchive") may be what you are looking for. Newer versions support LZMA compression, the algorithm used by 7-Zip. It is also possible to define filters to store some file types uncompressed and save time, e.g. media files and archives that already have their own compression. My favorite feature is compressing existing (uncompressed) archives so I can quickly make a backup now and run the CPU intensive LZMA compression at a more convenient time or on a more powerful machine:

dar --empty-dir \
  --fs-root /home \
  --create home-backup-2016-01-11 \
  --prune lost+found

And then later and/or elsewhere:

dar -+ home-backup-2016-01-11-compressed-encrypted \
  -A home-backup-2016-01-11 \
  -zxz:6 \
  -K "aes:" \
  -an -ag -Z "*.mpg" -Z "*.avi" -Z "*.flac" -Z "*.cr2" \
  -Z "*.vob" -Z "*.jpg" -Z "*.jpeg" -Z "*.mpeg" -Z "*.png" \
  -Z "*.mp3" -Z "*.ogg" -Z "*.deb" -Z "*.tgz" -Z "*.tbz2" \
  -Z "*.rpm" -Z "*.xpi" -Z "*.run" -Z "*.sis" -Z "*.gz" \
  -Z "*.Z" -Z "*.bz2" -Z "*.zip" -Z "*.jar" -Z "*.rar" \
  -Z "*.xz" -Z "*.dar" -Z "*.7z" -acase

As shown above, encryption is also possible, all while still allowing extraction of individual files. However, dar does not seem to have an equivalent to the --to-command. It's hard to tell from your question whether you intended to used that feature for anything but decompression.

(Yes, I know this question is old. This is for the people who, like me, googled "tar compress individually" and got this as the first result.)