I want to TAR and compress a user's home directory as efficiently as possible. The problem is there are a lot of large and already compressed (gzip/bzip2) and hard-to compress files (binaries/data) in this user's home directory as well as a lot of easy to compress files (source code). Is it possible to skip compression of already-compressed and/or hard-to-compress files while only compressing the easy ones?

The main goal is to maximize data transfer for a GigE network via something like:

tar cf - path | gzip -c | nc host port

If I don't compress at all, the bottle neck is the network. If I do compress, the bottle neck is the CPU time. Disk I/O is not a problem. The directory is about 150TB total, but I'm stuck with a single GigE path.


tar does not compress, gzip does it to what it gets from tar. But gzip gets only one thing to deal with (stdout of tar) here, so even if it could pick things to compress and others not to, there is not a second item to choose.

What you can do is use find for compressed files and tar-tee-netcat them without re-compression. The file tee wrote can be used for a second tar-gzip-netcat cycle as a exception list given to tar (-X).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.