I want to TAR and compress a user's home directory as efficiently as possible. The problem is there are a lot of large and already compressed (gzip/bzip2) and hard-to compress files (binaries/data) in this user's home directory as well as a lot of easy to compress files (source code). Is it possible to skip compression of already-compressed and/or hard-to-compress files while only compressing the easy ones?
The main goal is to maximize data transfer for a GigE network via something like:
tar cf - path | gzip -c | nc host port
If I don't compress at all, the bottle neck is the network. If I do compress, the bottle neck is the CPU time. Disk I/O is not a problem. The directory is about 150TB total, but I'm stuck with a single GigE path.