will rysnc -z have any compression advantage if the input file is already gzipped? I have a large 100GB compressed file to send over the network across servers and it consistently failed(broken pipe) after various amount of time. Wondering if I should try the -z flag.
Compressing in transit an already-compressed file is usually not worth the CPU time. There are caveats. In the process of comparing two files, using rsync with compression can speed up the comparison of hashes of the data.
If you only want to sync compressed versions of large files on more than one system, one place to look would be certain builds of gzip. On an Ubuntu system, I get:
$ gzip -h Usage: gzip [OPTION]... [FILE]... Compress or uncompress FILEs (by default, compress FILES in-place). Mandatory arguments to long options are mandatory for short options too. -c, --stdout write on standard output, keep original files unchanged -d, --decompress decompress -f, --force force overwrite of output file and compress links -h, --help give this help -l, --list list compressed file contents -L, --license display software license -n, --no-name do not save or restore the original name and time stamp -N, --name save or restore the original name and time stamp -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files -t, --test test compressed file integrity -v, --verbose verbose mode -V, --version display version number -1, --fast compress faster -9, --best compress better --rsyncable Make rsync-friendly archive With no FILE, or when FILE is -, read standard input. Report bugs to .
I have a 468MB sql file that I compressed to 57MB with the
$ rsync -avvz --progress -h fooboo:foo.sql.gz . opening connection using ssh fooboo rsync --server --sender -vvlogDtprz . foo.sql.gz receiving file list ... 1 file to consider delta-transmission enabled foo.sql.gz 59.64M 100% 43.22MB/s 0:00:01 (xfer#1, to-check=0/1) total: matches=7723 hash_hits=9468 false_alarms=0 data=22366 sent 54.12K bytes received 22.58K bytes 17.05K bytes/sec total size is 59.64M speedup is 777.59
Not bad. Rsync only had to transfer a small amount of the newer compressed file.
rsync will not make an already compressed file significantly smaller during transit.
It is unlikely that your failed transfers will be fixed by adding the -z flag. I would suggest trying to rsync the file(s) uncompressed. rsync will then compress on the fly. You then have the advantage that should the source file change and you need to rsync again, only the changed bytes will be transferred. If you change a compressed file rsync will most likely have to retransmit it in its entirety. See here for more details:
Here is a guide for linux: http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/ And for Windows: http://www.online-tech-tips.com/computer-tips/how-to-split-a-large-file-into-multiple-smaller-pieces/