Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I wanted to backup a path from a computer in my network to another computer in the same network over a 100 Mbit/s line. For this I did

dd if=/local/path of=/remote/path/in/local/network/backup.img

which gave me a very low network transfer speed of something about 50 to 100 kB/s, which would have taken forever. So I stopped it and decided to try gzipping it on the fly to make it much smaller so that the amount to transfer is less. So I did

dd if=/local/path | gzip > /remote/path/in/local/network/backup.img.gz

But now I get something like 1 MB/s network transfer speed, so a factor of 10 to 20 faster. After noticing this, I tested this on several paths and files, and it was always the same.

Why does piping dd through gzip also increase the transfer rates by a large factor instead of only reducing the bytelength of the stream by a large factor? I'd expected even a small decrease in transfer rates instead, due to the higher CPU consumption while compressing, but now I get a double plus. Not that I'm not happy, but I am just wondering. ;)

share|improve this question
512 bytes was the standard block size for file storage in early Unix. Since everything is a file in Unix/Linux, it became the default for just about everything. Newer versions of most utilities have increased that but not dd. – DocSalvager Jun 5 '14 at 20:04
up vote 86 down vote accepted

dd by default uses a very small block size -- 512 bytes (!!). That is, a lot of small reads and writes. It seems that dd, used naively in your first example, was generating a great number of network packets with a very small payload, thus reducing throughput.

On the other hand, gzip is smart enough to do I/O with larger buffers. That is, a smaller number of big writes over the network.

Can you try dd again with a larger bs= parameter and see if it works better this time?

share|improve this answer
Thanks, tried direct copy without gzip and a blocksize of bs=10M -> fast network transfer of something about 3 or 4 MB/s. Higher blocksize + gzip did not change anything compared to small blocksize + gzip. – Foo Bar May 29 '14 at 14:27
If you want to see what high block sizes do try another dd after the gzip. – Joshua May 29 '14 at 16:05
Is gzip doing its own output buffering, or does it just use stdio? – Barmar May 30 '14 at 19:42
@Barmar If I'm reading the source correctly, it simply write(3)s to the buffer. – Cong Ma Jun 3 '14 at 12:28
@CongMa you can also try and use pigz instead of gzip, it will work even faster – GioMac Jan 28 at 14:19

Cong is correct. You are streaming the blocks off of disk uncompressed to a remote host. Your network interface, network, and your remote server are the limitation. First you need to get DD's performance up. Specifying a bs= parameter that aligns with the disks buffer memory will get the most performance from the disk. Say bs=32M for instance. This will then fill gzip's buffer at sata or sas line rate strait from the drives buffer. The disk will be more inclined to sequential transfer giving better through put. Gzip will compress the data in stream and send it to your location. If you are using NFS that will allow the nfs transmission to be minimial. If you are using SSH then you encur the SSH encapsulation and encryption overhead. If you use netcat then you have no encryption over head.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .