Good block size for disk cloning with diskdump (dd)

Question

I use dd in its simplest form to clone a hard drive

dd if=INPUT of=OUTPUT

However, I read in the manpage that dd knows a blocksize parameter. I was wondering whether there is an optimal value for the blocksize parameter that will speed up the cloning procedure?

Cheers!

Stefan

Answer 1

64k seems to be a good pick:

Results:

  no bs=        78s     144584+0 records
  bs=512        78s     144584+0 records
  bs=1k         38s     72292+0 records
  bs=2k         38s     36146+0 records
  bs=4k         38s     18073+0 records
  bs=5k         39s     14458+1 records
  bs=50k        38s     1445+1 records
  bs=500k       39s     144+1 records
  bs=512k       39s     144+1 records
  bs=1M         39s     72+1 records
  bs=5M         39s     14+1 records
  bs=10M        39s     7+1 records

(taken from here).

this matches with my own findings regarding read/write buffering for speeding up an io-heavy converter-program i was once pimping @work.

Answer 2

Yes, but you won't find it without lots of testing. I've found that 32M is a good value to use though.

Answer 3

dd will happily copy using the BS of whatever you want, and will copy of partial block.

Basically, the block size (bs) parameter seems to set the amount of memory thats used to read in a lump from one disk before trying to write that lump to the other.

If you have lots of RAM, then making the BS large (but entirely contained in RAM) means that the I/O sub-system is utilised as much as possible by doing massively large reads and writes - exploiting the RAM. Making the BS small means that the I/O overhead as a proportion of total activity goes up.

Of course in this there is a law of diminishing returns. My rough approximation is that a block size in the range about 128K to 32M is probably going to give performance such that the overheads are small compared to the plain I/O, and going larger won't make a lot of difference. The reason for the lower bound being 128K to 32M is - it depends on your OS, hardware, and so on.

If it were me, I'd do a few experiments timing a copy/clone using a BS of 128K and again using (say) 16M. If one is appreciably faster, use it. If not, then use the smaller BS of the two.

Answer 4

First, there are much better tools to clone a hd than dd. A simple file copy ( cp -a ) is a perfectly good way. If you want to keep the same fs UUID and the boot loader intact ( as well as any fragmentation present ), then clonezilla or ghost4linux are good candidates.

Because dd is a dumb animal, it wastes time copying free space, and requires that the destination be at least as large as the source, and any additional size is unused, requiring you to manually expand the fs later.

As for what block size is best, it depends on a number of factors. The largest is whether the source and destination are different partitions on the same disk, or different disks ( on different controllers ). Whether you use the direct flag is important as well. If you do not enable direct and you are copying between two disks, then very large block sizes ( above 1 MB ) actually slow things down, since the block cache will make dd wait a bit after such a large write before it can read from the source again so you don't keep both disks busy. If you are copying between two partitions on the same disk, then this delay can actually be a good thing and larger block sizes are better because there are fewer seeks.

I usually find using the direct flag with 32 or so MB block size works best going between partitions on the same disk, and 128-512k either without the direct flag, or with direct but piping one input dd to another output dd for drive to drive.