I use dd in its simplest form to clone a hard drive:
dd if=INPUT of=OUTPUT
However, I read in the manpage that dd knows a blocksize parameter. Is there an optimal value for the blocksize parameter that will speed up the cloning procedure?
64k seems to be a good pick:
(taken from here).
this matches with my own findings regarding read/write buffering for speeding up an io-heavy converter-program i was once pimping @work.
dd will happily copy using the BS of whatever you want, and will copy a partial block (at the end).
Basically, the block size (bs) parameter seems to set the amount of memory thats used to read in a lump from one disk before trying to write that lump to the other.
If you have lots of RAM, then making the BS large (but entirely contained in RAM) means that the I/O sub-system is utilised as much as possible by doing massively large reads and writes - exploiting the RAM. Making the BS small means that the I/O overhead as a proportion of total activity goes up.
Of course in this there is a law of diminishing returns. My rough approximation is that a block size in the range about 128K to 32M is probably going to give performance such that the overheads are small compared to the plain I/O, and going larger won't make a lot of difference. The reason for the lower bound being 128K to 32M is - it depends on your OS, hardware, and so on.
If it were me, I'd do a few experiments timing a copy/clone using a BS of 128K and again using (say) 16M. If one is appreciably faster, use it. If not, then use the smaller BS of the two.
For those that end up here via Google, even if this discussion is a bit old...
Keep in mind that dd is dumb for a reason: the simpler it is, the fewer ways it can screw up.
Complex partitioning schemes (consider a dual-boot hard drive that additionally uses LVM for its Linux system) will start pulling bugs out of the woodwork in programs like Clonezilla. Badly-unmounted filesystems can blow ntfsclone sky-high.
A corrupt filesystem cloned sector-by-sector is no worse than the original. A corrupt filesystem after a failed "smart copy" may be in REALLY sorry shape.
When in doubt, use dd and go forensic. Forensic imaging requires sector-by-sector copies (in fact, it can require more sectors than you're going to be able to pull off with dd, but that's a long story). It is slow and tedious but it will get the job done correctly.
Also, get to know the "conv=noerror,sync" options, so that you can clone drives that are starting to fail-- or make ISOs from scratched (cough) CDs-- without it taking months.
Yes, but you won't find it without lots of testing. I've found that 32M is a good value to use though.
As others have said, there is no universally correct block size; what is optimal for one situation or one piece of hardware may be terribly inefficient for another. Also, depending on the health of the disks it may be preferable to use a different block size than what is "optimal".
One thing that is pretty reliable on modern hardware is that the default block size of 512 bytes tends to be almost an order of magnitude slower than a more optimal alternative. When in doubt, I've found that 64K is a pretty solid modern default. Though 64K usually isn't THE optimal block size, in my experience it tends to be a lot more efficient than the default. 64K also has a pretty solid history of being reliably performant: You can find a message from the Eug-Lug mailing list, circa 2002, recommending a block size of 64K here: http://firstname.lastname@example.org/msg12073.html
For determining THE optimal output block size, I've written the following script that tests writing a 128M test file with dd at a range of different block sizes, from the default of 512 bytes to a maximum of 64M. Be warned, this script uses dd internally, so use with caution.
I've only tested this script on a Debian (Ubuntu) system and on OSX Yosemite, so it will probably take some tweaking to make work on other Unix flavors.
By default the command will create a test file named dd_obs_testfile in the current directory. Alternatively, you can provide a path to a custom test file by providing a path after the script name:
The output of the script is a list of the tested block sizes and their respective transfer rates like so:
(Note: The unit of the transfer rates will vary by OS)
To test optimal read block size, you could use more or less the same process, but instead of reading from /dev/zero and writing to the disk, you'd read from the disk and write to /dev/null. A script to do this might look like so:
An important difference in this case is that the test file is a file that is written by the script. Do not point this command at an existing file or the existing file will be overwritten with random data!
For my particular hardware I found that 128K was the most optimal input block size on a HDD and 32K was most optimal on a SSD.
Though this answer covers most of my findings, I've run into this situation enough times that I wrote a blog post about it: http://blog.tdg5.com/tuning-dd-block-size/ You can find more specifics on the tests I performed there.
This StackOverflow post may also be helpful: dd: How to calculate optimal blocksize?