12

I am looking for a way to clone single disk drive to more than one disk drive at the same time.

I have prepared system images on 1TB disks, and it takes almost 2 hours to clone one disk to another, and then it goes up linearly, in order to have, say, 30 disks cloned.

Is it possible to clone one disk to more than one target drive simultaneously?

9 Answers 9

17

You can use bash's "process substitution" along with the tee command to do this:

cat drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) | dd of=/dev/sdd

or for clarity (at the expense of a little efficiency) you can make the last dd be called the same way as the others and send the stdout of tee to /dev/null:

cat drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | /dev/null

and if you have it installed you can use pipe viewer instead of cat to get a useful progress indicator:

pv drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) | dd of=/dev/sdd

This reads the source image only once, so the source drive does suffer head-thrashing which will probably be why you see exponential slow-down when you try copy the image multiple times by other methods. Using tee like above, the processes should run at the speed of the slowest destination drive.

If you have the destination drives are connected via USB, be aware that they may all be sharing bus bandwidth so writing many in parallel may be no faster than writing them in sequentially because the USB bus becomes the bottleneck not the source or destination drives.

The above assumes you are using Linux or similar (it should work on OSX too though the device names may be different), if you are using Windows or something else then you need a different solution.

Edit

Imaging over the network has a similar problem to imaging many drives over USB - the transport channel becomes the bottleneck instead of the drives - unless the software you use supports some form of broadcast or multicast transmission.

For the dd method you could probably daisy-chain netcat + tee + dd processes on each machine like so:

  1. Source machine cat/pv/dds the data through nc to destination machine 1.
  2. Destination machine 1 has nc listening for the data from the source machine and piping it through tee which is in turn sending it to dd (and so to the disk) and another nc process which sends to destination machine 2.
  3. Destination machine 2 has nc listening for the data from the destination machine 1 and piping it through tee which is in turn sending it to dd (and so to the disk) and another nc process which sends to destination machine 3.
  4. and so on until the last machine which just has nc picking up the data from the previous machine and sending it to disk via dd.

This way you are potentially using your full network bandwidth assuming that you your switch and network cards have all negotiated a full-duplex link. Instead of the source machine sending 10 copies of the data out (assuming 10 destination machines) so each is limited to 1/10th of the outgoing bandwidth it is only sending 1. Each destination machine is taking one copy of the data and sending it out again. You might need to tweak the buffer size settings of pv, nc and dd to get closer to best practical performance.

If you can find some software that just supports multicast though, that would be much easier (and probably a little faster)! But the above is the sort of hacky solution I might be daft enough to try...

Edit Again

Another thought. If the drive image compresses well (which it will if large chunks of it are full of zeros) the outgoing bandwidth of the source machine need not be a problem even if sending to many destinations at once. Just compress the image first, transmit that to everywhere using tee+nc, and decompress on the destinations (network->nc->decompressor->dd->disk).

4
  • Thanks for in-depth explanation of the process. It was really helpful. I am exploring network cloning options right now (using multicast), and if it fails, I'll explore this DD vector some more.
    – mr.b
    May 26, 2010 at 15:13
  • You didn't say that the drives were attached to different computers...that makes things completely different!
    – marcusw
    May 26, 2010 at 17:30
  • This comes closest to both local computer and network of computers one-to-many disk duplication, simultaneously. Thanks for in-depth explanation!
    – mr.b
    May 31, 2010 at 14:03
  • For Network cloning options that are Linux based, you may want to consider Clonezilla. It can clone via multicast as well as several local cloning options including DD.
    – user35060
    Sep 12, 2013 at 16:35
5

First answer on google suggested (on a Linux system): dd if=/dev/sdb of=- | tee >(dd of=/dev/sdc) >(dd of=/dev/sdd) >(dd of=/dev/sde), where /dev/sdb is the hard drive you want to clone and /dev/sdc, /dev/sdb, and /dev/sde are drives to clone to (you can add as many more of these as you want, just copypaste). A LiveCD should do it, and remember to be careful with your drive letters!

5
  • 1
    True. I did some research prior to asking this question, but I was interested in hearing other people experiences. DD is a great little utility, but: 1) it lacks awareness of actual data content (it will copy empty space, too, sector-by-sector), and 2) someone has reported performance issues when doing clone to more than one drive in this way. Any more ideas?
    – mr.b
    May 26, 2010 at 15:11
  • "[...]where /dev/sdb is the hard drive you want to clone and /dev/sdc, /dev/sdb, and /dev/sde are drives to clone to[...]" ---> "[...]and remember to be careful with your drive letters!" :D I Agree!!!
    – dag729
    May 26, 2010 at 17:17
  • 1
    @mr.b: 1) In my mind, a bit-for-bit copy is better than taking potentially dangerous shortcuts. 2) Meh, the best way I can think of is to mess with the source of dd to make it copy to more than one location simultaneously, but it would be quicker to just copy the slow way.
    – marcusw
    May 26, 2010 at 17:28
  • Are you sure about the of=-? The just creates a - output file rather than outputting to stdout for me. This can be resolved by just leaving out the of option. Jul 30, 2011 at 12:41
  • Using "tee" that way won't work. Please see: joshhead.wordpress.com/2011/08/04/…
    – user35060
    Sep 12, 2013 at 16:37
2

All i know is that there are some things called Hard Drive Duplicators. These are special Devices to clone (duplicate) HDs to multiple Drives at the same time. Maybe this article helps you.

3
  • Yeah, I am aware of existence of these devices; however, they are everything but cheap, especially in my zero-budget case :( Thanks for mentioning them, though.
    – mr.b
    May 26, 2010 at 15:12
  • 1
    While not completely zero-cost, if you have a spare machine (with a decent power supply) that you could commandeer and can afford a couple of cheap SATA controllers (there is a two-port one in my home machine that cost a tenner) you could make your own simple duplication machine and use the dd+tee method or other software if you can find some that supports "read from single source, write to many" efficiently. May 26, 2010 at 23:14
  • (This is what I assumed you were trying to do in my initial answer.) May 26, 2010 at 23:20
1

Since nobody has mentioned it yet, I'll mention Clonezilla and their Server Edition. (unfortunately, there doesn't appear to be a direct link to it, but you can find "Server Edition" in the site's left nav menu...)

I've had great luck with Clonezilla Live edition but have yet to try Server Edition. Looks pretty slick though.

1
  • I'll second Clonezilla as what appears to be a good solution for you based on the feedback you've provided.
    – user35060
    Sep 12, 2013 at 16:39
1

If you are using Mac OS X this is built in. From the machine your are going to serve the image from start a multicast asr session. From the clients launch to the boot disk, open terminal, and connect to the asr multicast stream. Free.

Details: http://www.bombich.com/mactips/multicast.html

1

I found 2 useful links on the web relating to this. One used dd without cat to do the diskdupe:

dd if=/dev/sdb | tee >(dd of=/dev/sdc) | tee >(dd of=/dev/sdj) | dd of=/dev/sdh

http://joshhead.wordpress.com/2011/08/04/multiple-output-files-with-dd-utility

This is further expanded with another link to show a progress meter:

dd if=/dev/sdb | pv -s $(blockdev --getsize64 /dev/sdb) | tee >(dd of=/dev/sdc) | tee >(dd of=/dev/sdj) | dd of=/dev/sdh

http://www.commandlinefu.com/commands/view/6177/dd-with-progress-bar-and-statistics

1

So many answers here base on

cat … | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) … | dd …

or similar contraption, while sole tee can read from an image/disk and write to disks, like this:

<source tee /dev/sda /dev/sdb … >/dev/sdz

Remember tee writes to its stdout, that's why the last destination is provided as a redirection.

You probably need sudo tee … to write to sda and such. The quirk is you need an elevated shell in the first place to set up the redirection to /dev/sdz. This can be solved by:

<source sudo tee /dev/sda /dev/sdb … /dev/sdz >/dev/null

but even this won't work if you need sudo to access the source. Then (and virtually only then) cat may be useful:

sudo cat source | sudo tee /dev/sda /dev/sdb … /dev/sdz >/dev/null

Or you can spawn an elevated shell for the task:

sudo sh -c '<source tee /dev/sda /dev/sdb … >/dev/sdz'

My point is you don't need process substitution, you don't need dd or cat. Reading and writing can be done with tee alone. It won't show you progress though, so pv is a sane addition:

pv source | tee /dev/sda /dev/sdb … >/dev/sdz

or (if sudo needed):

sudo pv source | sudo tee /dev/sda /dev/sdb … /dev/sdz >/dev/null

Still without >(…), useless cat or dd.

1
  • This is a very neat solution! Thank you :) Also, it is quite cool to see that this problem is still somehow relevant even 12 years after I asked the question ;)
    – mr.b
    Jan 27, 2022 at 13:34
0

I wanted to expand on David's answer

pv drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) | dd of=/dev/sdd

The drive.image can actually be another device, like /dev/sde

Second, the dd command will work magnitudes faster with a proper bs setting. I used bs=64k and saw a 6 times speed increase for copying a 40 Gig partition, from 1 hour to 10 minutes.

So the final command will look like this:

pv drive.image | tee >(dd bs=64k of=/dev/sda) >(dd bs=64k of=/dev/sdb) >(dd bs=64k of=/dev/sdc) | dd bs=64k of=/dev/sdd

If your source is a drive instead of a file, it'll look like this:

pv /dev/sde | tee >(dd bs=64k of=/dev/sda) >(dd bs=64k of=/dev/sdb) >(dd bs=64k of=/dev/sdc) | dd bs=64k of=/dev/sdd

0

Parallel hard disk duplication is a common task in computer forensics. dc3dd (man page) is a dedicated tool which allows parallel copying of a single source toward multiple destinations and works like UNIX dd, with multiple of= options allowed.

It is also possible to enable the hashing of the source volume and of the copies to verify their integrity.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .