0

I have to transfer roughly 250gb of data from a Master Drive to lots of USB drives, all of identical size and brand. When I copy using the GUI in Mac OSX, it waits to copy all the files to one drive and then copies the same files to drive 2.

Is there a way to copy the files across so that each file/bit is only read once and then written multiple times? This would presumably be faster than reading the same data over and over again. I have seen this question asked before, but I couldn't understand how the terminal commands worked.

Just so you know, I have very little experience with terminal, so if anyone has any great code, could they please explain how I would adjust it work with for me?

Many thanks

James

  • If you have any programming experience then you could simply write a program for it. It's not that hard. – ratchet freak Apr 12 '16 at 12:05
  • 1
    Note that the OS should cache the data that is read, so it's still only actually read once. Whether the I/O scheduler is smart enough to parallelize the writes is another matter; I don't think this is a common enough scenario to optimize for. – user Apr 12 '16 at 12:18
  • @user I have the exact same problem as the Asker, your comment simplified my approach a lot! – Caveman Oct 13 at 17:33
2

I know little about Mac OSX, still I believe its standard tools are similar to ones in Linux.

If it was for Linux/Bash I would try:

tar -c /source/dirA/ /source/file1 | tee >(cd /foo/destination3/ && tar -x) >(cd /bar/destination2/ && tar -x) >(cd /foobar/destination1/ && tar -x) > /dev/null

Every destination will receive dirA/ and file1. You may give more arguments to the first tar (directories or files). In order to add another destination simply put additional tee argument in similar manner.

How does it work? First tar converts directories and files to a single bitstream that can be used in a pipe. The tee command forks that stream; every copy but one is extracted by tar in proper destination. The last copy moves down the pipe; it is discarded into /dev/null. (One may use the last copy for destination0 but the syntax would be different so I decided to keep it simple with tee only).

The code is a starting point. You may need to adjust it to work with OSX.

| improve this answer | |
  • Thanks for your detailed answer, sounds like an excellent solution, I happen to have found a rather simple GUI based solution but much appreciated – James Harding Apr 13 '16 at 12:17
  • You’re right; this does “make sense.”  (+1 for paying it forward.) – Scott Apr 10 '17 at 19:09
0

After a little head scratching, my boss reminded me that the free software Davinci Resolve (a colour grading application) now has some data wrangling facilities included. This allows you to select a source folder, multiple destinations and then it will copy each file simultaneously to each drive with one read and multiple writes.

I confirmed this through activity monitor which showed the data written per second to be three times greater than the read, and I was writing to three drives

https://www.blackmagicdesign.com/uk/products/davinciresolve

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.