What's the best way to perform a parallel copy on Unix?

Question

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network overhead a regular copy cp remote_folder/* ~/local_folder/ takes a very long time (10 mins).

I believe it's because the files are being copied sequentially – each file waits until the previous is finished before the copy begins.

What's the simplest way to increase the speed of this copy? (I assume it is to perform the copy in parallel.)

Zipping the files before copying will not necessarily speed things up because they may be all saved on different disks on different servers.

Answer 1

As long as you limit the copy commands you're running you could probably use a script like the one posted by Scrutinizer

SOURCEDIR="$1"
TARGETDIR="$2"
MAX_PARALLEL=4
nroffiles=$(ls $SOURCEDIR|wc -w)
setsize=$(( nroffiles/MAX_PARALLEL + 1 ))
ls -1 $SOURCEDIR/* | xargs -n $setsize | while read workset; do
  cp -p $workset $TARGETDIR &
done
wait

Expanded as requested

Answer 2

One way would be to use rsync which will only copy the changes - new files and the changed parts of other files.

http://linux.die.net/man/1/rsync

Running any form of parallel copy operation will probably flood your network and the copy operation will just grind to a halt or suffer from bottlenecks at the source or destination disk.

Answer 3

There are many things one may have to consider depending on the topology you have. But before you start thinking about complex solutions, you could simply try to divide the task to two jobs and check if the time needed will reduce significantly:

The next time try:

  cp remote_folder/[a-l]* ~/local_folder/ &
  cp remote_folder/[!a-l]* ~/local_folder/ &
  wait
  wait

(you may want to replace [a-l]* to something else that matches about half of the files - maybe [0-4]* - depending on the contents of the folder)

If time improves not dramatically it may be more important to check if it's neccessary to copy all files (what's the ratio of changed files to all files?)