Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 3 down vote favorite
2

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network overhead a regular copy cp remote_folder/* ~/local_folder/ takes a very long time (10 mins).

I believe it's because the files are being copied sequentially – each file waits until the previous is finished before the copy begins.

What's the simplest way to increase the speed of this copy? (I assume it is to perform the copy in parallel.)

Zipping the files before copying will not necessarily speed things up because they may be all saved on different disks on different servers.

share|improve this question
Zipping the files before copying will speed things up massively because there will not need to be any more "did you get that file", "yes, I did", "here's the next one", "okay", ... It's those "turnarounds" that slow you down. – David Schwartz Jan 15 at 17:19
It's probably disk speed, rather than network speed, that is your limiting factor, and if that is the case then doing this per file in parallel will make the operation slower, not faster, because you will force the disk to constantly seek back and forth between files. – Joel Coehoorn Jan 15 at 17:20
While zipping might not be a good idea (running compression algo over 1000s of files might take a little while), tar might be viable. – Rob Jan 15 at 17:27
@JoelCoehoorn still, there are cases when this is not the case: e.g. multiple spindles + small files (or simply random reads). In this scenario, "parallel cp" would help. – CAFxX Apr 13 at 6:37

4 Answers

up vote 4 down vote accepted

As long as you limit the copy commands you're running you could probably use a script like the one posted by Scrutinizer

SOURCEDIR="$1"
TARGETDIR="$2"
MAX_PARALLEL=4
nroffiles=$(ls $SOURCEDIR|wc -w)
setsize=$(( nroffiles/MAX_PARALLEL + 1 ))
ls -1 $SOURCEDIR/* | xargs -n $setsize | while read workset; do
  cp -p $workset $TARGETDIR &
done
wait

Expanded as requested

share|improve this answer
Note of warning though: This script breaks with filenames containing spaces or globbing characters. – slhck♦ Aug 24 '11 at 21:56
@OldWolf -- Can you explain how this script works? For example, which part does the parallelization? – dsg Aug 24 '11 at 22:42
2  
@dsg: The & at the end of the cp command allows the while loop to continue and start the next cp command without waiting. The xargs command passes the filenames in groups of 4 (MAX_PARALLEL) to the while loop. – RedGrittyBrick Aug 24 '11 at 23:21
Doesn't worked for me. I'm not sure it is possible to speed up cp. You obviosly can speed up calculation through the multithreading. But I don't think same holds for hard drive data coping. – Adobe Sep 4 '11 at 17:12
up vote 2 down vote

One way would be to use rsync which will only copy the changes - new files and the changed parts of other files.

http://linux.die.net/man/1/rsync

Running any form of parallel copy operation will probably flood your network and the copy operation will just grind to a halt or suffer from bottlenecks at the source or destination disk.

share|improve this answer
up vote 0 down vote

There are many things one may have to consider depending on the topology you have. But before you start thinking about complex solutions, you could simply try to divide the task to two jobs and check if the time needed will reduce significantly:

The next time try:

  cp remote_folder/[a-l]* ~/local_folder/ &
  cp remote_folder/[!a-l]* ~/local_folder/ &
  wait
  wait

(you may want to replace [a-l]* to something else that matches about half of the files - maybe [0-4]* - depending on the contents of the folder)

If time improves not dramatically it may be more important to check if it's neccessary to copy all files (what's the ratio of changed files to all files?)

share|improve this answer
up vote 0 down vote

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel -j10 cp {} destdir/ ::: *

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.