19

I have about 200,000 files that I am transferring to a new server today. I haven't done anything on such a large scale before, and wanted to get some advice on how I should go about this. I am moving them between two Centos 6 distros, and they are in different locations in the country. I don't have enough HDD space on the original server to tar up all of the directories and files into one massive tarball, so my question is how should I transfer all of these files? rsync? some special way of using rsync? Any input/suggestions on how to do it would be amazing.

Thanks

EDIT: For those wondering, i HIGHLY suggest using a screen when running a large rsync command like this. Especially when something silly may occur and you lose the connection to the server A which you are running the rsync command from. Then just detach the screen and resume it later.

|improve this question
  • 4
    Have you tried rsync yet? Maybe on a small set of files or so? Should be the ideal tool for that. – slhck Mar 26 '13 at 18:14
  • It's almost certainly not the best tool for this job, but you may be interested in the fact that you can stream tar through an ssh connection rather than having to compress to a file before moving the file: tar cz | ssh user@example.com tar xz – Aesin Mar 27 '13 at 1:00
  • 2
    it could be off topic, but (especially for an initial load, and then using rsync for subsequent updates) : "Never underestimate the bandwidth of a station wagon full of tapes" (ie: have you considered placing a 2nd hd (or plug a usb2/usb3 disk), backup on it, and send that one via fedex to the remote location? It could be MUCH faster than anything else, and save bandwidth for other uses. – Olivier Dulac Mar 27 '13 at 9:10
  • I don't have any BW limits on one provider, and the other I won't reach this month. So I don't really have an issue wasting it :P – MasterGberry Mar 27 '13 at 15:54
  • 1
    @OlivierDulac what-if.xkcd.com/31 – Bob Mar 28 '13 at 10:07
24

Just to flesh out Simon's answer, rsync is the perfect tool for the job:

   Rsync  is  a  fast  and extraordinarily versatile file copying
   tool.  It can copy locally,  to/from  another  host  over  any
   remote  shell,  or to/from a remote rsync daemon.  It offers a
   large number of options  that  control  every  aspect  of  its
   behavior  and permit very flexible specification of the set of
   files to be copied.  It is famous for its delta-transfer algo‐
   rithm,  which reduces the amount of data sent over the network
   by sending only the differences between the source  files  and
   the  existing  files in the destination.  Rsync is widely used
   for backups and mirroring and as an improved copy command  for
   everyday use.

Assuming you have ssh access to the remote machine, you would want to do something like this:

rsync -hrtplu path/to/local/foo user@remote.server.com:/path/to/remote/bar

This will copy the directory path/to/local/foo to /path/to/remote/bar on the remote server. A new subdirectory named bar/foo will be created. If you only want to copy the contents of a directory, without creating a directory of that name on the target, add a trailing slash:

rsync -hrtplu path/to/local/foo/ user@remote.server.com:/path/to/remote/bar

This will copy the contents of foo/ into the remote directory bar/.

A few relevant options:

 -h,                         output numbers in a human-readable format 
 -r                          recurse into directories
 -t, --times                 preserve modification times
 -p, --perms                 preserve permissions
 -l, --links                 copy symlinks as symlinks
 -u, --update                skip files that are newer on the receiver
 --delete                    delete extraneous files from dest dirs
 -z, --compress              compress file data during the transfer
 -C, --cvs-exclude           auto-ignore files in the same way CVS does
 --progress                  show progress during transfer
 --stats                     give some file-transfer stats
|improve this answer
  • so could i do rsync -hrtplu --progress if I wanted to see the progress as it is going along? – MasterGberry Mar 26 '13 at 18:54
  • @MasterGberry yup, exactly. I have a backup script that runs rsync --progress --stats -hrtl --update source destination. – terdon Mar 26 '13 at 18:57
  • I seem to be having issues getting it to run. rsync -hrtplu --progress --rsh='ssh -p2202' is what i am using and it can't connect. Keep getting 255 error. But I am sshd'd into it. So I know its not the firewall...do I need to provide the password via the cmd also? or wouldn't it just ask me after for it? – MasterGberry Mar 26 '13 at 19:08
  • Derp, nvm. I forgot about outbound traffic on my firewall. Thanks – MasterGberry Mar 26 '13 at 19:17
  • Important note : with rsync, be extra careful when using the "--delete" : read a lot about it, test on other (/tmp/...) folders, and beware the changes when adding or not a trailing "/" at the end of the source dir(s) or destination dir. – Olivier Dulac Mar 28 '13 at 10:26
14

It depends on how fast it needs to be copied, and how much bandwidth is available.

For a poor network connection consider the bandwidth of a truck filled with tapes. (Read: mail a 2.5 inch HDD, or just drive it there yourself. 300 gigabit drives should be easy to find).

If it is less time critical or you you plenty of bandwidth then rsync is great. If there is an error you can just continue without re-copying the earlier files.

[Edit] I forgot to add that you can run rsync several times if your data gets used during the copy.

Example:
1) Data in use. Rsync -> All data gets copied. This may take some time.
2) Run rsync again, only the changed files get copied. This should be fast.

You can do this several times until there are no changes, or you can do it the smart/safe way by making the data read-only during the copy. (e.g. if it is on a used shared set that share to read-only. Or rsync the data, then at night set the share read-only while you run it a second time).

|improve this answer
  • 1
    No server should be living somewhere where bandwidth can't handle 300G in a reasonable amount of time... – Dan Mar 27 '13 at 7:59
  • 1
    That depends on what is 'reasonable'. Say the weakest link is 100 mbit (I do not care if that is the upload limit from on office or the download at the other). That roughly allows for 10MB/sec. ( div by 10 seems reasonable, I know you can get slightly more if all goes perfectly well. E.g. nobody else is using the line for anything at all). 10MB/sec ~~ 600MB/min ~~ 36000MB/hour ~~ 36 GB/hour ~~ 300GB is 8h20min. That is doable overnight. That also makes a lot of assumptions. E.g. if the upload is only 2 mbit ( We have offices with those speeds) it takes 50 times as long (415h, or 17.3 days). – Hennes Mar 27 '13 at 11:56
  • Yikes! Yes, 8-10 is reasonable, but I was indeed making a number of assumptions. – Dan Mar 27 '13 at 15:44
  • 2
    @Dan If it is a requirement that the server is up and serving requests, saturating the upstream bandwidth is probably a bad idea. So you would have to artificially throttle the transfer speed to account for that. – Bob Mar 28 '13 at 10:10
6

I would go for rsync! I am using it to backup my server to an offsite server and it works fine. Usually there are a few MBs to copy but some days it goes up to 20-30GB and it allways worked without a problem.

|improve this answer
0

rsync over NFS using Gigabit connection will took nearly about 10h. It will be better to copy data on HDD and move them between server. If you need to make one-to-one copy of actually disk, use dd or something like that, to create raw image of disk. Using ssh (scp) cause a huge overhead. Empirically tested on Gigabit connection. rsync is good in making incremental synchronization between two servers used in HA or in backup mode. I guess.

|improve this answer
  • The language and style of this answer need to be improved. – FSMaxB Mar 27 '13 at 7:09
  • Rsync is especially great the if files can change during the copy. Just run it a few times. First time all data gets copied. Second time only what got changed during the first (long) copy. A third time would be done at night or with the shares read-only. – Hennes Mar 27 '13 at 12:00
  • will took nearly about 10h. It will be better to copy data on HDD and move them between server. except that it's across the country, so it'd take longer. – Rob Mar 27 '13 at 13:07
  • @FSMaxB: I will do this later, thx. – Pawel Mar 28 '13 at 9:45
  • @Rob: I have read this ;) That the servers are in two different location. So, you need to calculate, what will be better for You. Taking a journey accross the country (checking the cost of fuel, etc.) or using network connection. What will be more beneficial. – Pawel Mar 28 '13 at 9:48
0

first time use NFS and tar/untar (NFS is the fastest protocol in this case, tar to save network bandwidth by more CPU utilization)

tar cf - * | ( cd /target; tar xfp -)

next time/s use rsync

|improve this answer
  • If your have enough CPU power you can improve on this by adding gzip to the loop. And without NFS you can use netcat. (Or even both: tar -cf - * | gzip | nc -p 4567 and nc -l 4567 | gunzip | tar xf -. – Hennes Mar 30 '13 at 11:33
  • thanks Hennes, that was my idea, but I forgot gzip in the pipes – jet Mar 30 '13 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.