1

I am in the process of creating an offsite backup, and need to do a initial load of data. Currently, that's about 400Gb, give or take 10Gb or so... The backup system is producing files which are about 4Gb each, and has some other, smaller related files also. So, i need to transfer all 400ish gigs to a remote server, but how? What is the best method?

I have full remote access to the server, so i can install anything i need to install. There are Windows, Linux and a Solaris VM running on the box itself, so any of those can be used there, and i have Windows and Linux at home. I have 2 internet connections in house, 10Mb/s uploading on each, so something that could potentially split the number of connections would be handy (kind of like GetRight, but in reverse... PutRight?).

[Update] I have tried the option below of using RSync, but there is only one connection being made between my workstation and server. When uploading large files to Amazon, there is an option of breaking them apart locally and unbreaking them on the server end. multiple files are also send over multiple connections. so, a 4Gb file could be split into 800 ish 5Mb chunks, and sent 4 or 8 at a time, using as much of your upload bandwidth as possible. So, how would i do this automatically? I could hack something together to do it and use SFTP to send the files, but is there an automagic option?

| |
3

rsync will work the best, most likely.

rsync -avzP /source user@host:/destination

The above syntax can be used to copy the files over ssh and will allow you to resume the transfer where it left off in the event of a disconnection. The -v and -P are both optional, but highly recommended in your case.

-v tells rsync to print the current file to the screen.

-z tells rsync to compress the data. This can greatly increase the transfer speed over WAN connections.

-P tells rsync to do two things: First, provide a percentage progress to the screen, the same as --progress. Second, it will leave partially transferred files on the server, so if there's a disconnect, it leaves that partial file on the server so you can resume it. This is the same as --partial

As it is intended to be an offsite backup, there are plenty of options that you can throw into rsync to make it very robust. You can very easily turn it into an incremental backup as well if you'd like.

During a second transfer, if any of the data has changed, rsync will just copy the newest changes while leaving the rest alone.

| |
  • thanks for the info. I will be using this for only one run, so no incremental backups are needed... The backup software is allowing me to backup once to a folder, take the contents of the folder and bring it somewhere else (usually done on USB key or HDD) and then continue backups there... Thanks again. will look into it! – TiernanO Oct 5 '12 at 14:55
  • Some additional arguments you will be interested in (not necessary, but they can make things easier for you): --delete --inplace --link-dest (fairly advanced) – UtahJarhead Oct 5 '12 at 14:56
  • 1
    Since it's a one-off, use the syntax exactly as it is in the answer and you'll be golden, assuming both ends have ssh available to them! – UtahJarhead Oct 5 '12 at 14:58
  • yea, both servers are controlled by me, and SSH is available remotely. – TiernanO Oct 5 '12 at 15:01
  • Carry on, sir! Good luck! – UtahJarhead Oct 5 '12 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.