1

I am trying to transfer large data from windows to linux by using SCP.

scp GEOS* username@ip:/path/to/destination

It works. But it is pretty slow and the connection would be reset two hours later.

files.tar                                                                                                                                                     50% 9109MB   1.1MB/s 2:12:35 ETAConnection reset by IP port 22
files.tar                                                                                                                                                     50% 9109MB   1.0MB/s 2:27:19 ETAlost connection

I wonder if there is any another more stable way to achieve it.

And also, I heard that rsync can continue interrupted transfers. I tired

rsync -ht --progress --stats  GEOS* /path/to/destination

And I tired it again after rebooting my pc. It seems like that I transfer files from zero again.

I wonder how I can continue the interrupted transfer.

Thanks, Lixu

  • 1
    You have to use --append or --append-verify with rsync to make it append to the current version of the file. – xenoid Oct 12 at 22:49
  • If you have access to the Windows machine, have you tried zipping the files.tar file before you try grabbing it from Linux? – Mark Stewart Oct 12 at 23:52
0

It's tricky to work out what you're actually doing because neither of your quoted example commands is valid - as written each would either fail or else could not do what you have said it does.

I'm going to guess you meant these:

scp GEOS* username@ip:/path/to/destination
rsync -ht --progress --stats GEOS* username@ip:/path/to/destination

If that's not right, please edit your question to show what commands you actually tried and I'll see if I can update my answer accordingly.


The rsync command works in one of two different ways depending on how it's called.

  1. Rsync to two paths that look like they're local.

    rsync /path/to/source/GEOS* /path/to/destination/
    

    This version of the rsync command will not be usefully restartable, for two reasons.

    • There is no metadata transferred with the file so rsync cannot tell if the file is a likely candidate for a shortcut (use rsync -a instead of just rsync to fix this).
    • You are using a source and destination path that "looks like" it's local. On this basis rsync cannot run as client/server and will not attempt to skip parts of the files that have already been transferred.
  2. Rsync between a local and remote host.

    rsync -ht --progress --stats GEOS* username@ip:/path/to/destination
    

    This version of the rsync command runs in client/server mode, and because it includes the file timestamps (-t), will restart in the event of a network failure.

    Notice the difference between this command (local/remote) and the second reason for #1 failing to be usefully restartable (local/local).

I would not recommend the use of --append or --append-verify unless you can absolutely and totally guarantee that the source file has not changed in any way since the beginning of the first rsync attempt. Even then I wouldn't generally recommend it simply because it doesn't really save very much time and it does greatly increase the opportunity for a mis-copy when it's misused. If there is any doubt whatsoever about the file having been changed, using --append or one of its variants will guarantee an undetectably wrongly copied file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.