11

I'm occasionally downloading a very large file via scp and there's a small chance each time of the connection dropping and cutting the transfer mid-way.

Is there a way to resume it?

9

You can try the following approch: instead of scp use dd to skip over the downloaded part and append the remainder to the file.

sofar=`ls -l ./destfile | awk '{print $5}'`
ssh rsys "dd if=./srcfile bs=1 skip=$sofar" >> ./destfile

Possible optimization: work with big blocks. Let's leave this as a homework.

  • I haven't tested this yet, but am accepting the answer tentatively since using dd sounds like a great approach for solution. – GJ. May 10 '12 at 12:19
  • I just tested it, works perfectly, thank you! – Kossak Aug 23 '15 at 18:38
  • @Kossak, nice to hear! – yrk Aug 23 '15 at 18:42
  • dd with small blocksizes can be slow (just 350 kB/s here). Fortunately, scp transfers seem to come in 1024 byte blocks. If so for your file (or else, by truncating it to the closest 1024 byte block), you can speed it up like this: ssh rsys "dd if=./srcfile bs=1024 skip=$sofar" >> ./destfile. (Note that $sofar then has to be the number of 1024 byte blocks to skip!) – tanius Jul 8 '16 at 19:56
  • 2
    It took me some time to realize that rsys in the second line is just an example hostname not some special argument for ssh. I suggest you replace it with user@host to avoid confusion. – Ahmed Essam Oct 31 '17 at 18:11
6

With scp, no.

If both ends have it, you can use rsync -LvzP remoteserver:path/to/file localfile to transfer a single file.

3

Yes, there are ways to resume from the point of interruption, but it is not possible using scp. sftp reget filename does what you need. Yarek and Grawity have provided valid solutions that I +1 to both, but for resuming from a point of interruption, I like rsync. The example commands provided both assume you are retrieving a file from a remote server to your local workstation (downloading). Please keep in mind that the final two parameters should be considered source_file and target_file in that order. The syntax of the filename varies based on whether the source or target file is local or remote. If I were sending (uploading) [text] files, I would rewrite the examples provided as:

#From local to remote
sofar=`ssh remote_system ls -l interrupted_file | awk '{print $5}'`;
dd if=source_file bs=1 skip=$sofar | ssh remote_system "cat >> ./interrupted_file"

And to the rsync solution, I add -e ssh. You should consider whether or not you need verbosity, compression, preserve ownership, permissions, timestamp, recurse directories, etc. Check man pages and google, regarding the -L parameter. You might want symlinks to remain as links instead of referencing them.

rsync -P -e ssh local_source_file remoteserver:path/to/interrupted_target_file

1

There's another solution (besides the rsync or dd solutions stated here) which I'm surprised nobody mentioned: it's the reput command of sftp.

0
scp -o ConnectTimeout 60

Lets you specify the timeout for a connection. May keep your connection from breaking up.

0

Yes, if both ends support sftp - after scp remoteuser@remotehost:/absolute/filename . fails you can resume by doing sftp remoteuser@remotehost and then reget /absolute/filename to resume the download.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.