31

I'm occasionally downloading a very large file via scp and there's a small chance each time of the connection dropping and cutting the transfer mid-way.

Is there a way to resume it?

1
  • 1
    Check ulimit -a | grep 'file size' when it's failing with File too large.
    – Betlista
    Feb 15, 2022 at 13:05

8 Answers 8

25

You can try the following approch: instead of scp use dd to skip over the downloaded part and append the remainder to the file.

sofar=`ls -l ./destfile | awk '{print $5}'`
ssh user@host "dd if=./srcfile bs=1 skip=$sofar" >> ./destfile

Possible optimization: work with big blocks. Let's leave this as a homework.

6
  • I haven't tested this yet, but am accepting the answer tentatively since using dd sounds like a great approach for solution.
    – GJ.
    May 10, 2012 at 12:19
  • I just tested it, works perfectly, thank you!
    – Kossak
    Aug 23, 2015 at 18:38
  • @Kossak, nice to hear!
    – yrk
    Aug 23, 2015 at 18:42
  • 1
    dd with small blocksizes can be slow (just 350 kB/s here). Fortunately, scp transfers seem to come in 1024 byte blocks. If so for your file (or else, by truncating it to the closest 1024 byte block), you can speed it up like this: ssh rsys "dd if=./srcfile bs=1024 skip=$sofar" >> ./destfile. (Note that $sofar then has to be the number of 1024 byte blocks to skip!)
    – tanius
    Jul 8, 2016 at 19:56
  • 7
    It took me some time to realize that rsys in the second line is just an example hostname not some special argument for ssh. I suggest you replace it with user@host to avoid confusion.
    – A.Essam
    Oct 31, 2017 at 18:11
39

With scp, no.

If both ends have it, you can use rsync -LvzP remoteserver:path/to/file localfile to transfer a single file.

3
  • This saved me hours of downloading a huge file which got interrupted. Thank you.
    – phette23
    Mar 26, 2020 at 18:06
  • I am getting following error while using this command "X11 forwarding request failed on channel 0" Mar 24, 2021 at 8:12
  • 10 years later and you just saved me 6 hours
    – Andrew
    Sep 4, 2022 at 23:46
8

Yes, there are ways to resume from the point of interruption, but it is not possible using scp. sftp reget filename does what you need. Yarek and Grawity have provided valid solutions that I +1 to both, but for resuming from a point of interruption, I like rsync. The example commands provided both assume you are retrieving a file from a remote server to your local workstation (downloading). Please keep in mind that the final two parameters should be considered source_file and target_file in that order. The syntax of the filename varies based on whether the source or target file is local or remote. If I were sending (uploading) [text] files, I would rewrite the examples provided as:

#From local to remote
sofar=`ssh remote_system ls -l interrupted_file | awk '{print $5}'`;
dd if=source_file bs=1 skip=$sofar | ssh remote_system "cat >> ./interrupted_file"

And to the rsync solution, I add -e ssh. You should consider whether or not you need verbosity, compression, preserve ownership, permissions, timestamp, recurse directories, etc. Check man pages and google, regarding the -L parameter. You might want symlinks to remain as links instead of referencing them.

rsync -P -e ssh local_source_file remoteserver:path/to/interrupted_target_file

7

There's another solution (besides the rsync or dd solutions stated here) which I'm surprised nobody mentioned: it's the reput command of sftp.

5

Yes, if both ends support sftp - after scp remoteuser@remotehost:/absolute/filename . fails you can resume by doing sftp remoteuser@remotehost and then reget /absolute/filename to resume the download.

2
scp -o ConnectTimeout 60

Lets you specify the timeout for a connection. May keep your connection from breaking up.

1

The accepted answer of yrk does work if you are copying from a remote to a local machine. If in your case you are trying to run scp -3 between two remotes that cannot communicate directly to eachother you can use the following solution. Assume 'source_file' is located on 'host_source' and 'dest_file' is located on 'host_dest'.

  1. ssh into host_dest and check the current file size

    ls -l dest_file | awk '{print $5}'

    Write down the output, e.g. 7544684

  2. From your local machine:

    ssh user@host_source "dd if=source_file bs=1 skip=7544684" | ssh user@host_dest -T "cat >> dest_file"

1
0

I think it should be:

scp -o ConnectTimeout=60

instead of:

scp -o ConnectTimeout 60

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .