Can't download a large amount of data; can I download it by batch?

Question

I would like to download my backup image of a Google Compute Engine instance, but it is too large to download (900+ MB). It is a tar.gz file.

However, the problem is I'm now in China, and GCP is blocked. So I must use VPN.

However, the VPN is so slow and unstable, and thus I cannot download such a large amount of data. Whenever I tried to download it, the download never completed before it is stopped and eventually failed at the midst of the large volume of download. I use Google Chrome to download it.

Since the download worked until the point of a failure (say, 428/932 MB), it may be possible to download it if you can resume it again or separate the amount of downloaded volume.

Can I download it by separating the amount of downloaded contents to avoid the failure?

Answer 1

First, you could compress the file by 7Zip, whose compression is much better than gzip, to reduce the download size.

Second, you could use a download manager that resumes broken downloads, so you can continue your download when it dies from where it stopped.

Not knowing the operating system you use to download, here are some download managers that are available on all platforms :

• Xtreme Download Manager
• JDownloader

More very good download managers are available if you are using Windows.

Answer 2

Programs like curl or wget have options --continue (wget) and --continue-at (curl), which allows you to resume interrupted downloads, providing the data has not been corrupted.

wget, by default will automatically try to resume the download when it occurs an interruption, so you don't have to worry about unstable transfer.

curl's --continue-at option generally requires a number value, but if you pass it - instead of a number, it will use its output file to determine from which point it should start downloading.

Sample usage (from man wget) wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z, curl ftp://sunsite.doc.ic.ac.uk/ls-lR.Z -o myfile.Z --continue-at -

If you need to use some specific cookies, headers, sending POST data or whatever to receive the file, modern browsers (through the developer tools panel, i.e. F12) allow you to copy the curl request command which can be used to access that resource. (see images)

Firefox:

Chromium:

Just copy that as curl and then paste into console appending appropriate flags. If the command fails, you can just call curl once again (the cookies may be invalidated, so you may need to ask the browser for new curl command) and continue downloading.

This may seem like complicated process to get the command, but it's very versatile - one can download nearly all kind of materials which you view in your browser by curl command provided by browser, possibly guessing the next links¹ and pasting output together.

---

¹ A book may be uploaded so that you don't load whole PDF by one request but page by page, each under different address. Therefore, you need to make requests to www.example.com/page1.pdf, www.example.com/page2.pdf and www.example.com/page3.pdf in order to get three pages.
Some video streams don't send video for one request but require the client to ask about each video segment separately.
In both cases, the mechanism is simple: guess the next link generation pattern, ask for all links and then "concatenate" (accordingly to the file type) the output.