0

How can I get wget (possibly with help from a bash script) to try a series of mirrors if a site is down (or times out part way through downloading)? For example, first try download https://www.example.com/file.gz, then if that fails try https://mirror1.example.com/file.gz, then https://another.example.com/file.gz?

2

Try this:

#!/bin/sh
for site in www.example.com mirror1.example.com another.mirror.com
do
    wget --timeout=60 --continue $site/file.gz  &&  break
done

The --continue switch allows you to pickup downloading where the last site left off.

Note that you can separately set --dns-timeout, --connect-timeout and --read-timeout if wanted.

0

Ken Jackson's answer is spot on about how you would do this. However, I'm going to assume here that you already have all the links in a single separate file. In this case, you don't really need any bash scripts to do the job. You can simply run:

$ wget --timeout=60 --tries=1 --continue --input-file <inputfile>

For a nice looking output, I also use -q --show-progress which will force Wget to display only the progress bars and nothing else.

What the above command does is it tries to download every link from the file you gave it. However, after the entire file is downloaded, it will simply query each server, realise the file already exists and then move on. It wastes a very tiny amount of extra bandwidth, but is far easier to type in and doesn't need to multiple invokations of the binary.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.