How can I get
wget (possibly with help from a bash script) to try a series of mirrors if a site is down (or times out part way through downloading)? For example, first try download
https://www.example.com/file.gz, then if that fails try
How can I get
#!/bin/sh for site in www.example.com mirror1.example.com another.mirror.com do wget --timeout=60 --continue $site/file.gz && break done
--continue switch allows you to pickup downloading where the last site left off.
Note that you can separately set
--read-timeout if wanted.
Ken Jackson's answer is spot on about how you would do this. However, I'm going to assume here that you already have all the links in a single separate file. In this case, you don't really need any bash scripts to do the job. You can simply run:
$ wget --timeout=60 --tries=1 --continue --input-file <inputfile>
For a nice looking output, I also use
-q --show-progress which will force Wget to display only the progress bars and nothing else.
What the above command does is it tries to download every link from the file you gave it. However, after the entire file is downloaded, it will simply query each server, realise the file already exists and then move on. It wastes a very tiny amount of extra bandwidth, but is far easier to type in and doesn't need to multiple invokations of the binary.