3

My goals:

  1. I want to download all *.zip files on a page where all relevant links are redirects done by a server script ( */download.php?standard=yes&file=*.zip ).
  2. I want the original link to be used to name the files, since the name given by the server is in this case always "download.zip".

Specifically, I want to download the OsmAnd maps from here: http://download.osmand.net/rawindexes/

How do I do it?

What I tried:

I could use wget to do something similar with the maps for MAPS.ME:

wget -rl1 --accept="mwm" http://direct.mapswithme.com/direct/latest/

But in this case the links were direct, and there was no problem with the file names.

I also tried to use curl -L, but that's without recursion, and it doesn't seem to work for the kind of redirects that are used on that page.

0

Wget works fine but you need to quote the URL since it's not escaped.

i.e.

wget 'http://download.osmand.net/download.php?standard=yes&file=Afghanistan_asia_2.obf.zip' -O Afghanistan_asia_2.obf.zip

The -O option controls the output name. It should be fairly easy to create a shell script that will do that given a list of filenames.

Edit: You can get file names by downloading the page and doing a regex search for the file pattern:

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_]*.obf.zip" | cut -c6-

Which could be combined with xargs and piped directly into wget to download each file:

xargs -I{} wget 'http://download.osmand.net/download.php?standard=yes&file={}' -O {}

Or your full, one line command (works in Cygwin, other *nix may require some playing with quotes and xargs):

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_]*.obf.zip" | cut -c6- | xargs -I{} wget 'http://download.osmand.net/download.php?standard=yes&file={}' -O {}
1

Some filenames have "-" in them. So the script to get the list of files should be changed with one character. This works for me:

wget -nv 'http://download.osmand.net/rawindexes/' -O - | grep -oE "file=[A-Za-z0-9_-]*.obf.zip" | cut -c6-

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.