Downloading web pages via list of urls?

Question

I'm currently using sitesucker to pull down all the files for a client's web site. It gets me most of the pages, but the problem is that some of the pages we have aren't really accessed by link, but by a postal code finder. Is there a way I can use this tool or another tool even to give it a list of the URLs, and it downloads the files for me as need be, keeping the folder structure much like site sucker does.

My apologies if this is a poor place to ask this question :).

Answer 1

You can use wget: create a plain file with the list of the url you need to download (example pippo.txt):

pippo.txt:

http://url1.zip
http://url2.htm
http://url3/
ftp://url4/file.zip

Then use wget to download the url list:

cd /some/folder/
wget -i /path/to/pippo.txt

Done

wget for linux is a base tool https://www.gnu.org/software/wget/ for windows there is a bin port: http://users.ugent.be/~bpuype/wget/

Answer 2

If you have access to a Linux machine (perhaps virtually on Virtualbox), and depending on what you're trying to do, you might want to take a look at wget.

Using the -r option you can do things like recursive gets of a website.

You could do:

wget -r http://website/page?postcode=1234
wget -r http://website/page?postcode=5678

Easily scriptable/Excelable.