Downloading web pages via list of urls?

Question

I'm currently using sitesucker to pull down all the files for a client's web site. It gets me most of the pages, but the problem is that some of the pages we have aren't really accessed by link, but by a postal code finder. Is there a way I can use this tool or another tool even to give it a list of the URLs, and it downloads the files for me as need be, keeping the folder structure much like site sucker does.

My apologies if this is a poor place to ask this question :).

Answer 1

You can use wget: create a plain file with the list of the url you need to download (example pippo.txt):

pippo.txt:

http://url1.zip
http://url2.htm
http://url3/
ftp://url4/file.zip

Then use wget to download the url list:

cd /some/folder/
wget -i /path/to/pippo.txt

Done

wget for linux is a base tool https://www.gnu.org/software/wget/ for windows there is a bin port: http://users.ugent.be/~bpuype/wget/

Answer 2

If you have access to a Linux machine (perhaps virtually on Virtualbox), and depending on what you're trying to do, you might want to take a look at wget.

Using the -r option you can do things like recursive gets of a website.

You could do:

wget -r http://website/page?postcode=1234
wget -r http://website/page?postcode=5678

Easily scriptable/Excelable.

Answer 3

I've written a browser extension to Chrome to do just this using Google Chrome's download API which'll work across all platforms (operating systems) i.e. Windows/Mac/Linux :-)

It's called TabSave and originally it was for this, I'm adding the ability to do some things researchers might want to do with giving PDFs titles straight from the browser but it's very light and can grab from open tabs if that's handy.

You can get the extension here, and all the code is open source on GitHub here.

Feel free to fork if you want to go a different direction with it :-)

Answer 4

You can do this in windows with the following powershell script:

foreach($url in Get-Content .\docs.txt) {
   $file = "c:\temp\out\" + ($url).split('/')[-1]
   Invoke-WebRequest -Uri $url -Outfile $file
}

docs.txt is a file with a url per line. This script uses the end section of the path to create a destination filename, so you might need to adjust this if required.

Answer 5

In A1 Website Download for Windows and Mac you can solve this in two ways:

a) Configure it to dive through HTML forms (e.g. dropdowns). Only caveat is that in large forms the combinations of options can give millions of unique URLs.

b) Seed the crawl by importing a list of URLs before starting the download.