Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm currently using sitesucker to pull down all the files for a client's web site. It gets me most of the pages, but the problem is that some of the pages we have aren't really accessed by link, but by a postal code finder. Is there a way I can use this tool or another tool even to give it a list of the URLs, and it downloads the files for me as need be, keeping the folder structure much like site sucker does.

My apologies if this is a poor place to ask this question :).

share|improve this question

3 Answers 3

up vote 3 down vote accepted

You can use wget: create a plain file with the list of the url you need to download (example pippo.txt):

pippo.txt:

http://url1.zip
http://url2.htm
http://url3/
ftp://url4/file.zip

Then use wget to download the url list:

cd /some/folder/
wget -i /path/to/pippo.txt

Done

wget for linux is a base tool https://www.gnu.org/software/wget/ for windows there is a bin port: http://users.ugent.be/~bpuype/wget/

share|improve this answer

If you have access to a Linux machine (perhaps virtually on Virtualbox), and depending on what you're trying to do, you might want to take a look at wget.

Using the -r option you can do things like recursive gets of a website.

You could do:

wget -r http://website/page?postcode=1234
wget -r http://website/page?postcode=5678

Easily scriptable/Excelable.

share|improve this answer
2  
wget has been ported to Windows. gnuwin32.sourceforge.net/packages/wget.htm –  Patrick S. Mar 28 '12 at 13:16

I've written a browser extension to Chrome to do just this using Google Chrome's download API which'll work across all platforms (operating systems) i.e. Windows/Mac/Linux :-)

It's called TabSave and originally it was for this, I'm adding the ability to do some things researchers might want to do with giving PDFs titles straight from the browser but it's very light and can grab from open tabs if that's handy.

You can get the extension here, and all the code is open source on GitHub here.

Feel free to fork if you want to go a different direction with it :-)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.