Website downloader (cache?) to view sites offline

Question

Is there a portable way to download or cache all pages of a website for viewing offline? I have a cross country flight tomorrow and I'd like to be able cache a few webpages (particularly the python doc page (http://docs.python.org/) and the pyqt reference (http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/pyqt4ref.html).

Ideally I'd like a firefox addon or something like that, but anything will work fine as long as I can run it on linux.

Answer 1

I use HTTrack.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer

Answer 2

I use wget with these options to mirror a site for offline use

wget -cmkE -np http://example.com/a/section/i/like

where

-m turns on mirroring options for mirroring a site locally

-c continues a previous download in case I have already downloaded some pages

-k converts absolute href to point to local ones for offline viewing

-E ensures files have .html extension after download.

-np only downloads objects under /a/section/i/ and does not cache the whole site.

For example I wanted to download south documentation but not south tickets, etc...

wget -cmkE -np http://south.aeracode.org/docs/index.html

I use Windows and run wget on cygwin but there is also a native windows wget port.

Although, in your case, you can download python offline docs from python docs section

Answer 3

Try http://www.downthemall.net/ a Firefox plugin. I've used it to download 250 pages of PDFs in 20+ separate files. It is extremely powerful. It has a wildcard/query syntax that lets you surgically get only the files that you want and none of the irrelevant ones you don'.

Answer 4

Some Firefox extensions that I know of:

• ScrapBook - https://addons.mozilla.org/en-US/firefox/addon/427/

• SpiderZilla ( This doesn't work for Linux but might be of interest to Windows users looking for a similar extension/software. Also, it has to be edited to work with the latest version of Firefox. This is an extension based on HTTrack. ) - https://addons.mozilla.org/en-US/firefox/addon/1616/

• ScrapBook Plus - https://addons.mozilla.org/en-US/firefox/addon/8186/

• Read It Later - https://addons.mozilla.org/en-US/firefox/addon/7661/

Answer 5

You can download a whole website or part of a website with wget.

wget -r http://docs.python.org/

Check the wget manual for other options you may want to pass, e.g. to limit your bandwidth usage, to control recursion depth, to set up exclusion lists, etc.

Another approach to offline browsing is to use a caching proxy. Wwwoffle is one which has a lot of features to facilitate retention for offline browsing, such as overrides to server-specified expiration dates and a recursive pre-fetching capability. (I've been using wwwoffle since my dial-up days.)