21

I would like to save this webpage and all the pages it links to. and hope to have the same linking between the saved webpages.

Are there some ways instead of opening and saving each linked pages?

3 Answers 3

15

You can do what you'd like with the wget command line utility. If you provide it with the -r option, it will recursively download web pages. For example:

wget -r http://mat.gsia.cmu.edu/orclass/integer/integer.html

This will download that webpage and anything it links to. You can also make it only recurse a certain number of levels, to do this, you simply provide -r with a number. Like such:

wget -r 5 http://mat.gsia.cmu.edu/orclass/integer/integer.html
8
  • @Mark: Thanks! I now try to download mat.gsia.cmu.edu/orclass and the pages it links using command wget -r mat.gsia.cmu.edu/orclass. wget will create a directory mat.gsia.cmu.edu under the one I specified and download the pages under it. But the links between the downloaded pages does not have mat.gsia.cmu.edu in their paths, so it becomes a problem and I cannot go from one page to another by clicking the links. I was wondering why and how to solve the problem? Thanks!
    – Tim
    Apr 23, 2011 at 14:33
  • I don't think that you can recursively download external links, @Tim.
    – Wuffers
    Apr 23, 2011 at 16:11
  • Does "external links" mean those not under the current path?
    – Tim
    Apr 23, 2011 at 16:39
  • @Tim: By external links I mean links that refer outside of mat.gsi.cmu.edu
    – Wuffers
    Apr 23, 2011 at 16:40
  • 1
    @Tim: Oh, ok. Sorry for the misunderstanding. I think that you could try editing the HTML files yourself to check and try to make them work.
    – Wuffers
    Apr 23, 2011 at 17:12
14

This thread is old now, but others might look at it.

A modern version of wget has a number of useful options for recursing links and patching them to be local relative links so that you can navigate a local copy of a web site. Use the -r option to recurse, the -k option to patch local links, the -H option to traverse into domains other than the original one, the -D option to limit which domains you traverse into, the -l option to limit the depth of recursion, and the -p option to make sure that the leaves of your traversal have everything they need to display correctly.

For example, the following will download a page and everything it immediately links to, making it locally browsable, the -p option ensures that if the linked-to-pages contain images, that they are downloaded, too:

wget -r -l 1 -p -k -H -D example1.com,example2.com http://example.com/page/in/domain

-r recursive (follow links)

-l depth of recursion (0 is infinity)

-p get all images and etc. all needed to display HTML pages

-k convert links of js,css stuff to local links

-H go to foreign hosts when recursive

-D comma-separated list of accepted domains

Using a command similar to the one above, I was able to download a chunk of a wiki page, with external links, onto my local disk without downloading megabytes of extraneous data. Now, when I open the root page in my browser, I can navigate the tree without an Internet connection. The only irritant was that the root page was buried in subdirectories and I had to create a top-level redirect page in order to make it convenient to display. It may take some trial-and-error to get it right. Read the wget man page and experiment.

6

You can use a website crawler like httrack, which is free.

From the website;

[httrack] allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online.

3
  • 1
    +1 Excellent application! But it's grabbing all the linked zip files as well, which I didn't want. But then I should have probably read the instructions first!
    – finlaybob
    Mar 28, 2014 at 16:29
  • Yup, it can/will follow all links so will download files. (@Finlaybob are you aware the homepage listed on your profile has been hacked?)
    – RJFalconer
    Mar 30, 2014 at 20:04
  • I was not! I'll look into it - thanks for letting me know!
    – finlaybob
    Mar 31, 2014 at 16:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .