Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to mirror a particular website (all the pages under that particular domain) any pages (but not whole sites) that the website links to.

I'm confused about the how to do this

wget -r --level=inf (or some other variant) will mirror the site.

wget -r -H --level=1 will get all the links (from all domains) to the first level.

Anyone have any ideas on how I could combine these, to get the entire of the main site and one level deep into external sites. I've been banging my head against the manual all afternoon.


share|improve this question

migrated from Mar 15 '11 at 10:24

This question came from our site for professional and enthusiast programmers.

This is unfortunately impossible with wget (and the attempt at solving this with -H -l 1 does not do what you expect). What you want is HTTrack.

httrack --ext-depth=1

This can also be abbreviated as httrack %e1 Note that HTTrack counts levels starting at 1, not 0, so it won't follow links found on external pages unless you increase the depth.

share|improve this answer

I would use a combination wget -m -k -K -p && wget -r -k -K -H -N -l 1

About the two commands: wget -m -k -K -p will mirror (-m = -r --level=inf -N) it, convert the links to your local mirror (-k), backs up the original file before it gets converted (-K) and downloads all prerequisites for proper viewing the mirror (-p).

After that the second command wget -r -k -K -H -N -l 1 would do essentially the same but only for one level spanning all hosts and it would check the timestamps with -N, so you wouldn't download the same files again. I didn't include the -p option here, because it could download very much then...

share|improve this answer

You must log in to answer this question.