Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to use wget to create a local mirror of a website. But I am finding that I am not getting all the linking pages.

Here is the website


I don't want all pages that begin with web.archive.org, but I do want all pages that begin with http://web.archive.org/web/20110722080716/http://cst-www.nrl.navy.mil/lattice/.

When I use wget -r, in my file structure I find


but I don't have all files that are part of this database, e.g.


Perhaps httrack would do better, but right now that's grabbing too much.

So, by which means is it possible to grab a local copy of an archived website from the Internet Archive Wayback Machine?

share|improve this question
Man! I tried to mirror exactly the same page (and really get angry that I didn't it when the original site was still online, which would have been much more easy). I think a problem is, that not all files are accessible under the 20110722080716 snapshot, hence wget's -np option won't help. –  mpy Feb 1 '14 at 11:16
Have you checked manually that the missing pages are actually archived? Archive.org doesn't always archive every single page. –  nitro2k01 Feb 3 '14 at 9:44

5 Answers 5

While helpful, prior responses fail to concisely, reliably, and repeatably solve the underlying question. In this post, we briefly detail the difficulties with each and then offer a modest httrack-based solution.


Before we get to that, however, consider perusing mpy's well-written response. In this sadly neglected post, mpy rigorously documents the Wayback Machine's obscure (and honestly obfuscatory) archival scheme.

Unsurprisingly, it ain't pretty. The Wayback Machine does not archive sites to a single directory. Instead, it ephemerally spreads a single site across two or more numerically identified sibling directories, substantially complicating mirroring.

Understanding the subtle pitfalls presented by this scheme is core to understanding the inadequacy of prior solutions. Let's get on with it, shall we?

Prior Solution 1: wget

The related StackOverflow question "Recover old website off waybackmachine" is probably the worst offender in this regard, recommending wget for Wayback mirroring. Sadly, such recommendation is fundamentally unsound.

In the absence of complex external URL rewriting (e.g., via Privoxy), wget cannot be used to reliably mirror Wayback-archived sites. As mpy details under "Problem 2 + Solution," the mirroring tool(s) you adopt must non-transitively mirror only URLs corresponding to the target site (rather than transitively mirroring URLs corresponding to both the target site and sites linked to from such site).

A concrete example is probably in order. When mirroring domain kearescue.com, such tool(s) must:

  • Include all URLs matching https://web.archive.org/web/*/http://kearescue.com, signifying assets provided by the target site (e.g., https://web.archive.org/web/20140521010450js_/http_/kearescue.com/media/system/js/core.js).
  • Exclude all other URLs, signifying assets provided by other sites merely linked to from the target site (e.g., https://web.archive.org/web/20140517180436js_/https_/connect.facebook.net/en_US/all.js).

Failure to exclude such URLs typically pulls in approximately half of the Internet archived at the time such site was archived, particularly for sites embedding externally-hosted assets (e.g., YouTube videos).

That would be bad. While wget does provide command-line option --exclude-directories accepting one or more patterns matching URLs to be excluded, such patterns support only a glob-style * matching zero or more characters excluding /. Since the URLs to be excluded contain arbitrarily many / characters, wget cannot be used to exclude such URLs and hence cannot be used to mirror Wayback-archived sites.

Curiously, this issue has been on public record since at least 2009. It has yet to be addressed. Next!

Prior Solution 2: Scrapbook

Prinz recommends ScrapBook, a Firefox plugin. A Firefox plugin.

That was probably all you needed to know. While ScrapBook's Filter by String... functionality does address the aforementioned "Problem 2 + Solution," it does not address the subsequent "Problem 3 + Solution" -- namely, that of extraneous duplicates.

That said, it's questionable whether ScrapBook even adequately addresses the former problem. As mpy admits:

Although Scrapbook failed so far to grab the site completely...

Unreliable and overly simplistic solutions are non-solutions. Next!

Prior Solution 3: wget + Privoxy

mpy then provides a robust solution leveraging both wget and Privoxy. While wget is reasonably straightforward to configure, Privoxy is anything but.

Thanks to the technical hurdle of properly installing, configuring, and using Privoxy, we have yet to confirm mpy's solution as such. It should conceivably work in a scalable, robust manner. Given the practical barriers, however, this solution is probably more appropriate to large-scale automation than the average webmaster attempting to recover small- to medium-scale sites.

Is wget + Privoxy worth a look? Absolutely. But most superusers would probably be better serviced by simpler, more readily applicable solutions.

== New Solution: httrack ==

Enter httrack, a command-line utility implementing a superset of wget's mirroring functionality. (That's good.)

Specifically, httrack provides both arbitrary pattern matching-based URL exclusion and simplistic site restructering. The former solves mpy's "Problem 2 + Solution"; the latter, "Problem 3 + Solution."

In the abstract example below, replace:

  • ${wayback_url} by the URL of the top-level directory archiving the entirety of the target site (e.g., https://web.archive.org/web/20140517175612/http://kearescue.com).
  • ${domain_name} by the same domain name present in ${wayback_url}, excluding prefixing http:// (e.g., kearescue.com).

Here we go. Install httrack and run the following at the CLI (in the local directory to store such site):

command httrack\
    --user-agent='Mozilla/5.0 (X11;U; Linux i686; en-GB; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5'\

On completion, the current directory should contain one subdirectory for each filetype accessible from such top-level URL. Typically, this includes:

  • css, containing all mirrored CSS stylesheets.
  • html, containing all mirrored HTML pages.
  • js, containing all mirrored JavaScript.
  • ico, containing the mirrored favicon.

And so forth. Since httrack internally rewrites all mirrored HTML pages to reflect such structure, such site should now be browsable as is without subsequent modification. If you prematurely halt and would like to restart such mirroring, simply append option --continue to the above command and retry.

That's it. No external contortions, error-prone URL rewriting, or rule-based proxy servers required.

Enjoy, fellow superusers.

share|improve this answer
I'm glad to hear that at least one person read my answer thoroughly. And thanks for your further analysis and the httrack solution. +1 –  mpy Jun 21 '14 at 21:28
The httrack solution was perfect, thank you so much! –  ChrisChinchilla Mar 25 at 14:03


This option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets.

Ordinarily, when downloading a single HTML page, any requisite documents that may be needed to display it properly are not downloaded. Using -r together with -l can help, but since Wget does not ordinarily distinguish between external and inlined documents, one is generally left with "leaf documents" that are missing their requisites.

For instance, say document 1.html contains an "" tag referencing 1.gif and an "" tag pointing to external document 2.html. Say that 2.html is similar but that its image is 2.gif and it links to 3.html. Say this continues up to some arbitrarily high number.


Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing.

Note that Wget will behave as if -r had been specified, but only that single page and its requisites will be downloaded. Links from that page to external documents will not be followed. Actually, to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to -p:

wget -E -H -k -K -p http://<site>/<document>

SO wget -E -H -k -K -p http://web.archive.org/web/20110722080716/http://cst-www.nrl.navy.mil/lattice will be your best suit for you. But I recommend another tool, a firefox extension scrapbook


ScrapBook is a Firefox extension, which helps you to save Web pages and easily manage collections. Key features are lightness, speed, accuracy and multi-language support. Major features are:
* Save Web page
* Save snippet of Web page
* Save Web site
* Organize the collection in the same way as Bookmarks
* Full text search and quick filtering search of the collection
* Editing of the collected Web page
* Text/HTML edit feature resembling Opera's Notes

How to mirror a site
Install scrapbook and restart firefox

  1. Load page in browser [web page to be mirrored]
  2. Right click on the page -> Save page as ...
  3. select level from In depth Save and press save enter image description here
  4. select Restrict to Drirectory/Domain from Filter
    enter image description here

Wait for it to mirroring to complete. After mirroring you can access the web site offline from ScrapBook menu.

share|improve this answer
Although Scrapbook failed so far to grab the site completely, it was closer to a possible solution than the other suggestions. Especially its Filter by String... option was more helpful than to filter by host/domain. Hence, I award the bounty to you :) –  mpy Feb 8 '14 at 11:16

Unfortunately none of the answers were able to solve the problem of making a complete mirror from an archived website (without duplicating every file a dozens of times). So I hacked together another approach. Hacked is the important word as my solution is neither a general solution nor a very simple (read: copy&paste) one. I used the Privoxy Proxy Server to rewrite the files on-the-fly while mirroring with wget.

But first, what is so difficult about mirroring from the Wayback Machine?

Problem 1 + Solution

The Wayback toolbar is handy for interactive use, but might interfere with wget. So get rid of it with a privoxy filter rule

FILTER: removewaybacktoolbar remove Wayback toolbar

Problem 2 + Solution

I wanted to capture the whole site, so needed a not-too-small recursion depth. But I don't want wget to crawl the whole server. Usually you use the no-parent option -np of wget for that purpose. But that will not work here, because you want to get


but also


(notice the changed timestamp in the paths). Omitting -np will end up wget crawling up to (...)http://cst-www.nrl.navy.mil, and finally retrieve the whole navi.mil site. I definitely don't want that! So this filter tries to emulate the -np behavior with the Wayback machine:

FILTER: blocknonparentpages emulate wget -np option
s|/web/(.*)/http(.*)([" ])|http://some.local.server/404$3|gU

I'll leave it as an exercise to dig into the syntax. What this filter does is the following: It replaces all Wayback URLs like http://web.archive.org/web/20110801041529/http://www.nrl.navy.mil/ with http://some.local.server/404 as long as they do not contain http://cst-www.nrl.navy.mil/lattice/.

You have to adjust http://some.local.server/404. This is to send an 404 error to wget. Probably privoxy can do that more elegant. However, the easiest way for me was just to rewrite the link to a non-existent page on a local http server, so I stuck with this.

And, you also need to adjust both occurences of http://cst-www.nrl.navy.mil/lattice/ to reflect the site you want to mirror.

Problem 3 + Solution

And finally some archived version of a page might link to page in another snapshot. And that to yet another one. And so on... and you'll end up with a lot of snapshots of the same page -- and wget will never manage to finish until it has fetched all snapshots. I really don't want that, neither! Here it helps a lot, that the Wayback machine is very smart. You can request a file


even if it's not included in the 20110801041529 snapshot. It automatically redirect you to the correct one:


So, another privoxy filter to rewrite all snapshots to the most recent one

FILTER: rewritewaybackstamp rewrite Wayback snapshot date

Effectively every 14-digit-number enclosed in /.../ gets replaced with 20120713212803 (adjust that to the most recent snapshot of your desired site). This might be an issue if there are such numbers in the site structure not originating from the Wayback machine. Not perfect, but fine for the Strukturtypen site.

The nice thing about that is, that wget ignores the new location it is redirected to and saves the file -- in the above exampe -- as web.archive.org/web/20110801041529/http://cst-www.nrl.navy.mil/lattice/struk/a_f.html.

Using wget to mirror archived site

So, finally with these privoxy filters (defined in user.filter) enabled in user.action via

{ +filter{removewaybacktoolbar} +filter{blocknonparentpages} +filter{rewritewaybackstamp} }

you can use wget as usual. Don't forget to tell wget to use the proxy:

export http_proxy="localhost:8118"
wget -r -p -k -e robots=off http://web.archive.org/web/20120713212803/http://cst-www.nrl.navy.mil/lattice/index.html

I used these options, but -m should work, too. You'll end up with the folders


as the Wayback machine separates images (im_), style sheets (cs_) etc. I merged everything together and used some sed magic to replace the ugly relative links (../../../../20120713212803js_/http:/cst-www.nrl.navy.mil/lattice) accordingly. But this isn't really necessary.

share|improve this answer

Be careful with the below command because it grabs a lot. The 1 after the 'l' tells it to grab all pages for links on the site that are 1 level deep. If you want it to spider deeper change this to a 2 but it might never end because it could get caught in a loop.

wget -rHpkl 1 -e robots=off http://www.example.com/

I'm not sure which parts of the site you want to keep and which parts you don't care about but you should probably white list and/or blacklist the different parts of the site to get only what you want and to prevent yourself from downloading all of archive.org or the internet.

Use -D www.examle.com,www.another.example.com to whitelist only the domains you want or use --exclude-domains www.examle.com,www.another.example.com to blacklist what you don't want.

share|improve this answer
Thanks, but the problem with white/blacklisting is that all archived websites come from the web.archive.org host. I want to mirror everything what wget -np would have mirrored once the original site was still online. -l doesn't help much either, since it has to be increased to 3 or 4, hence resulting in ascending the website hierarchy too much. –  mpy Feb 3 '14 at 9:47

The format of the URLs for the Internet Archive include the date and time the site was archived. To save space assets that haven't changed are linked back to a previous version of a site.

For example in this url http://web.archive.org/web/20000229123340/http://www.yahoo.com/ the date the site was crawled was Feb 29, 2000 at 12:33 and 40 seconds.

So to get all of http://web.archive.org/web/20110722080716/http://cst-www.nrl.navy.mil/lattice/ you need to start at that but also grab all linked assets from http://web.archive.org/web/*/http://cst-www.nrl.navy.mil/lattice/.

share|improve this answer
Exactly, and that is the problem. Let's say page A links to B. So, the current version A links to old version B. But B includes also a link to A. So the old version of A gets retrieved, too and links again to older version. This (at a (needed) crawl depths of 4) leads to the result, that you end up with dozens of versions of the index page, but not all needed files. –  mpy Feb 8 '14 at 10:15

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.