Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to mirror the following web site completely:

I use the following wget command:

wget -m

The web site is a custom CMS and contains lots of pages having the following form of urls:

Oddly, wget gets a few of these pages but not all of them. I wonder what might be the reason for this?

Note: There is no constraining due to robots.txt.


Looking at the source code of the web site, I noticed that the pages that are not detected and crawled by wget have a common property. Their anchor urls are written by the following javascript function:

function yazilar()
var ab = '</a><br class=\"hide\" />';
var aa = '<a class=' + '\"nav sub\" href=\"kategori.php?id=';
var ac = '';

var arr = new Array();
arr[0] = '12\">'+ac+' Belâgat';
arr[1] = '15\">'+ac+' Bilim ve Teknoloji';
maxi = 14;
for(i=0;i<maxi;i++) {
    a = aa + arr[i] + ab;

So, it looks like wget cannot detect anchor tags that are generated dynamically.

share|improve this question
In other words, you need wget to execute Javascript? – Bob Apr 16 '12 at 8:45
up vote 1 down vote accepted

Javascript is rendered by the browser. wget does exactly what it's supposed to do, fetching the content. Browsers do the same thing initially. They get the content exactly how you posted above. But then it renders the Javascript part and builds the links. wget can't do that. So, no, you can't get links that are generated dynamically, using just wget. You can try something like PhantomJS though.

share|improve this answer

As stated already, wget is not able to generate pages that use client-side JavaScript code. If you know the basics of Python programming, I would recommend using the Python library Scrapy for crawling the web site, together with Selenium, which is able to use an external browser to generate dynamic pages. You can do all this with a tiny amount of Python code. See for example Code Snippets Collection.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .