Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am not asking how to download a standard webpage, or website tree, as I know how to do that.

The problem I am having is that wget/downloadthemall/HTTrack/FDM/IDM, etc., do not seem to work with the blog format.

They should in theory, as it is still a standard webpage with links, yet they don't.

I have tried wget with both -m and -r -l3 to no avail, as well as downthemall.

The problem I am having is that these downloader programs do not seem to follow the tag system, or are not aware that the majority of content to follow is behind the "older posts" type links.

Is there a way to customise the use of one of these downloader programs to follow a specific path through a website, without scripting?

share|improve this question
It would be nice to know what blog, specifically, you are talking about. – digitxp Sep 19 '10 at 19:51
I can't make the question specific to any site or service as per the FAQ, but lets use blogger as an example. – user48869 Sep 19 '10 at 20:12

Are the blog links pointing to another domain, or something that looks like another domain? For example, you might be telling wget to fetch everything from "", but the links point to "", which resolves to the same page but still possibly might confuse wget.

share|improve this answer
No, the links all seem to be on the same domain, but for whatever reason wont parse the image tags and save images. The cgi stuff also seems to cause a problem, as I often get the same page multiple times with many different filenames based on what wget or whatever requested... – user48869 Sep 20 '10 at 12:08
Maybe the image directories might be protected from hotlinking, and one way to do that is refuse downloads if the HTTP referer is not correct. Investigate wget's http-referer option. – LawrenceC Sep 21 '10 at 2:39

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.