Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm running

 wget -mk --no-check-certificate

And it seems to be pulling down all the pages and resources and blog posts, but it's missing a large number of the /actionstream/* URIs. It gets some of them, and each of these pages has prev/next links which should mean it can get all of them, but it doesn't. No errors in the output.

I tried changing to start at but that didn't work either, it in fact got even less of the /actionstream/* items.

There must be some switch that I'm missing. I thought the -l inf from -m would do it.

Any ideas?

share|improve this question
up vote 1 down vote accepted

You are most likely hit by bug #31354: wget does not parse html files linked by link rel="...". Fix is already in trunk for some time (since r2434). Below I pasted excerpt from ChangeLog regarding it.

2010-10-18  Manfred Koizar  (tiny change)

    * html-url.c (tag_handle_link): Do not assume external links type
    to be always "text/html".

Last officially released version is unfortunately pretty old (1.12 came out in September 2009), so you have to download and build more recent version. GNU software has site, where you can obtain more recent versions.

share|improve this answer
This ChangeLog entry is one of those stealth examples, where you smuggle more than it seems, w/o any proper note. Really BAD practice! Tiny change, huh? – przemoc May 5 '11 at 23:45
I updated to 1.12-2474 and get the same problem. There are clearly links to the pages that did not get downloaded in the ones that did. I'm very confused. – singpolyma May 6 '11 at 15:21
@singpolyma Can you tell what's missing in my output of wget -mk --no-check-certificate --spider -o log '‌​ormond-thats/'; grep '^--' log | gzip > available at BTW there is a lot of read errors, and some links are broken (46 in my wget run). – przemoc May 8 '11 at 19:47
It seems possible nothing is missing there. grep '/actionstream/' your-log | grep -v try: | wc -l says 10201 which seems plausible, unlike the < 200 that I get downloaded when I run without --spider. What are they retries? Are those the webserver timing out? – singpolyma May 9 '11 at 13:48
After seeing your log I thought to try adding -t inf After doing so wget sucks up RAM really fast and eventually the kernel kills it... – singpolyma May 10 '11 at 13:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .