Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm running

 wget -mk --no-check-certificate https://singpolyma.net/

And it seems to be pulling down all the pages and resources and blog posts, but it's missing a large number of the /actionstream/* URIs. It gets some of them, and each of these pages has prev/next links which should mean it can get all of them, but it doesn't. No errors in the output.

I tried changing to start at https://singpolyma.net/actionstream/reply-httpidenti-canotice72478545-russellmcormond-thats/ but that didn't work either, it in fact got even less of the /actionstream/* items.

There must be some switch that I'm missing. I thought the -l inf from -m would do it.

Any ideas?

share|improve this question
up vote 1 down vote accepted

You are most likely hit by bug #31354: wget does not parse html files linked by link rel="...". Fix is already in trunk for some time (since r2434). Below I pasted excerpt from ChangeLog regarding it.

2010-10-18  Manfred Koizar  (tiny change)

    * html-url.c (tag_handle_link): Do not assume external links type
    to be always "text/html".

Last officially released version is unfortunately pretty old (1.12 came out in September 2009), so you have to download and build more recent version. GNU software has alpha.gnu.org/gnu site, where you can obtain more recent versions.

share|improve this answer
    
This ChangeLog entry is one of those stealth examples, where you smuggle more than it seems, w/o any proper note. Really BAD practice! Tiny change, huh? – przemoc May 5 '11 at 23:45
    
I updated to 1.12-2474 and get the same problem. There are clearly links to the pages that did not get downloaded in the ones that did. I'm very confused. – singpolyma May 6 '11 at 15:21
    
@singpolyma Can you tell what's missing in my output of wget -mk --no-check-certificate --spider -o log 'https://singpolyma.net/actionstream/reply-httpidenti-canotice72478545-russellmc‌​ormond-thats/'; grep '^--' log | gzip > singpolyma.net-urls.gz available at http://min.us/mvoZpQV? BTW there is a lot of read errors, and some links are broken (46 in my wget run). – przemoc May 8 '11 at 19:47
    
It seems possible nothing is missing there. grep '/actionstream/' your-log | grep -v try: | wc -l says 10201 which seems plausible, unlike the < 200 that I get downloaded when I run without --spider. What are they retries? Are those the webserver timing out? – singpolyma May 9 '11 at 13:48
    
After seeing your log I thought to try adding -t inf After doing so wget sucks up RAM really fast and eventually the kernel kills it... – singpolyma May 10 '11 at 13:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .