Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 0 down vote favorite
1

I'm running

 wget -mk --no-check-certificate https://singpolyma.net/

And it seems to be pulling down all the pages and resources and blog posts, but it's missing a large number of the /actionstream/* URIs. It gets some of them, and each of these pages has prev/next links which should mean it can get all of them, but it doesn't. No errors in the output.

I tried changing to start at https://singpolyma.net/actionstream/reply-httpidenti-canotice72478545-russellmcormond-thats/ but that didn't work either, it in fact got even less of the /actionstream/* items.

There must be some switch that I'm missing. I thought the -l inf from -m would do it.

Any ideas?

share|improve this question

1 Answer

up vote 1 down vote accepted

You are most likely hit by bug #31354: wget does not parse html files linked by link rel="...". Fix is already in trunk for some time (since r2434). Below I pasted excerpt from ChangeLog regarding it.

2010-10-18  Manfred Koizar  (tiny change)

    * html-url.c (tag_handle_link): Do not assume external links type
    to be always "text/html".

Last officially released version is unfortunately pretty old (1.12 came out in September 2009), so you have to download and build more recent version. GNU software has alpha.gnu.org/gnu site, where you can obtain more recent versions.

share|improve this answer
This ChangeLog entry is one of those stealth examples, where you smuggle more than it seems, w/o any proper note. Really BAD practice! Tiny change, huh? – przemoc May 5 '11 at 23:45
I updated to 1.12-2474 and get the same problem. There are clearly links to the pages that did not get downloaded in the ones that did. I'm very confused. – singpolyma May 6 '11 at 15:21
@singpolyma Can you tell what's missing in my output of wget -mk --no-check-certificate --spider -o log 'https://singpolyma.net/actionstream/reply-httpidenti-canotice72478545-russellmc‌​ormond-thats/'; grep '^--' log | gzip > singpolyma.net-urls.gz available at http://min.us/mvoZpQV? BTW there is a lot of read errors, and some links are broken (46 in my wget run). – przemoc May 8 '11 at 19:47
It seems possible nothing is missing there. grep '/actionstream/' your-log | grep -v try: | wc -l says 10201 which seems plausible, unlike the < 200 that I get downloaded when I run without --spider. What are they retries? Are those the webserver timing out? – singpolyma May 9 '11 at 13:48
After seeing your log I thought to try adding -t inf After doing so wget sucks up RAM really fast and eventually the kernel kills it... – singpolyma May 10 '11 at 13:21
show 8 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.