Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to periodically traverse a site to update the server cache. It is a database driven site with very many (thousands of) pages. I use wget to mirror the site locally on the same server, so I use this command:

wget --mirror localhost

After some time it stops suddenly with this message:

HTTP request sent, awaiting response... Terminated

It happens always, but not on the exact same url. And I have tried on another server as well with the same result.

The --debug option does not provide any helpful information, nor does Apache's log file.

What could be the cause of this problem? I suspect some buffer running out of memory etc. Or a stack overflow.

Alternatively, are there other command line tools that can do the same?

It's Wget 1.11.4 on Debian Lenny.

Martin

share|improve this question

It's possible that the server is performing some sort of analysis of your download patterns and squashing your requests. Take a look at some of wget's options for limiting the rate of your requests -- look at --limit-rate, --wait, and --random-wait.

share|improve this answer
    
Thanks for the suggestions. Unfortunately none of these options helps. In the meantime I have found httrack. I will now see if that works better.... – marlar Aug 20 '10 at 20:53

I have found no ways to make wget traverse the full site without terminating prematurely, but I stumbled upon httrack which does the job perfectly.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .