I need to periodically traverse a site to update the server cache. It is a database driven site with very many (thousands of) pages. I use wget to mirror the site locally on the same server, so I use this command:

wget --mirror localhost

After some time it stops suddenly with this message:

HTTP request sent, awaiting response... Terminated

It happens always, but not on the exact same url. And I have tried on another server as well with the same result.

The --debug option does not provide any helpful information, nor does Apache's log file.

What could be the cause of this problem? I suspect some buffer running out of memory etc. Or a stack overflow.

Alternatively, are there other command line tools that can do the same?

It's Wget 1.11.4 on Debian Lenny.



It's possible that the server is performing some sort of analysis of your download patterns and squashing your requests. Take a look at some of wget's options for limiting the rate of your requests -- look at --limit-rate, --wait, and --random-wait.

  • Thanks for the suggestions. Unfortunately none of these options helps. In the meantime I have found httrack. I will now see if that works better.... – marlar Aug 20 '10 at 20:53

I have found no ways to make wget traverse the full site without terminating prematurely, but I stumbled upon httrack which does the job perfectly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.