Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
How can I download an entire website

Would somebody have a recommendation for a web site crawler that can be invoked and equipped with settings from the command line?

This would need to run in a Windows environment.

Saving the data, following stylesheet links etc. is not an issue. I only need the crawler to start with a page, parse it, and follow all the links on the same domain so that in the end, all pages on the site have been requested once.

Background: I'm setting up a web site that gets frequently uploaded from an office location. Combining data from various sources, it has several levels of caching. I don't want the first user to visit the site after a fresh upload to have to wait until the page has been generated and saved in the cache.

share|improve this question

marked as duplicate by Arjan, random May 25 '10 at 4:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

up vote 2 down vote accepted

wget --mirror

share|improve this answer
wget continues to surprise me :D cheers, will try this first. – Pekka 웃 May 24 '10 at 17:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.