Possible Duplicate:
How can I download an entire website

Would somebody have a recommendation for a web site crawler that can be invoked and equipped with settings from the command line?

This would need to run in a Windows environment.

Saving the data, following stylesheet links etc. is not an issue. I only need the crawler to start with a page, parse it, and follow all the links on the same domain so that in the end, all pages on the site have been requested once.

Background: I'm setting up a web site that gets frequently uploaded from an office location. Combining data from various sources, it has several levels of caching. I don't want the first user to visit the site after a fresh upload to have to wait until the page has been generated and saved in the cache.

marked as duplicate by Arjan, random May 25 '10 at 4:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


wget --mirror

Not the answer you're looking for? Browse other questions tagged or ask your own question.