Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to crawl the internet starting at a certain website and get a list of URLs that were visited. This is what I have so far:

wget --spider -r --no-verbose -o out.txt <url>

The problem with this is that it creates a folder structure on the disk of the websites it visited. I don't want to download any files or create folders on disk. How can I achieve this?

share|improve this question
up vote 1 down vote accepted

Try the -nd or --no-directories option.

share|improve this answer

Maybe not exactly what you want, but the --delete-after option might work for you.

share|improve this answer
--delete-after does not delete the directory. – Dennis Feb 5 '13 at 18:30
@Dennis ok, good to know. – Michael K Feb 6 '13 at 10:08

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.