Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to crawl the internet starting at a certain website and get a list of URLs that were visited. This is what I have so far:

wget --spider -r --no-verbose -o out.txt <url>

The problem with this is that it creates a folder structure on the disk of the websites it visited. I don't want to download any files or create folders on disk. How can I achieve this?

share|improve this question
up vote 1 down vote accepted

Try the -nd or --no-directories option.

share|improve this answer

Maybe not exactly what you want, but the --delete-after option might work for you.

share|improve this answer
    
--delete-after does not delete the directory. – Dennis Feb 5 '13 at 18:30
    
@Dennis ok, good to know. – Michael K Feb 6 '13 at 10:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .