Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to crawl the internet starting at a certain website and get a list of URLs that were visited. This is what I have so far:

wget --spider -r --no-verbose -o out.txt <url>

The problem with this is that it creates a folder structure on the disk of the websites it visited. I don't want to download any files or create folders on disk. How can I achieve this?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

Try the -nd or --no-directories option.

share|improve this answer

Maybe not exactly what you want, but the --delete-after option might work for you.

share|improve this answer
    
--delete-after does not delete the directory. –  Dennis Feb 5 '13 at 18:30
    
@Dennis ok, good to know. –  Michael K Feb 6 '13 at 10:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.