0

I am trying to crawl the internet starting at a certain website and get a list of URLs that were visited. This is what I have so far:

wget --spider -r --no-verbose -o out.txt <url>

The problem with this is that it creates a folder structure on the disk of the websites it visited. I don't want to download any files or create folders on disk. How can I achieve this?

2 Answers 2

1

Try the -nd or --no-directories option.

0

Maybe not exactly what you want, but the --delete-after option might work for you.

2
  • --delete-after does not delete the directory.
    – Dennis
    Feb 5, 2013 at 18:30
  • @Dennis ok, good to know.
    – Michael K
    Feb 6, 2013 at 10:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .