4

Hi trying to determine all the valid urls under a given domain without having the mirror the site locally.

People generally want to download all the pages but I just want to get a list of the direct urls under a given domain (e.g. www.example.com), which would be something like www.example.com/page1, www.example.com/page2, etc.

Is there a way to use wget to do this? or is there a better tool for this?

  • 2
    In order to determine the links on each page, you will need to see the page (I.e. Download it) – Brian Adkins Sep 24 '13 at 18:59
  • @BrianAdkins: I am Ok with downloading but I woant to keep only the urls not the pages' contents – fccoelho Sep 24 '13 at 19:05
  • There's a --spider option that downloads the page, but doesn't save it. – LawrenceC Sep 25 '13 at 22:43
2

Here is a crude script:

curl -s whaturl |
  grep -o "<a href=[^>]*>" |
  sed -r 's/<a href="([^"]*)".*>/\1/' |
  sort -u

The grep picks all the hrefs. The sed picks out the url part from the href. The sort filters out duplicate links.

It will also work with wget -O - in place of curl -s.

Example output:

$ curl -s http://stackexchange.com/users/148837/lesmana?tab=accounts | grep -o "<a href=[^>]*>" | sed -r 's/<a href="([^"]*)".*>/\1/' | sort -u
/
/about
/about/contact
/blogs
/leagues
/legal
/legal/privacy-policy
/newsletters
/questions
/sites
/users/148837/lesmana
/users/148837/lesmana?tab=activity
/users/148837/lesmana?tab=favorites
/users/148837/lesmana?tab=reputation
/users/148837/lesmana?tab=subscriptions
/users/148837/lesmana?tab=top
/users/login?returnurl=%2fusers%2f148837%2flesmana%3ftab%3daccounts
http://area51.stackexchange.com/users/16563/lesmana
http://askubuntu.com/users/1366/
http://blog.stackexchange.com
http://blog.stackoverflow.com/2009/06/attribution-required/
http://chat.stackexchange.com/
http://creativecommons.org/licenses/by-sa/3.0/
http://gaming.stackexchange.com/users/2790/
http://meta.stackoverflow.com
http://meta.stackoverflow.com/users/147747/
http://programmers.stackexchange.com/users/116/
http://serverfault.com/users/45166/
http://stackoverflow.com/users/360899/
http://superuser.com/users/39401/
http://twitter.com/stackexchange
http://unix.stackexchange.com/users/1170/
http://www.facebook.com/stackexchange
https://plus.google.com/+StackExchange
|improve this answer|||||
  • Very nice! I had ignored CURL because it can't recurse. I found that httrack solves this problem adequately. – fccoelho Sep 24 '13 at 19:54
4

Ok, I had to find my own answer:

the tool I use was httrack.

httrack -p0 -r2 -d www.example.com
  • the -p0 option tells it to just scan (not save pages);
  • the -rx option tells it the depth of the search
  • the -d options tells it to stay on the same principal domain

there is even a -%L to add the scanned URL to the specified file but it doesn't seem to work. But that's not a problem, because under the hts-cache directory you can find a TSV file named new.txt containing all the urls visited and some additional information about it. I could extract the URLs from it with the following python code:

with open("hts-cache/new.txt") as f:
    t = csv.DictReader(f,delimiter='\t')
    for l in t:
        print l['URL']
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.