How do I extract all the external links of a web page and save them to a file?

Question

How do I extract all the external links of a web page and save them to a file?

If you have any command line tools that would be great.

Answer 1

You will need 2 tools, lynx and awk, try this:

$ lynx -dump http://www.google.com.br | awk '/http/{print $2}' > links.txt

If you need numbering lines, use command nl, try this:

$ lynx -dump http://www.google.com.br | awk '/http/{print $2}' | nl > links.txt

Answer 2

1. Use Beautiful Soup to retrieve the web pages in question.
2. Use awk to find all URLs that do not point to your domain

I would recommend Beautiful Soup over screen scraping techniques.