11

How do I extract all the external links of a web page and save them to a file?

If you have any command line tools that would be great.

18

You will need 2 tools, lynx and awk, try this:

$ lynx -dump http://www.google.com.br | awk '/http/{print $2}' > links.txt

If you need numbering lines, use command nl, try this:

$ lynx -dump http://www.google.com.br | awk '/http/{print $2}' | nl > links.txt
7

Here's an improvement on lelton's answer: you don't need awk at all for lynx's got some useful options.

lynx -listonly -nonumbers -dump http://www.google.com.br

if you want numbers

lynx -listonly -dump http://www.google.com.br
0
  1. Use Beautiful Soup to retrieve the web pages in question.
  2. Use awk to find all URLs that do not point to your domain

I would recommend Beautiful Soup over screen scraping techniques.

0

if command line is not a force you can use Copy All Links Firefox extension.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.