Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How do I extract all the external links of a web page and save them to a file?

If you have any command line tools that would be great.

share|improve this question
up vote 11 down vote accepted

You will need 2 tools, lynx and awk, try this:

$ lynx -dump | awk '/http/{print $2}' > links.txt

If you need numbering lines, use command nl, try this:

$ lynx -dump | awk '/http/{print $2}' | nl > links.txt
share|improve this answer

Here's an improvement on lilton's answer: you don't need awk at all for lynx's got some useful options.

lynx -listonly -nonumbers -dump

if you want numbers

lynx -listonly -dump
share|improve this answer
  1. Use Beautiful Soup to retrieve the web pages in question.
  2. Use awk to find all URLs that do not point to your domain

I would recommend Beautiful Soup over screen scraping techniques.

share|improve this answer

if command line is not a force you can use Copy All Links Firefox extension.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.