Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

i have txt file that contains some website.i'd like to extract websites how can i do this?"GET",filename,false );
<li><a href="" TARGET=_BLANK></a></li>
<a href="" TARGET="other"><img src="art/docomo.gif" width="104" height="80" align="middle" border="0" alt="NTT Docomo"></a>
<a href="" TARGET="other"><img src="art/google.gif" width="121" height="50" align="middle" border="0" alt="Google"></a>
<a href="" TARGET="other"><img src="art/hp_
<a href="" TARGET="other"><img src="art/ibmlogo.gif" width="100" height="38" align="middle" border="0" alt="IBM"></a>
<a href="" TARGET="other"><img src="art/microsoft.gif" width="135" height="50" align="middle" border="0" alt="Microsoft Research"></a>
<a href="" TARGET="other"><img src="art/vmware.gif" width="140" height="40" align="middle" border="0" alt="vmware"></a>
function domainForHostname(hostname) {
Vary: Host
Allow: GET

and the output:

Edit: another input example

GET /mail/channel/bind?at=1a319b156176da12-1125900daa3&SID=6FA1DA0F17C9527D&RID=42925&TYPE=terminate&zx=ey4h3dgxde6e HTTP/1.1
GET / HTTP/1.1
GET / HTTP/1.1
GET /index.php?language=en&partner=%22%3E%3Cscript%3Ealert(123456789)%3C/script%3E  HTTP/1.1
GET /google/ads/sidenav_sky.js?7927068257853024 HTTP/1.1
GET /pagead/ads?client=ca-ostg_js&dt=1178313198977&adsafe=high&oe=utf8&num_ads=5&output=js&channel=slashdot_sidenav& image_size=125x600&region=apple%20askslashdot%20backslash%20books%20developers%20games%20hardware%20interviews%20it%20linux%20politics%20science%20yro%20default&feedback_link=on& HTTP/1.1
GET /adj/ostg.slashdot/mainpage_p1_leader;pg=index;logged_in=0;tile=1;ord=7927068257853024? HTTP/1.1
Host:"GET",filename,false );
GET /BurstingPipe/$$http: // /0%3B20810775/20828668/1%3B%3B%7Esscs%3D%3f$$&ord=5013023 HTTP/1.1
GET /BurstingRes//Site-454/Type-0/24E08935-2B74-491B-B910-A06A81D9FE97.jpg HTTP/1.1
<li><a href="" TARGET=_BLANK></a></li>
GET /topics/topicdoj.gif HTTP/1.1
GET /medias/btnPanierFermer.gif HTTP/1.1
GET /__utm.gif?utmwv=1&utmn=1735117121&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=7.0%20r25&utmdt=Slashdot%3A%20News%20for%20nerds%2C%20stuff%20that%20matters& HTTP/1.1
GET /search?q=hotos HTTP/1.1
GET /hotos05/ HTTP/1.1
GET /7205/20070504/ HTTP/1.1
GET /events/hotos05/index.html HTTP/1.1
GET /styles/core.css?new HTTP/1.1
share|improve this question
up vote 5 down vote accepted

The first example use grep to grab all the links like:

$ grep -o 'http[^"]*' file

For the second use awk to print the second field on lines where the first is Host::

$ awk '$1=="Host:"{print $2}' file
share|improve this answer
Much more elegant than my solution! – Journeyman Geek Apr 18 '13 at 13:58

A simplistic approach using elinks(1) ability to dump files, as described in the manpage

   -dump [0|1] (default: 0)
       Print formatted plain-text versions of given URLs to stdout.

could be:

$ elinks -dump < infile | awk '$0~/\s*[[:digit:]]*\. http/ {print $2}'

This could catch unwanted lines, of course. Improve the regex used to match your criteria.

Other text-mode browsers (lynx, links) and some pagers (w3m) also have a dump option.

share|improve this answer
what is elinks doing here? and how can i done it with out elinks?tnx – Arash Apr 18 '13 at 10:15

Assuming you want to extract this from an existing file (in this case called blag.text) you can use cat blag.txt| grep http |cut -d \" -f2 for the first example

First, extract lines containing http with grep. This gives you lines like <li><a href="" TARGET=_BLANK></a></li>. We then use the quotation marks as delimiters for cut, but since quotation marks are also used to enclose strings, we need to escape it with /

For the second, you'd probably want to grep for "host" then use : as the deliminator (you can also use the space after the colon the same way) cat blag2.txt | grep Host |cut -d : -f2 is how I'd do it, though cat blag2.txt | grep Host |cut -d \ -f2 is more elegant. There's two spaces after / one the space we're using the deliminator, and one to seperate that from the next arguement.

share|improve this answer

grep with -o option extracts text of given pattern each per line. For example, the following command extracts all citations of the form \cite{citationKey} from a latex file.

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex

To redirect the output to another file, use

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex > outputfile.tex
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.