Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

i have txt file that contains some website.i'd like to extract websites how can i do this?

xhr_object.open("GET",filename,false );
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
<a href="http://www.docomolabs-usa.com/" TARGET="other"><img src="art/docomo.gif" width="104" height="80" align="middle" border="0" alt="NTT Docomo"></a>
<a href="http://www.google.com/" TARGET="other"><img src="art/google.gif" width="121" height="50" align="middle" border="0" alt="Google"></a>
<a href="http://www.hpl.hp.com/" TARGET="other"><img src="art/hp_
<a href="http://www.ibm.com/" TARGET="other"><img src="art/ibmlogo.gif" width="100" height="38" align="middle" border="0" alt="IBM"></a>
<a href="http://research.microsoft.com/" TARGET="other"><img src="art/microsoft.gif" width="135" height="50" align="middle" border="0" alt="Microsoft Research"></a>
<a href="http://www.vmware.com/" TARGET="other"><img src="art/vmware.gif" width="140" height="40" align="middle" border="0" alt="vmware"></a>
function domainForHostname(hostname) {
Vary: Host
Allow: GET

and the output:

http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

Edit: another input example

Host: mail.google.com
Host: mail.google.com
GET /mail/channel/bind?at=1a319b156176da12-1125900daa3&SID=6FA1DA0F17C9527D&RID=42925&TYPE=terminate&zx=ey4h3dgxde6e HTTP/1.1
Host: mail.google.com
GET / HTTP/1.1
Host: www.slashdot.org
GET / HTTP/1.1
Host: slashdot.org
GET /index.php?language=en&partner=%22%3E%3Cscript%3Ealert(123456789)%3C/script%3E  HTTP/1.1
Host: store.dakar.com
GET /google/ads/sidenav_sky.js?7927068257853024 HTTP/1.1
Host: genweb.ostg.com
GET /pagead/ads?client=ca-ostg_js&dt=1178313198977&adsafe=high&oe=utf8&num_ads=5&output=js&channel=slashdot_sidenav&url=http%3A%2F%2Fslashdot.org%2F&ad_type=text_image& image_size=125x600&region=apple%20askslashdot%20backslash%20books%20developers%20games%20hardware%20interviews%20it%20linux%20politics%20science%20yro%20default&feedback_link=on&loc=http%3A%2F%2Fslashdot.org%2F&cc=96&flash=7&u_h=1024&u_w=1280&u_ah=1024&u_aw=1280&u_cd=24&u_tz=-420&u_his=6&u_java=true&u_nplug=1&u_nmime=2 HTTP/1.1
Host: pagead2.googlesyndication.com
GET /adj/ostg.slashdot/mainpage_p1_leader;pg=index;logged_in=0;tile=1;ord=7927068257853024? HTTP/1.1
Host: ad.doubleclick.net
xhr_object.open("GET",filename,false );
GET /BurstingPipe/adServer.bs?cn=sb&c=17&pli=222855&pi=0&w=728&h=90&ncu=$$http: //ad.doubleclick.net/click%3Bh=v8/3548/3/0/%2a/k%3B99478854%3B0-0%3B0%3B13358359%3B255-0 /0%3B20810775/20828668/1%3B%3B%7Esscs%3D%3f$$&ord=5013023 HTTP/1.1
Host: bs.serving-sys.com
GET /BurstingRes//Site-454/Type-0/24E08935-2B74-491B-B910-A06A81D9FE97.jpg HTTP/1.1
Host: ds-ll.serving-sys.com
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
GET /topics/topicdoj.gif HTTP/1.1
Host: images.slashdot.org
GET /medias/btnPanierFermer.gif HTTP/1.1
Host: store.dakar.com
GET /__utm.gif?utmwv=1&utmn=1735117121&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=7.0%20r25&utmdt=Slashdot%3A%20News%20for%20nerds%2C%20stuff%20that%20matters&utmhn=slashdot.org&utmr=-&utmp=/&utmac=UA-32013-5&utmcc=__utma%3D9273847.1199151991.1141182756.1178312303.1178313207.20%3B%2B__utmb%3D9273847%3B%2B__utmc%3D9273847%3B%2B__utmz%3D9273847.1163632638.13.3.utmccn%3D(organic)%7Cutmcsr%3Dgoogle%7Cutmctr%3DVM%2Bbased%2Brootkits%7Cutmcmd%3Dorganic%3B%2B HTTP/1.1
Host: www.google-analytics.com
GET /search?q=hotos HTTP/1.1
Host: www.google.com
GET /hotos05/ HTTP/1.1
Host: www.usenix.org
GET /7205/20070504/ HTTP/1.1
Host: www.thelocal.se
GET /events/hotos05/index.html HTTP/1.1
Host: www.usenix.org
GET /styles/core.css?new HTTP/1.1
Host: www.thelocal.se
share|improve this question
up vote 5 down vote accepted

The first example use grep to grab all the links like:

$ grep -o 'http[^"]*' file
http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

For the second use awk to print the second field on lines where the first is Host::

$ awk '$1=="Host:"{print $2}' file
mail.google.com
mail.google.com
mail.google.com
www.slashdot.org
slashdot.org
store.dakar.com
genweb.ostg.com
pagead2.googlesyndication.com
ad.doubleclick.net
bs.serving-sys.com
ds-ll.serving-sys.com
images.slashdot.org
store.dakar.com
www.google-analytics.com
www.google.com
www.usenix.org
www.thelocal.se
www.usenix.org
www.thelocal.se
share|improve this answer
1  
Much more elegant than my solution! – Journeyman Geek Apr 18 '13 at 13:58

A simplistic approach using elinks(1) ability to dump files, as described in the manpage

   -dump [0|1] (default: 0)
       Print formatted plain-text versions of given URLs to stdout.

could be:

$ elinks -dump < infile | awk '$0~/\s*[[:digit:]]*\. http/ {print $2}'
http://www.dakar.com/
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.hpl.hp.com/
http://research.microsoft.com/
http://www.vmware.com/

This could catch unwanted lines, of course. Improve the regex used to match your criteria.

Other text-mode browsers (lynx, links) and some pagers (w3m) also have a dump option.

share|improve this answer
    
what is elinks doing here? and how can i done it with out elinks?tnx – Arash Apr 18 '13 at 10:15

Assuming you want to extract this from an existing file (in this case called blag.text) you can use cat blag.txt| grep http |cut -d \" -f2 for the first example

First, extract lines containing http with grep. This gives you lines like <li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>. We then use the quotation marks as delimiters for cut, but since quotation marks are also used to enclose strings, we need to escape it with /

For the second, you'd probably want to grep for "host" then use : as the deliminator (you can also use the space after the colon the same way) cat blag2.txt | grep Host |cut -d : -f2 is how I'd do it, though cat blag2.txt | grep Host |cut -d \ -f2 is more elegant. There's two spaces after / one the space we're using the deliminator, and one to seperate that from the next arguement.

share|improve this answer

grep with -o option extracts text of given pattern each per line. For example, the following command extracts all citations of the form \cite{citationKey} from a latex file.

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex

To redirect the output to another file, use

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex > outputfile.tex
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .