Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

i have txt file that contains some website.i'd like to extract websites how can i do this?

xhr_object.open("GET",filename,false );
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
<a href="http://www.docomolabs-usa.com/" TARGET="other"><img src="art/docomo.gif" width="104" height="80" align="middle" border="0" alt="NTT Docomo"></a>
<a href="http://www.google.com/" TARGET="other"><img src="art/google.gif" width="121" height="50" align="middle" border="0" alt="Google"></a>
<a href="http://www.hpl.hp.com/" TARGET="other"><img src="art/hp_
<a href="http://www.ibm.com/" TARGET="other"><img src="art/ibmlogo.gif" width="100" height="38" align="middle" border="0" alt="IBM"></a>
<a href="http://research.microsoft.com/" TARGET="other"><img src="art/microsoft.gif" width="135" height="50" align="middle" border="0" alt="Microsoft Research"></a>
<a href="http://www.vmware.com/" TARGET="other"><img src="art/vmware.gif" width="140" height="40" align="middle" border="0" alt="vmware"></a>
function domainForHostname(hostname) {
Vary: Host
Allow: GET

and the output:

http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

Edit: another input example

Host: mail.google.com
Host: mail.google.com
GET /mail/channel/bind?at=1a319b156176da12-1125900daa3&SID=6FA1DA0F17C9527D&RID=42925&TYPE=terminate&zx=ey4h3dgxde6e HTTP/1.1
Host: mail.google.com
GET / HTTP/1.1
Host: www.slashdot.org
GET / HTTP/1.1
Host: slashdot.org
GET /index.php?language=en&partner=%22%3E%3Cscript%3Ealert(123456789)%3C/script%3E  HTTP/1.1
Host: store.dakar.com
GET /google/ads/sidenav_sky.js?7927068257853024 HTTP/1.1
Host: genweb.ostg.com
GET /pagead/ads?client=ca-ostg_js&dt=1178313198977&adsafe=high&oe=utf8&num_ads=5&output=js&channel=slashdot_sidenav&url=http%3A%2F%2Fslashdot.org%2F&ad_type=text_image& image_size=125x600&region=apple%20askslashdot%20backslash%20books%20developers%20games%20hardware%20interviews%20it%20linux%20politics%20science%20yro%20default&feedback_link=on&loc=http%3A%2F%2Fslashdot.org%2F&cc=96&flash=7&u_h=1024&u_w=1280&u_ah=1024&u_aw=1280&u_cd=24&u_tz=-420&u_his=6&u_java=true&u_nplug=1&u_nmime=2 HTTP/1.1
Host: pagead2.googlesyndication.com
GET /adj/ostg.slashdot/mainpage_p1_leader;pg=index;logged_in=0;tile=1;ord=7927068257853024? HTTP/1.1
Host: ad.doubleclick.net
xhr_object.open("GET",filename,false );
GET /BurstingPipe/adServer.bs?cn=sb&c=17&pli=222855&pi=0&w=728&h=90&ncu=$$http: //ad.doubleclick.net/click%3Bh=v8/3548/3/0/%2a/k%3B99478854%3B0-0%3B0%3B13358359%3B255-0 /0%3B20810775/20828668/1%3B%3B%7Esscs%3D%3f$$&ord=5013023 HTTP/1.1
Host: bs.serving-sys.com
GET /BurstingRes//Site-454/Type-0/24E08935-2B74-491B-B910-A06A81D9FE97.jpg HTTP/1.1
Host: ds-ll.serving-sys.com
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
GET /topics/topicdoj.gif HTTP/1.1
Host: images.slashdot.org
GET /medias/btnPanierFermer.gif HTTP/1.1
Host: store.dakar.com
GET /__utm.gif?utmwv=1&utmn=1735117121&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=7.0%20r25&utmdt=Slashdot%3A%20News%20for%20nerds%2C%20stuff%20that%20matters&utmhn=slashdot.org&utmr=-&utmp=/&utmac=UA-32013-5&utmcc=__utma%3D9273847.1199151991.1141182756.1178312303.1178313207.20%3B%2B__utmb%3D9273847%3B%2B__utmc%3D9273847%3B%2B__utmz%3D9273847.1163632638.13.3.utmccn%3D(organic)%7Cutmcsr%3Dgoogle%7Cutmctr%3DVM%2Bbased%2Brootkits%7Cutmcmd%3Dorganic%3B%2B HTTP/1.1
Host: www.google-analytics.com
GET /search?q=hotos HTTP/1.1
Host: www.google.com
GET /hotos05/ HTTP/1.1
Host: www.usenix.org
GET /7205/20070504/ HTTP/1.1
Host: www.thelocal.se
GET /events/hotos05/index.html HTTP/1.1
Host: www.usenix.org
GET /styles/core.css?new HTTP/1.1
Host: www.thelocal.se
share|improve this question

3 Answers 3

up vote 3 down vote accepted

The first example use grep to grab all the links like:

$ grep -o 'http[^"]*' file
http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

For the second use awk to print the second field on lines where the first is Host::

$ awk '$1=="Host:"{print $2}' file
mail.google.com
mail.google.com
mail.google.com
www.slashdot.org
slashdot.org
store.dakar.com
genweb.ostg.com
pagead2.googlesyndication.com
ad.doubleclick.net
bs.serving-sys.com
ds-ll.serving-sys.com
images.slashdot.org
store.dakar.com
www.google-analytics.com
www.google.com
www.usenix.org
www.thelocal.se
www.usenix.org
www.thelocal.se
share|improve this answer
1  
Much more elegant than my solution! –  Journeyman Geek Apr 18 '13 at 13:58

A simplistic approach using elinks(1) ability to dump files, as described in the manpage

   -dump [0|1] (default: 0)
       Print formatted plain-text versions of given URLs to stdout.

could be:

$ elinks -dump < infile | awk '$0~/\s*[[:digit:]]*\. http/ {print $2}'
http://www.dakar.com/
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.hpl.hp.com/
http://research.microsoft.com/
http://www.vmware.com/

This could catch unwanted lines, of course. Improve the regex used to match your criteria.

Other text-mode browsers (lynx, links) and some pagers (w3m) also have a dump option.

share|improve this answer
    
what is elinks doing here? and how can i done it with out elinks?tnx –  Christopher Apr 18 '13 at 10:15

Assuming you want to extract this from an existing file (in this case called blag.text) you can use cat blag.txt| grep http |cut -d \" -f2 for the first example

First, extract lines containing http with grep. This gives you lines like <li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>. We then use the quotation marks as delimiters for cut, but since quotation marks are also used to enclose strings, we need to escape it with /

For the second, you'd probably want to grep for "host" then use : as the deliminator (you can also use the space after the colon the same way) cat blag2.txt | grep Host |cut -d : -f2 is how I'd do it, though cat blag2.txt | grep Host |cut -d \ -f2 is more elegant. There's two spaces after / one the space we're using the deliminator, and one to seperate that from the next arguement.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.