4

i have txt file that contains some website.i'd like to extract websites how can i do this?

xhr_object.open("GET",filename,false );
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
<a href="http://www.docomolabs-usa.com/" TARGET="other"><img src="art/docomo.gif" width="104" height="80" align="middle" border="0" alt="NTT Docomo"></a>
<a href="http://www.google.com/" TARGET="other"><img src="art/google.gif" width="121" height="50" align="middle" border="0" alt="Google"></a>
<a href="http://www.hpl.hp.com/" TARGET="other"><img src="art/hp_
<a href="http://www.ibm.com/" TARGET="other"><img src="art/ibmlogo.gif" width="100" height="38" align="middle" border="0" alt="IBM"></a>
<a href="http://research.microsoft.com/" TARGET="other"><img src="art/microsoft.gif" width="135" height="50" align="middle" border="0" alt="Microsoft Research"></a>
<a href="http://www.vmware.com/" TARGET="other"><img src="art/vmware.gif" width="140" height="40" align="middle" border="0" alt="vmware"></a>
function domainForHostname(hostname) {
Vary: Host
Allow: GET

and the output:

http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

Edit: another input example

Host: mail.google.com
Host: mail.google.com
GET /mail/channel/bind?at=1a319b156176da12-1125900daa3&SID=6FA1DA0F17C9527D&RID=42925&TYPE=terminate&zx=ey4h3dgxde6e HTTP/1.1
Host: mail.google.com
GET / HTTP/1.1
Host: www.slashdot.org
GET / HTTP/1.1
Host: slashdot.org
GET /index.php?language=en&partner=%22%3E%3Cscript%3Ealert(123456789)%3C/script%3E  HTTP/1.1
Host: store.dakar.com
GET /google/ads/sidenav_sky.js?7927068257853024 HTTP/1.1
Host: genweb.ostg.com
GET /pagead/ads?client=ca-ostg_js&dt=1178313198977&adsafe=high&oe=utf8&num_ads=5&output=js&channel=slashdot_sidenav&url=http%3A%2F%2Fslashdot.org%2F&ad_type=text_image& image_size=125x600&region=apple%20askslashdot%20backslash%20books%20developers%20games%20hardware%20interviews%20it%20linux%20politics%20science%20yro%20default&feedback_link=on&loc=http%3A%2F%2Fslashdot.org%2F&cc=96&flash=7&u_h=1024&u_w=1280&u_ah=1024&u_aw=1280&u_cd=24&u_tz=-420&u_his=6&u_java=true&u_nplug=1&u_nmime=2 HTTP/1.1
Host: pagead2.googlesyndication.com
GET /adj/ostg.slashdot/mainpage_p1_leader;pg=index;logged_in=0;tile=1;ord=7927068257853024? HTTP/1.1
Host: ad.doubleclick.net
xhr_object.open("GET",filename,false );
GET /BurstingPipe/adServer.bs?cn=sb&c=17&pli=222855&pi=0&w=728&h=90&ncu=$$http: //ad.doubleclick.net/click%3Bh=v8/3548/3/0/%2a/k%3B99478854%3B0-0%3B0%3B13358359%3B255-0 /0%3B20810775/20828668/1%3B%3B%7Esscs%3D%3f$$&ord=5013023 HTTP/1.1
Host: bs.serving-sys.com
GET /BurstingRes//Site-454/Type-0/24E08935-2B74-491B-B910-A06A81D9FE97.jpg HTTP/1.1
Host: ds-ll.serving-sys.com
<li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>
GET /topics/topicdoj.gif HTTP/1.1
Host: images.slashdot.org
GET /medias/btnPanierFermer.gif HTTP/1.1
Host: store.dakar.com
GET /__utm.gif?utmwv=1&utmn=1735117121&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=7.0%20r25&utmdt=Slashdot%3A%20News%20for%20nerds%2C%20stuff%20that%20matters&utmhn=slashdot.org&utmr=-&utmp=/&utmac=UA-32013-5&utmcc=__utma%3D9273847.1199151991.1141182756.1178312303.1178313207.20%3B%2B__utmb%3D9273847%3B%2B__utmc%3D9273847%3B%2B__utmz%3D9273847.1163632638.13.3.utmccn%3D(organic)%7Cutmcsr%3Dgoogle%7Cutmctr%3DVM%2Bbased%2Brootkits%7Cutmcmd%3Dorganic%3B%2B HTTP/1.1
Host: www.google-analytics.com
GET /search?q=hotos HTTP/1.1
Host: www.google.com
GET /hotos05/ HTTP/1.1
Host: www.usenix.org
GET /7205/20070504/ HTTP/1.1
Host: www.thelocal.se
GET /events/hotos05/index.html HTTP/1.1
Host: www.usenix.org
GET /styles/core.css?new HTTP/1.1
Host: www.thelocal.se

4 Answers 4

9

The first example use grep to grab all the links like:

$ grep -o 'http[^"]*' file
http://www.dakar.com
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.ibm.com/
http://research.microsoft.com/
http://www.vmware.com/

For the second use awk to print the second field on lines where the first is Host::

$ awk '$1=="Host:"{print $2}' file
mail.google.com
mail.google.com
mail.google.com
www.slashdot.org
slashdot.org
store.dakar.com
genweb.ostg.com
pagead2.googlesyndication.com
ad.doubleclick.net
bs.serving-sys.com
ds-ll.serving-sys.com
images.slashdot.org
store.dakar.com
www.google-analytics.com
www.google.com
www.usenix.org
www.thelocal.se
www.usenix.org
www.thelocal.se
1
  • 1
    Much more elegant than my solution!
    – Journeyman Geek
    Apr 18, 2013 at 13:58
2

A simplistic approach using elinks(1) ability to dump files, as described in the manpage

   -dump [0|1] (default: 0)
       Print formatted plain-text versions of given URLs to stdout.

could be:

$ elinks -dump < infile | awk '$0~/\s*[[:digit:]]*\. http/ {print $2}'
http://www.dakar.com/
http://www.docomolabs-usa.com/
http://www.google.com/
http://www.hpl.hp.com/
http://www.hpl.hp.com/
http://research.microsoft.com/
http://www.vmware.com/

This could catch unwanted lines, of course. Improve the regex used to match your criteria.

Other text-mode browsers (lynx, links) and some pagers (w3m) also have a dump option.

1
  • what is elinks doing here? and how can i done it with out elinks?tnx
    – Arash
    Apr 18, 2013 at 10:15
2

grep with -o option extracts text of given pattern each per line. For example, the following command extracts all citations of the form \cite{citationKey} from a latex file.

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex

To redirect the output to another file, use

grep -o '[\]cite{[a-zA-Z0-9,-]*}' inputfile.tex > outputfile.tex
1

Assuming you want to extract this from an existing file (in this case called blag.text) you can use cat blag.txt| grep http |cut -d \" -f2 for the first example

First, extract lines containing http with grep. This gives you lines like <li><a href="http://www.dakar.com" TARGET=_BLANK>dakar.com</a></li>. We then use the quotation marks as delimiters for cut, but since quotation marks are also used to enclose strings, we need to escape it with /

For the second, you'd probably want to grep for "host" then use : as the deliminator (you can also use the space after the colon the same way) cat blag2.txt | grep Host |cut -d : -f2 is how I'd do it, though cat blag2.txt | grep Host |cut -d \ -f2 is more elegant. There's two spaces after / one the space we're using the deliminator, and one to seperate that from the next arguement.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .