Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Lets say I have a sitemap.xml file with this data:

<url>
<loc>http://domain.com/pag1</loc>
<lastmod>2012-08-25</lastmod>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>
<url>
<loc>http://domain.com/pag2</loc>
<lastmod>2012-08-25</lastmod>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>
<url>
<loc>http://domain.com/pag3</loc>
<lastmod>2012-08-25</lastmod>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>

I want to extract all the locations from it (data between <loc> and </loc>).

Sample output be like:

http://domain.com/pag1
http://domain.com/pag2
http://domain.com/pag3

How to do this?

share|improve this question
    
What OS are you using? –  bobmagoo Aug 27 '12 at 11:35
    
Windows 7 Ultimate X64 / Windows 8 Pro X64 or Ubuntu 12.04 Linux. –  Akshat Mittal Aug 27 '12 at 13:13
    
Nice setup. Using Terminal on the Ubuntu box, my answer below will get you what you need. –  bobmagoo Aug 27 '12 at 13:22
    
You can also use any text editor like SublimeText2 which can use regexp, you can get all data with it, or you can use python see my answer below. –  Ishikawa Yoshi Aug 27 '12 at 14:35

3 Answers 3

up vote 1 down vote accepted

You can use python script here

This script get any links started with http

import re

f = open('sitemap.xml','r')
res = f.readlines()
for d in res:
    data = re.findall('>(http:\/\/.+)<',d)
    for i in data:
        print i

And in your case next script find all data wraped in tags

import re

f = open('sitemap.xml','r')
res = f.readlines()
for d in res:
    data = re.findall('<loc>(http:\/\/.+)<\/loc>',d)
    for i in data:
        print i

Here nice tool to play with regexp if you not familiar with it.

if you need to load remote file you can use next code

import urllib2 as ur
import re

f = ur.urlopen(u'http://server.com/sitemap.xml')
res = f.readlines()
for d in res:
  data = re.findall('<loc>(http:\/\/.+)<\/loc>',d)
  for i in data:
    print i
share|improve this answer
    
How to load a remote file like http://server.com/sitemap.xml. I am not so known to Python –  Akshat Mittal Aug 28 '12 at 14:09
    
you mean load with python? –  Ishikawa Yoshi Aug 28 '12 at 14:14
    
Yup, Like you have used f = open('sitemap.xml','r') to open the file, How to open a remote file on http server? –  Akshat Mittal Aug 28 '12 at 14:16
    
i update my post, you need to use urllib2 module –  Ishikawa Yoshi Aug 28 '12 at 14:22
    
Shows error AttributeError: 'list' object has no attribute 'findall' –  Akshat Mittal Aug 28 '12 at 14:33

If you're on a Linux box or something with the grep tool, you can just run:

grep -Po 'http(s?)://[^ \"()\<>]*' sitemap.xml

share|improve this answer
    
This worked but with a lot of mistakes (Incomplete URL's). –  Akshat Mittal Aug 28 '12 at 13:46
    
Weird, I just ran this over Google's sitemap.xml file and didn't see any issues. Which ones did it miss? –  bobmagoo Aug 29 '12 at 17:46
    
This missed many url's that contained "?" and "+". –  Akshat Mittal Aug 30 '12 at 9:50
    
Thank you. For anybody wants to save to file grep -Po 'http(s?)://[^ \"()\<>]*' sitemap.xml > links.txt –  trante Sep 18 at 11:51

Using XSLT, you can render it out with XPath

/url/loc
share|improve this answer
3  
Could you maybe expand your answer and show the XSLT instructions and the XPath queries needed? –  slhck Aug 27 '12 at 14:44
    
@slhck Exactly what I wanted to say,The answer should be more explainatory. –  Akshat Mittal Aug 28 '12 at 9:28
    
I read a few more about this and got this working at last. Upvoting but not a really good answer to be choosen. –  Akshat Mittal Aug 28 '12 at 13:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.