Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking for a (linux) command line tool to parse HTML files and extract some elements, ideally with some XPath-like syntax.

I have the following requirements:

  • It must be able to parse arbitrary HTML files (which may contain errors) in a robust manner
  • It must be able to extract text of elements and attributes

What I have tried so far:

xmlstarlet: would be perfect, but mostly reports errors in files (e.g. entity not defined), even xml fo or htmltidy does not help.

xmllint: the best I have found so far, but is not able to extract attribute texts. Something like //a/@href reports <a href="foo">, what I need is just foo. string(//a/@href) works, but queries only the first entry. data is not supported.

hxextract: works, but cannot extract attributes.

XQilla: would support XPath 2.0 and thus data. It also support xqilla:parse-html, but I have had no luck making this work.

Can you recommend me another tool?

share|improve this question

migrated from stackoverflow.com Nov 18 '12 at 16:28

This question came from our site for professional and enthusiast programmers.

5 Answers 5

up vote 11 down vote accepted

My Xidel is probably the best for that, supporting XPath 2, XQuery, and CSS selectors.

e.g.:

 xidel http://www.example.org -e //a/@href
share|improve this answer
    
Great tool, thank you for sharing! –  ipsec Nov 19 '12 at 7:15
1  
Problems: not available by default and also not found by apt-get, at least on debian. –  sorin Feb 28 '13 at 9:34
    
@SorinSbarnea: well, it is new and unknown. You could just download the binary and put it on a usb stick –  BeniBela Feb 28 '13 at 13:08
    
+1 for example how to extract only link addresses. Same for OP. –  Smit Johnth Oct 15 at 15:52
    
Your xidel? Unfortunately I can't upvote twice... –  Smit Johnth Oct 15 at 18:26

You can use BaseX for that. To parse invalid XML you can use TagSoup, which is already packaged in the official BaseX distributions or can be easily added by adding it to the classpath. You can find out more in the documentation. BaseX offers you a command-line interface (but also a GUI or Client/Server) and you can use full XQuery 3.0 (so also XPath, of course)

share|improve this answer

Late to the party, but the xpath command-line tool from the Perl XML::Path library distribution does this. Not sure if it's restricted in what it can accept but my understanding is that XPath 1.0 should certainly be fine.

If you are not alien to writing a few lines of Python, lxml is excellent. There are also (somewhat more crude) XPath libraries in the Python standard distribution.

share|improve this answer

html xml utilities are yet another set of tools for the same. It can be found here, http://www.w3.org/Tools/HTML-XML-utils/README

share|improve this answer

There is pup utility, it supports only CSS selector syntax, but works with non well-formed HTML. It supports extraction of text and outputting of whole element subtrees.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.