Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking for a (linux) command line tool to parse HTML files and extract some elements, ideally with some XPath-like syntax.

I have the following requirements:

  • It must be able to parse arbitrary HTML files (which may contain errors) in a robust manner
  • It must be able to extract text of elements and attributes

What I have tried so far:

xmlstarlet: would be perfect, but mostly reports errors in files (e.g. entity not defined), even xml fo or htmltidy does not help.

xmllint: the best I have found so far, but is not able to extract attribute texts. Something like //a/@href reports <a href="foo">, what I need is just foo. string(//a/@href) works, but queries only the first entry. data is not supported.

hxextract: works, but cannot extract attributes.

XQilla: would support XPath 2.0 and thus data. It also support xqilla:parse-html, but I have had no luck making this work.

Can you recommend me another tool?

share|improve this question

migrated from stackoverflow.com Nov 18 '12 at 16:28

This question came from our site for professional and enthusiast programmers.

add comment

2 Answers 2

up vote 6 down vote accepted

My Xidel is probably the best for that, supporting XPath 2, XQuery, and CSS selectors.

e.g.:

 xidel http://www.example.org -e //a/@href
share|improve this answer
    
Great tool, thank you for sharing! –  ipsec Nov 19 '12 at 7:15
1  
Problems: not available by default and also not found by apt-get, at least on debian. –  sorin Feb 28 '13 at 9:34
    
@SorinSbarnea: well, it is new and unknown. You could just download the binary and put it on a usb stick –  BeniBela Feb 28 '13 at 13:08
add comment

You can use BaseX for that. To parse invalid XML you can use TagSoup, which is already packaged in the official BaseX distributions or can be easily added by adding it to the classpath. You can find out more in the documentation. BaseX offers you a command-line interface (but also a GUI or Client/Server) and you can use full XQuery 3.0 (so also XPath, of course)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.