Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

(see solution below)

I have XML files which I parse with a Python script (I did not write it but it does the job perfectly). The problem is that the XML file is large (~ 1GB) and the parsing take ages due to memory congestion. The XML file is full of useless information in certain elements - what would be the best way to get rid of them? I tried xmlstarlet but it is too "XML-oriented", ie. it takes ages for the same reasons that the Python script.

What I just need to do is to get rid of given elements in a dumb way: remove everything between <mytag> and </mytag> all through the file (there are multiple <mytag>...</mytag> pairs, all to be removed).

I would really appreciate your ideas since I am sure there are good ways to do that without reinventing the wheel.

Thank you!

EDIT: I finally ended up with

perl -pe "undef $/;s/<mytag>.*?<\/mytag>//msg" < inputfile.xml > outputfile.xml

which I did not realize @Vlad posted as well.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

When working with very large XML files, the recommended approach is to use a SAX event-driven parser. lxml can do that in Python, here's an excellent article on the topic: High-performance XML parsing in Python with lxml.

Another option would be to use something like sed to remove those tags from the file.

Or a Perl script:

perl -i.bak -pe 'BEGIN{undef $/;} s/<mytag>.*<\/mytag>//smg' file.xml
share|improve this answer
    
I tried to use sed (sed 's-<mytag>.*</mytag>--g' a.xml) but no luck so far –  WoJ Feb 16 '12 at 12:40
    
@WoJ sed is line-based. There are tricks to use multi-line patterns but they mostly involve concatenating all lines into one and applying substitution over it. In your case this would probably not be very efficient. See austinmatzko.com/2008/04/26/sed-multi-line-search-and-replace and stackoverflow.com/questions/5710424/sed-multiline-range-match for a Perl alternative. –  Vlad Feb 16 '12 at 12:49
    
... and thanks for the SAX reference –  WoJ Feb 16 '12 at 13:50
    
Sorry - I did not realize you posted the perl solution! This is what I also ended up with :) –  WoJ Feb 16 '12 at 13:54
    
@WoJ - I'm curios how it went performance-wise. Care to share the details: input file size, output file size, processing time, OS, CPU and memory? –  Vlad Feb 16 '12 at 14:26

Search and Replace with a text editor that can do it with wildcards? Preferably one that doesn't try to load the whole file on opening (or it will take ages). Most Hex-editors also have text search-replace capabilities.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.