0

I would like to write a script that:

  • Takes some RSS-Feed URL as input
  • Downloads the feed
  • Deletes all <item> ...</item> occurrences where the title tag does not mach some regular expression.

The following example should illustrate this. Let's say we have an RSS Feed with these three items:

  • Project Foo - Let's get started!
  • Something Else Entirely
  • Another update on Project Foo

I want to keep only those items that have "Project Foo" in their title.

Example input file:

<?xml version="1.0" encoding="iso-8859-1"?>
<rss version="2.0">
<channel>
<title>My glorious newsfeed</title>
<description>...</description>
<link>...</link>
<language>...</language>
<pubDate>...</pubDate>

<item>
<title>Project Foo - Let's get started!</title>
<link>...</link>
<description>...</description>
<pubDate>...</pubDate>
</item>

<item>
<title>Something else entirely</title>
<link>...</link>
<description>...</description>
<pubDate>...</pubDate>
</item>

<item>
<title>Another update on Project Foo</title>
<link>...</link>
<description>...</description>
<pubDate>...</pubDate>
</item>

</channel>
</rss>

Example output file:

<?xml version="1.0" encoding="iso-8859-1"?>
<rss version="2.0">
<channel>
<title>My glorious newsfeed</title>
<description>...</description>
<link>...</link>
<language>...</language>
<pubDate>...</pubDate>

<item>
<title>Project Foo - Let's get started!</title>
<link>...</link>
<description>...</description>
<pubDate>...</pubDate>
</item>

<item>
<title>Another update on Project Foo</title>
<link>...</link>
<description>...</description>
<pubDate>...</pubDate>
</item>

</channel>
</rss>

If possible, I would like to stay away from the likes of python and do this with command-line tools. But I am a big newbie using sed etc. and need some help :)

Here's what I have so far:

cat sample-feed.xml \
  | tr -d '\n' \
  | sed $'s/\<item\>/\\\n\<item\>/g;s/\<\/channel\><\/rss\>/\\\n\<\/channel\><\/rss\>/g' \
  | sed '/^\<item\>/ d'

First, I delete all newlines. Then, I add newlines to bring every <item>...</item> onto its own line. The final command so far deletes all lines that start with <item>. For

The result is a valid rss-feed without any items:

<?xml version="1.0" encoding="iso-8859-1"?><rss version="2.0"><channel><title>My glorious newsfeed</title><description>...</description><link>...</link><language>...</language><pubDate>...</pubDate>    
</channel></rss>

For making this work with URL's instead of local files I would just replace the cat sample-feed.xml with curl -s <some url>.

What is still missing though is a modification to the command sed '/^\<item\>/ d' that only deletes lines that start with <item> but do not contain "Project Foo".

So, if you could help me figure out what the last line should say I would be very happy. On the other hand, I am sure there is a more elegant way to do this. From what I've seen sed is quite powerful and it should be possible to do this in one sed command.

Looking forward to your answers :-)

1

As suggested in the comments I tried using xmlstarlet to solve this and it works well. Here's my script

xml ed -d '//item[not(contains(title,"Project Foo"))]' < sample_rss.xml

Let's assume the feed content is in the file sample_rss.xml. That content gets fed into xml ed -d, which deletes any note matching the given XPath Expression. The XPath expression looks for any <item> that does not have a node <title> which contains the text "Project Foo".

This seems to work well and I am also very happy with the execution time:

real    0m0.003s
user    0m0.001s
sys     0m0.002s

Beware of namespaces

If you want to make this work with proper rss or atom feeds, you may notice that the feed contains an XML Namespace (xmlns) attribute, just like in this example from YouTube:

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015" xmlns:media="http://search.yahoo.com/mrss/" xmlns="http://www.w3.org/2005/Atom">
   ...
</feed>

Then, above script will no longer work! It caused me quite some headache to fix it, but here is how to make it work:

xml ed -d '//_:entry[not(contains(_:title,"Project Foo"))]' < youtube_rss.xml

More on this namespace problem here: http://xmlstar.sourceforge.net/doc/UG/ch05.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.