Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

First, I'm no NAS-pro and I'm no Unix-pro. Second, I read the other RSS-downloader question, but rsstail is a C-source and tvrss is a GUI-program. I also checked out NetVibes, but I can't sort by title.

I bought a Zyxel NSA310 and thought it would be a good idea to have an RSS reader/downloader on it. There is a broadcatcher on it, but it doesn't store the RSS-items and I don't need that additional logic where downloads are initiated based on the RSS-items.

It should download the RSS-feeds and store the contents in readable files. When I get home from a trip I want to see all RSS-items of those feeds and not just the latest 30 or something. Since I use PuTTY to connect to my NAS I can't use any GUI-program. As far as the output is concerned I don't know what to look for, simple text would be enough though. To me it makes sense to sort the RSS-items by title and by date.

To let you know what's on the NAS:

root@NAS:~# uname -a
Linux NAS 2.6.31.8 #4 Fri Jul 20 18:10:42 CST 2012 armv5tel GNU/Linux
root@NAS:~# busybox
BusyBox v1.17.2 (2012-07-20 17:55:00 CST) multi-call binary.
Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko
and others. Licensed under GPLv2.
See source distribution for full notice.

Usage: busybox [function] [arguments]...
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.

Currently defined functions:
        [, [[, acpid, addgroup, adduser, adjtimex, arp, ash, awk, basename,
        beep, bzip2, cat, catv, chat, chgrp, chmod, chown, chroot, chrt, chvt,
        cksum, clear, cmp, cp, crond, crontab, cryptpw, cttyhack, cut, date,
        dd, deallocvt, delgroup, deluser, depmod, devmem, df, diff, dirname,
        dmesg, dnsdomainname, du, echo, ed, egrep, eject, env, expand, expr,
        false, fgrep, flock, free, freeramdisk, fsck, fsync, getopt, getty,
        grep, gunzip, gzip, halt, head, hexdump, hostname, hwclock, id,
        ifconfig, init, insmod, install, ionice, ip, ipaddr, ipcrm, ipcs,
        iplink, iproute, iprule, kill, killall, less, linuxrc, ln, logger,
        login, losetup, ls, lsmod, lspci, lsusb, lzop, lzopcat, man, microcom,
        mkdir, mke2fs, mkfs.ext2, mknod, mkpasswd, mkswap, mktemp, modinfo,
        modprobe, more, mountpoint, mv, netstat, nslookup, openvt, passwd,
        pidof, ping, ping6, pivot_root, poweroff, printenv, printf, ps, pwd,
        raidautorun, rdev, readahead, readlink, reboot, reset, resize, rev, rm,
        rmdir, rmmod, route, rtcwake, script, scriptreplay, sed, seq, setfont,
        setlogcons, setsid, sh, sha256sum, sha512sum, showkey, sleep, sort,
        split, start-stop-daemon, strings, su, swapoff, swapon, sync, tac,
        tail, tar, tee, telnet, telnetd, test, time, timeout, top, touch, tr,
        traceroute, traceroute6, true, tty, ttysize, ubiattach, ubidetach,
        udhcpc, udpsvd, uname, unexpand, uniq, unlzop, unxz, unzip, uptime,
        usleep, vi, volname, wall, wc, which, who, whoami, xargs, xz, xzcat,
        yes, zcat, zcip

I have no cc/gcc or make. I can't find a package installer neither.

Google found stuff for me coded in Python using feedreader, but those require distutils, which I don't have and I don't seem to be able to install that.

The only solution I can think of right now is using wget in connection with crontab. However, I'd have to do all the parsing and all in shell-script (since I don't know Python or Perl). That seems inconvenient.

What can I do/try? Any help is appreciated.

share|improve this question
    
as an update: I'm writing an RSS-reader with awk... Wish me luck ;) –  sjngm Mar 30 '13 at 18:43

1 Answer 1

up vote 1 down vote accepted

Here's what seems to work for me. I had this running with crontab for a while and it works for me. It supports RSS and Atom I think. However, if you intend to use it, consider it a starting point. If you adapt it and the change is useful, feel free to come back and tell the community.

The result is a directory structure with the feed name being the directory name and the various entries written as files. Those files contain the unchanged contents and at the bottom a link to the source. The filenames consist of the item title and its timestamp, the date of the file is set to that timestamp. That way you can sort them by name or time easily.

rss-parser.awk:

function removeFirstTag(str) {
    sub("[^>]*>", "", str)
    return str
}
function extractCdata(str) {
    gsub(".*<!\\[CDATA\\[|]]>.*", "", str)
    return str
}
function cleanup(str) {
    return extractCdata(removeFirstTag(str))
}

BEGIN {
    written = 0
    existed = 0
    if (feedTitle) {
        system("mkdir -p '" outDir "/" feedTitle "'")
    }
}

/<\/description>|<\/content>/ {
    indescr = 0
}
/<\/summary>/ {
    insummary = 0
}
/<\/item>|<\/entry>/ {
    gsub(/\//, "-", title)
    fname = outDir "/" feedTitle "/" title " - " dateSec
    fnameEnc = fname
    gsub(/\$/, "\\$", fnameEnc)
    gsub(/"/, "\\\"", fnameEnc)
    if (!description) {
        description = summary
    }
    if (link != "") {
        description = description "\nLink: " link
    } else if (guid != "") {
        description = description "\nLink: " guid
    }

    write = system(binPath "/stat \"" fnameEnc "\" -c \"%u\" 1> /dev/null 2>&1")
    if (write == 1) {
        print description > fname
        system(binPath "/touch \"" fnameEnc "\" -d \"" date "\"")
        written = written + 1
    } else {
        existed = existed + 1
    }
}

{
    if (indescr == 1) {
        description = description "\n" $0
    }
}

/<title[ >]/ {
    if (!feedTitle) {
        feedTitle = cleanup($0)
        system("mkdir -p '" outDir "/" feedTitle "'")
    } else {
        title = cleanup($0)
    }
}
/<item[ >]|<entry[ >]/ {
    title = ""
    description = ""
    summary = ""
    date = ""
    link = ""
    guid = ""
}
/<description[ >]|<content[ >]/ {
    indescr = 1
    description = cleanup($0)
}
/<summary[ >]/ {
    insummary = 1
    summary = cleanup($0)
}
/<link[ >]/ {
    link = cleanup($0)
    if (index(link, "magnet") == 1) {
        link = ""
    }
}
/<guid[ >]|<id[ >]/ {
    guid = cleanup($0)
}
/<pubDate[ >]|<published[ >]|<dc:date[ >]/ {
    date = cleanup($0)
    binPath "/date -d '" date "' +'%s'" | getline dateSec
}

END {
    print "Done : " feedTitle " (new: " written "/old: " existed ")"
    if (existed == 0 && written != 0) {
        print "=== NOTE === : All entries are new. You may want to check this feed more often"
    }
}

The file rss-parser.cfg consists of lines such as

http://thatonesite.com/feed-with-nice-title
http://thatothersite.com/feed=Specific feed title is better

rss-parser.sh:

#!/bin/sh

parse() {
    url="$1"
    name="$2"
    echo "Start: $name ($url)"
    wget -q -U "$userAgent" -O rss-news.xml $url
    sed -e 's/
//g' -e 's/<\([a-z/]\)/\n<\1/g' rss-news.xml | \
        $binPath/awk \
            -v "feedTitle=$name" \
            -v "binPath=$binPath" \
            -v "outDir=./data" \
            -f rss-parser.awk | \
        more
}

userAgent="Opera/9.80 (Windows NT 6.1; Win64; x64; U; de) Presto/2.10.289 Version/12.01"
binPath=/bin
cd `dirname $0`
if [ -z "$1" ]; then
    echo "================= `$binPath/date +"%d.%m.%Y %H:%M"`"
    start=`$binPath/date +%s`
    while read line
    do
        if [ ! -z "$line" -a "${line:0:1}" != "#" ]; then
            url=${line%%=*}
            name=${line#*=}
            if [ "$name" == "$url" ]; then
                name=""
            fi
            parse "$url" "$name"
        fi
    done < "rss-parser.cfg"
    echo "================= Took `$binPath/date -d "$start seconds ago" +%s` seconds"
else
    parse "$1" "$2"
fi

Without parameters it parses rss-parser.cfg. Otherwise parameters are

./rss-parser.sh [<URL> [<feed name>]]

I hope it's useful to anyone...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.