I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/).

The tool should "crawl" the sitemap regularly and look for new and deleted URLs and changes in the lastmod element. So whenever a page gets added/deleted/updated, the tool should apply the changes.

Some sitemaps list sub-sitemaps in sitemapindexsitemap. The tool should understand this and load all linked sub-sitemaps and look for URLs in there.

I know there are tools that allow me to extract all URLs from the sitemap, so that I could feed them to wget or similar tools (see for example: Extract Links from a sitemap(xml)). But this wouldn't help in getting noticed about updates to pages. Tracking the webpages itself for updates doesn't work, because "secondary" content on the pages changes daily, but lastmod gets only updated when relevant content changed.

closed as off-topic by and31415, random Dec 31 '14 at 16:31

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – and31415, random
If this question can be reworded to fit the rules in the help center, please edit the question.


Have you tried to script this with wget and cron? Look at wget's --spider flag. It looks to be all that you need, other than cron to run it occasionally.

Not the answer you're looking for? Browse other questions tagged or ask your own question.