4

We're creating a mashup site that pulls information from many sources all over the web. Many of these sites don't provide RSS feeds or APIs to access the information they provide. This leaves us with screen scraping as our method for collecting the data.

There are many scripting tools out there written in different scripting languages for screen scraping that require you to write scraping scripts in the language the scraper was written in. Scrapy, scrAPI, and scrubyt are a few written in Ruby and Python.

There are other web-based tools I've seen like Dapper that create XML or RSS feeds based on a webpage. It has a beautiful web-based interface that requires no scripting skills to use. This would be a great tool, if it were able to traverse multiple pages to gather data from hundreds pages of results.

We need something that will scrape information from paginated web sites, much like scrubyt, but with a user interface that a non-programmer could use. We'll script up our own solution if we need to, probably using scrubyt, but if there's a better solution out there, we want to use it. Does anything like this exist?

4 Answers 4

2

Yahoo Pipes comes to mind, it's easy to use for a non programmer, although you should really learn regex to get it's full potential.

3
  • 4
    Are you sure you should use regex on html? ( stackoverflow.com/questions/1732348/… ) Dec 31, 2009 at 17:48
  • Well a better option would be to use xpath (Y!P can use it via yql). Anyway having regex at hand is not bad at all.
    – solarc
    Dec 31, 2009 at 18:20
  • Yahoo pipes is dead
    – Suncatcher
    Oct 23, 2023 at 5:00
1

This wikipedia article contains lots of information on the subject, including a list of 15 open-source web crawlers :

Web crawler

1

Scrapinghub (from the creators of Scrapy) offers a pay-for service for non-programmers similar to Mozenda.

1

I've been using iMacros to scrape data from websites. It is usable by someone with no programming experience and with some basic programming skills you can greatly extend its capabilities. Here's a tutorial.

iMacros is particularly useful if you need to perform some action to retrieve the data. It can click on buttons, navigate through flash, select from menus, fill in forms etc.

There's also Scraperwiki, which requires programming skills. Non-programmers can pay for assistance.

Hopefully, Datatracker will be out soon. It's aimed at precisely this kind of job but for users without any programming skills.

1
  • all the mentioned services are dead
    – Suncatcher
    Oct 23, 2023 at 5:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .