Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We're creating a mashup site that pulls information from many sources all over the web. Many of these sites don't provide RSS feeds or APIs to access the information they provide. This leaves us with screen scraping as our method for collecting the data.

There are many scripting tools out there written in different scripting languages for screen scraping that require you to write scraping scripts in the language the scraper was written in. Scrapy, scrAPI, and scrubyt are a few written in Ruby and Python.

There are other web-based tools I've seen like Dapper that create XML or RSS feeds based on a webpage. It has a beautiful web-based interface that requires no scripting skills to use. This would be a great tool, if it were able to traverse multiple pages to gather data from hundreds pages of results.

We need something that will scrape information from paginated web sites, much like scrubyt, but with a user interface that a non-programmer could use. We'll script up our own solution if we need to, probably using scrubyt, but if there's a better solution out there, we want to use it. Does anything like this exist?

share|improve this question

Yahoo Pipes comes to mind, it's easy to use for a non programmer, although you should really learn regex to get it's full potential.

share|improve this answer
Are you sure you should use regex on html? (… ) – Michael Todd Dec 31 '09 at 17:48
Well a better option would be to use xpath (Y!P can use it via yql). Anyway having regex at hand is not bad at all. – solarc Dec 31 '09 at 18:20

Scrapinghub (from the creators of Scrapy) offers a pay-for service for non-programmers similar to Mozenda.

share|improve this answer

I've been using iMacros to scrape data from websites. It is usable by someone with no programming experience and with some basic programming skills you can greatly extend its capabilities. Here's a tutorial.

iMacros is particularly useful if you need to perform some action to retrieve the data. It can click on buttons, navigate through flash, select from menus, fill in forms etc.

There's also Scraperwiki, which requires programming skills. Non-programmers can pay for assistance.

Hopefully, Datatracker will be out soon. It's aimed at precisely this kind of job but for users without any programming skills.

share|improve this answer

This wikipedia article contains lots of information on the subject, including a list of 15 open-source web crawlers :

Web crawler

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.