Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to harvest some data like this

And insert the name, address, phone number, email, etc. into a database. Is there some software I can use that will take a webpage, let me specify some regexes or something, and then spit out all the matched data in a CSV or some format easily insertable into a DB?

share|improve this question
how about wget and grep eg, either in one step or wget redirect to a file then grep the file. – barlop Jun 27 at 9:49

FYI, mozenda works really well and is really easy to use... but they only have a free trial version, and their marketing strategy is really shady.

share|improve this answer
Well, harvesting addresses and contacts through crawling is a shady business. Real client-vendor relationships usually involve exchanges through private transactions and EDI. – hyperslug Jul 17 '10 at 17:42
@hyperslug: Point taken. Harvesting or web-scraping might generally by shady, but it doesn't necessarily have to be. Same with torrents (aren't necessarily illegal). Thus, I don't think the software needs to portray itself as such! Anyway.. that's not really the point. Mozenda only works online, and it's not free. Very close to what I want, but not quite :) – mpen Jul 19 '10 at 2:52

One way to do it is to custom write your own scraper. This

using HtmlAgilityPack;


HtmlDocument doc = new HtmlWeb().Load("");
foreach (HtmlNode td in doc.DocumentNode.SelectNodes("//td[@width='268']"))
    Console.WriteLine(td.SelectSingleNode("span[@class='BodyText']").InnerText.Replace("\t\t\t\t", "\r\n"));

produces this

2795 Highway Drive
Trail, BC V1R2T1
Telephone : 250 364-0202                                  
30285 Automall Drive
Abbotsford, BC V2T5M1
Telephone : 604 857-8888                                    
19447 Langley ByPass
Surrey, BC V3S6K1
Telephone : 604 539-2111                                  

Not perfect, but you see where this is going.

Not sure how a standalone application would know how to parse that automatically.

share|improve this answer
I ended up doing pretty much exactly that actually... C#/HtmlAgilityPack. A clever standalone app could do it with just a few clicks tho (click the items you want to mine, and it'll compute the xpath statement that locates the others). – mpen Jul 17 '10 at 6:49
Hm, I see. I was assuming you were looking for something fully automated, which I think would be very difficult. The semantic structure for the page you linked was very unintuitive, and unfortunately probably represents a typical case. Maybe with some training by the user, like selecting a pattern to match as you suggest it could be done. Still sounds tricky, code-wise, though I'm sure marketing depts would pay dearly for such an app... – hyperslug Jul 17 '10 at 10:24
Well... Mozenda ran surprisingly well on that test page actually. It picked everything out perfectly, and even navigated to the the next page just fine, even though there was no "next" button, and some letters were skipped. Not saying it's too easy to program...but clearly it's doable. I'm not sure how the semantic structure is unintuitive... the data is laid out quite plainly (visually, anyway). If you're referring to lack of semantics in the code... well, that hardly matters! As long as you can pick out something that uniquely identifies the set. – mpen Jul 19 '10 at 2:48
you could've written in your answer that the language you wrote in there was c# – barlop Jun 27 at 9:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .