Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to harvest some data like this http://www.newcardealers.ca/en/Dealers/List-A.aspx

And insert the name, address, phone number, email, etc. into a database. Is there some software I can use that will take a webpage, let me specify some regexes or something, and then spit out all the matched data in a CSV or some format easily insertable into a DB?

share|improve this question

2 Answers 2

FYI, mozenda works really well and is really easy to use... but they only have a free trial version, and their marketing strategy is really shady.

share|improve this answer
    
Well, harvesting addresses and contacts through crawling is a shady business. Real client-vendor relationships usually involve exchanges through private transactions and EDI. –  hyperslug Jul 17 '10 at 17:42
    
@hyperslug: Point taken. Harvesting or web-scraping might generally by shady, but it doesn't necessarily have to be. Same with torrents (aren't necessarily illegal). Thus, I don't think the software needs to portray itself as such! Anyway.. that's not really the point. Mozenda only works online, and it's not free. Very close to what I want, but not quite :) –  Mark Jul 19 '10 at 2:52

One way to do it is to custom write your own scraper. This

using HtmlAgilityPack;

...

HtmlDocument doc = new HtmlWeb().Load("http://www.newcardealers.ca/en/Dealers/List-A.aspx");
foreach (HtmlNode td in doc.DocumentNode.SelectNodes("//td[@width='268']"))
{
    Console.Write(td.SelectSingleNode("span[@class='BodyTextBold']").InnerText);
    Console.WriteLine(td.SelectSingleNode("span[@class='BodyText']").InnerText.Replace("\t\t\t\t", "\r\n"));
}

produces this

A.M. FORD SALES LIMITED
2795 Highway Drive
Trail, BC V1R2T1
Telephone : 250 364-0202                              http://www.amford.com               amford@amford.com         
ABBOTSFORD CHRYSLER LTD.
30285 Automall Drive
Abbotsford, BC V2T5M1
Telephone : 604 857-8888                              http://www.abbotsfordchrysler.com               paul@abbotsfordchrysler.com           
ACURA OF LANGLEY
19447 Langley ByPass
Surrey, BC V3S6K1
Telephone : 604 539-2111                              http://www.acuraoflangley.com               acura_info@acuraoflangley.com         
...

Not perfect, but you see where this is going.

Not sure how a standalone application would know how to parse that automatically.

share|improve this answer
    
I ended up doing pretty much exactly that actually... C#/HtmlAgilityPack. A clever standalone app could do it with just a few clicks tho (click the items you want to mine, and it'll compute the xpath statement that locates the others). –  Mark Jul 17 '10 at 6:49
    
Hm, I see. I was assuming you were looking for something fully automated, which I think would be very difficult. The semantic structure for the page you linked was very unintuitive, and unfortunately probably represents a typical case. Maybe with some training by the user, like selecting a pattern to match as you suggest it could be done. Still sounds tricky, code-wise, though I'm sure marketing depts would pay dearly for such an app... –  hyperslug Jul 17 '10 at 10:24
    
Well... Mozenda ran surprisingly well on that test page actually. It picked everything out perfectly, and even navigated to the the next page just fine, even though there was no "next" button, and some letters were skipped. Not saying it's too easy to program...but clearly it's doable. I'm not sure how the semantic structure is unintuitive... the data is laid out quite plainly (visually, anyway). If you're referring to lack of semantics in the code... well, that hardly matters! As long as you can pick out something that uniquely identifies the set. –  Mark Jul 19 '10 at 2:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.