1

I want to harvest some data like this http://www.newcardealers.ca/en/Dealers/List-A.aspx

And insert the name, address, phone number, email, etc. into a database. Is there some software I can use that will take a webpage, let me specify some regexes or something, and then spit out all the matched data in a CSV or some format easily insertable into a DB?

Improve this question
1
  • how about wget and grep eg, either in one step or wget redirect to a file then grep the file.
    – barlop
    Jun 27, 2016 at 9:49

2 Answers 2

Reset to default
2

FYI, mozenda works really well and is really easy to use... but they only have a free trial version, and their marketing strategy is really shady.

Improve this answer
2
  • 1
    Well, harvesting addresses and contacts through crawling is a shady business. Real client-vendor relationships usually involve exchanges through private transactions and EDI.
    – hyperslug
    Jul 17, 2010 at 17:42
  • @hyperslug: Point taken. Harvesting or web-scraping might generally by shady, but it doesn't necessarily have to be. Same with torrents (aren't necessarily illegal). Thus, I don't think the software needs to portray itself as such! Anyway.. that's not really the point. Mozenda only works online, and it's not free. Very close to what I want, but not quite :)
    – mpen
    Jul 19, 2010 at 2:52
1

One way to do it is to custom write your own scraper. This

using HtmlAgilityPack;

...

HtmlDocument doc = new HtmlWeb().Load("http://www.newcardealers.ca/en/Dealers/List-A.aspx");
foreach (HtmlNode td in doc.DocumentNode.SelectNodes("//td[@width='268']"))
{
    Console.Write(td.SelectSingleNode("span[@class='BodyTextBold']").InnerText);
    Console.WriteLine(td.SelectSingleNode("span[@class='BodyText']").InnerText.Replace("\t\t\t\t", "\r\n"));
}

produces this

A.M. FORD SALES LIMITED
2795 Highway Drive
Trail, BC V1R2T1
Telephone : 250 364-0202                              http://www.amford.com               [email protected]         
ABBOTSFORD CHRYSLER LTD.
30285 Automall Drive
Abbotsford, BC V2T5M1
Telephone : 604 857-8888                              http://www.abbotsfordchrysler.com               [email protected]           
ACURA OF LANGLEY
19447 Langley ByPass
Surrey, BC V3S6K1
Telephone : 604 539-2111                              http://www.acuraoflangley.com               [email protected]         
...

Not perfect, but you see where this is going.

Not sure how a standalone application would know how to parse that automatically.

Improve this answer
4
  • I ended up doing pretty much exactly that actually... C#/HtmlAgilityPack. A clever standalone app could do it with just a few clicks tho (click the items you want to mine, and it'll compute the xpath statement that locates the others).
    – mpen
    Jul 17, 2010 at 6:49
  • Hm, I see. I was assuming you were looking for something fully automated, which I think would be very difficult. The semantic structure for the page you linked was very unintuitive, and unfortunately probably represents a typical case. Maybe with some training by the user, like selecting a pattern to match as you suggest it could be done. Still sounds tricky, code-wise, though I'm sure marketing depts would pay dearly for such an app...
    – hyperslug
    Jul 17, 2010 at 10:24
  • Well... Mozenda ran surprisingly well on that test page actually. It picked everything out perfectly, and even navigated to the the next page just fine, even though there was no "next" button, and some letters were skipped. Not saying it's too easy to program...but clearly it's doable. I'm not sure how the semantic structure is unintuitive... the data is laid out quite plainly (visually, anyway). If you're referring to lack of semantics in the code... well, that hardly matters! As long as you can pick out something that uniquely identifies the set.
    – mpen
    Jul 19, 2010 at 2:48
  • you could've written in your answer that the language you wrote in there was c#
    – barlop
    Jun 27, 2016 at 9:48

Not the answer you're looking for? Browse other questions tagged .