Looked around for an hour but I'm not getting anyplace quickly.

I want to capture one web page weekly that has prices on it. The site is an auction site for livestock.

The twist here is that the target page name changes week to week, so, I need to be able to specify a LINK location on the home page, to get the page that I want.

I could write something in a day or two to screen scrape it, but there MUST be a program out there that does this already. I don't mind writing code, but I don't like to re-invent the wheel.

I just got HTTrack for other reasons, and if it will do it, just say so and I'll go figure it out. ( Literally just d/led it 30 minutes ago, so haven't looked at it at all yet, other than to to grab a website for a client. )

Looked at Selenium very quickly, but that appears to be overkill for what I want?

A bit more info: It's a link in, hopefully, the same spot on the homepage.
I'd like to save it with a date pre-pended to the local filename. I'd prefer to do this all on Windows as it's what I know the best. MUST be runable via scheduler in Win 7!

Eventually, I'd like to scrape the page to gather the prices and put them into excel, so if anyone has used a tool for this, that would be even better. (I still want to save the price page locally, as a backup to any screen scraping that I might do. )

  • does the nick change in any common/meaningful way? – Journeyman Geek Feb 13 '12 at 3:23

You could download the page with something like wget. Just put the command in a batch file and schedule the batch file to run once every week.

However, since you eventually want to parse the file, I would just look at ruby's open and write a script to save the page which you would later expand to do the parsing.

|improve this answer|||||

Because one of your requirements is to perform the task in Windows, and your link is easily defined and available from the homepage, I'd recommend using an AutoIT script to perform the function. I would specifically follow the examples listed for the IE User Defined Functions which has a healthy library of support functions to cut down on the programming burden for simple tasks.

|improve this answer|||||
  • I'll start digging through my old code library to pull some examples for you, and try and have them later in the week. – Simon Hova Feb 14 '12 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.