Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a list of several hundred URL's, within one password protected account, and having the following format:

https://somesite.com/somescript.php?nameA
https://somesite.com/somescript.php?nameB
https://somesite.com/somescript.php?nameC
...

For each of those URL's, I would like to extract a string value from the corresponding webpage with such value always located between a specific prefix-string and a postfix-string, and creating a local CSV file with the data extracted from each URL:

nameA,valueA
nameB,valueB
nameC,valueC
...

Is there any simple way or script for doing this in Mac OS X?

I already tried the iMacros plugin for Firefox, but it seems it doesn't work, at least on Mac OS X. Maybe Automator, but I can't figure out how to do it...

share|improve this question
Where does one get valueA from? Is it also in the URL and did you forgot to mention it? Please note that this should be relatively to do with a text editor and a regular expression search & replace. I don't know about a browser plugin capable of doing this though. I suppose you have these URLs in a list that you can copy paste to a text editor? Or are they really separate and not copyable? Because then writing an userscript with GreaseMonkey might be necessary, or well, a macro plugin. – Tom Wijsman Jan 2 '12 at 23:01
valueA is not on the URL, it's on the content page that URL leads to. Yes I have a list of the URLs and/or names, that's no issue, and only the last parameter (name) changes. A userscript in GreaseMonkey might be a good idea (thanks!), though I'm not familiar with that. Right now, I was considering a bash script using cURL [curl.haxx.se/] to extract content page and sed to parse the required values from the content. Main issues are handling web login through HTTPS and also all the regular expression madness. – user111780 Jan 3 '12 at 1:57
As there are logins, you might me looking more towards a programming language and libraries capable of passing form data and cookies. And then screen-scrape all the data from the pages. I doubt if cURL supports login sessions... – Tom Wijsman Jan 3 '12 at 2:26

1 Answer

up vote 0 down vote

This should be relatively easy to do with iMacros. If all the pages are within the same password protected account you could just log in normally through the browser and then run your script. Even if they are not behind the same login and password, it is possible to automate the login process although that adds a level of complexity.

iMacros returns its results in a csv format. If it is difficult to get iMacros to select the precise HTML element you want, I'll often select a larger part of the page and then extract the precise string I need with a Mid() function in Excel. The standalone full version of iMacros has a few helpful features that are not present in the Firefox plugin (at least in the GUI). You can use the full program for free as a 30 day trial.

Alternatively you can use wget to download all the pages and then work with them locally. It can retrieve pages from a list of urls. Wget also allows logging in, although admittedly I haven't tried that. Once you have them local you can process them with iMacros or even a macro running text editor such as notepad++.

A more powerful tool would be Scraperwiki. That however requires some programming experience.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.