Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify the latitude/longitude of a point where you want the value. Instead of extracting values manually, I would like automate data extraction using the command line; either I'd write a script to pull out values for an input longitude/latitude, or pull out all values for all locations, as long as I can also get the latitude/longitude of the returned points. What utility could scrape the data from the atlas, and be part of a command line script? scrapy looks promising but maybe there are better tools for this. Or if you could tell me what language the 'Query' tool uses, it would help me get started.

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

This site heavily uses javascript (jQuery).

I suggest doing the following as a first start:

  1. Install Firefox
  2. Install the Tamper Data addon
  3. Load the site, start tampering
  4. Play around with the query tool and have a look on the XMLHttpRequests you automatically generate (see Tamper Data logs)

Perhaps this is already enough to identify the relevant requests and how they are created. If this doesn't work well you would have to read the javascript sources.

Since all requests are most probably just HTTP GET and HTTP POST requests with specific parameters, you could start automating e.g. with a tool like curl. Also, scrapy indeed looks promising and seems to bring a lot of nice features (didn't test it myself, though).


Instead of using Firefox with the Temper Data addon, you can also use any other browser while capturing the HTTP traffic e.g. with wireshark.

share|improve this answer
    
Thanks, this helps me get started. I appreciate it. –  KE. Aug 10 '12 at 14:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.