2

I work on a research project that uses several databases provided by private companies. I need to get a huge amount of information from one of these databases to run a statistical test on the information. The problem is that the site is not very user friendly and only allow me to download 10 cases (one sheet) a time (there will be some 25 000 sheets to download...). Any suggestions?

This is what I tried so far: - DownThemAll (an application for Firefox web browser) allows me to get 10 sheets in one go. Which is great, but still far away from what I need. The problem here is that I would have to press a "next" button to see the number of the next sheets to download - and DownThemAll does not notice that it should press that button and the proceed to download the sheets listed on that page, and keep repeating the process.

Would there be a programing language I could use to do the following: do a for loop asking it to log on the database, get a a portion of the information, log out, then log on again, get a tibit more, log out, etc, until I get the whole info?

Update

Just writing to thank you all - I don´t have enough rep to vote your answers up - or just leave a comment thanking for your time.

Improve this question
1
  • I would say that every programming language can do that. I did something like this in Ruby, but you can also use Python, Java, etc. The tricky part is actually downloading something when you need to be logged on. It's however really really hard to help you if there are no concrete examples (I know you probably can't give out any).
    – slhck
    Apr 28, 2011 at 21:11

2 Answers 2

Reset to default
0

As slhck pointed, you can do that in pretty much any general purpose programming language.

Handling cookies and other browser specific behaviours can require a bit of extra work though.

There are libraries that try to emulate a browser to some extent, check mechanize http://wwwsearch.sourceforge.net/mechanize/ (python)

and curl http://curl.haxx.se/libcurl/ which have ports and bindings to a few languages AFAIK.

Of course, we're assuming that you can/should do that.

Improve this answer
0

You could try a recursive website downloader that supports login such as Offline Explorer Pro (not free). It lets you use its internal browser to login and will use the resulting cookie for crawling the site. Forum discussion on this.

If a free software exists that does this, I'd love to know.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .