I work on a research project that uses several databases provided by private companies. I need to get a huge amount of information from one of these databases to run a statistical test on the information. The problem is that the site is not very user friendly and only allow me to download 10 cases (one sheet) a time (there will be some 25 000 sheets to download...). Any suggestions?
This is what I tried so far: - DownThemAll (an application for Firefox web browser) allows me to get 10 sheets in one go. Which is great, but still far away from what I need. The problem here is that I would have to press a "next" button to see the number of the next sheets to download - and DownThemAll does not notice that it should press that button and the proceed to download the sheets listed on that page, and keep repeating the process.
Would there be a programing language I could use to do the following: do a for loop asking it to log on the database, get a a portion of the information, log out, then log on again, get a tibit more, log out, etc, until I get the whole info?
Just writing to thank you all - I don´t have enough rep to vote your answers up - or just leave a comment thanking for your time.