I'm a social science researcher, and I'm working with data from various public databases of NGO, government, etc. Let's assume that I've got no opportunity to ask the admins for the whole database. However, if I have enough patience, I'm able to download all the data one-by-one. But the size of the DB makes it almost impossible solving the problem with brute-force.

So, is there any way to download a public DB with all of it's components?

Here's an example: http://www.trademap.org/tradestat/Country_SelProductCountry_TS.aspx

You can see the Japanese Live animal import (USD) by the importing countries. Is there a faster way to download all the data for every country and products as well than clicking them one-by-one?

Thank you in advance.

  • What is possible depends on the public API, since you can't access the database directly, and run a query on the data. – Ramhound Sep 9 '16 at 13:44

I think the best option you have is to download the Excel file it produces.

Then use that file and upload/insert to your database.

Another option would be screen scraping but probably not worth the hassle since they provide you with the Excel file any way


It looks like the site you mentioned uses an ASP.NET ValueState.

Check out this answer for more information on how to decrypt it.


  • Thank you! They don't mention: what should I do after I've decoded the ViewState? – Übel Yildmar Sep 9 '16 at 14:33

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.