up vote 0 down vote favorite
share [g+] share [fb]

For the purpose of a research-paper, I would like to extract specific information from online databases (in my case wikipedia, imdb and others), to look for patterns or tendencies. I was wondering if there are any tools out there to faciliate or automate this process:

  1. Open specific pages from a list of links, or follow certain links to the desired pages (maybe fill in forms to generate them)
  2. Scan those pages for certain elements (like the contents blocks, or the existance/count of certain words)
  3. Extracting those contents and storing them in a new, easier accessible structure...

I am quite a newcomer in the field of data-gathering and analysis, so I hope this question makes some sense.

link|improve this question
I don't think Wikipedia and IMDB TOS permit crawling via httrack, etc. – hyperslug Jan 8 '10 at 13:17
hi hyperslug. Thanks for the notice.. again - I was a bit naive. IMDB's terms of service say that a license or written consent is required to crawl their pages and copy part of their content. imdb.com/help/show_article?conditions and imdb.com/licensing – V. Schreiber Jan 10 '10 at 13:37
i read through the terms of use of IMDB again and decided to work with the downloadable plain text version of their database, which they provide via ftp. it may be used only for private and non-commercial purpose only, but at least i can use the info to prepare myself for the tests. so, since i'll write a small program to process the offline data myself, the question for web crawlers has become obsolete. thanks for your help and advice! – V. Schreiber Jan 22 '10 at 13:27
feedback

2 Answers

up vote 1 down vote

What you want is something like a web crawler. I haven't used any often enough to know of one that accomplishes exactly what you're looking for. You might have to join the crawler (also called spider) with a grep-style searching program to find the "certain elements". Those two tools together can be used to accomplish number 3. I would also recommend looking at sed and awk to process the information.

link|improve this answer
Thank you! I will take a look at them. – V. Schreiber Jan 9 '10 at 21:52
feedback
up vote 1 down vote

All sites have the information protected by international copyright laws. To copy them you have to ask the permission of the copyright holder. Exception to this is when the site has a license that allows copying of the information (like Wikipedia). To download Wikipedia database follow the following link: http://en.wikipedia.org/wiki/Wikipedia_database

link|improve this answer
Wow! I didn't think about that. Thanks for pointing it out to me. Just thought that if any person could read it online, it would also be allowed to have a machine read it (guess that was a little naive). But still, I'm not interested in copying entire sites, just scanning them for the existence of information (in case of imdb), and copying certain numbers (from wikipedia). – V. Schreiber Jan 9 '10 at 21:51
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.