vote up 0 vote down star

For the purpose of a research-paper, I would like to extract specific information from online databases (im my case wikipedia, imdb and other), to look for patterns or tendencies. I was wondering if there are any tools out there to faciliate or automate this process:

  1. Open specific pages from a list of links, or follow certain links to the desired pages (maybe fill in forms to generate them)
  2. Scan those pages for certain elements (like the contents blocks, or the existance/count of certain words)
  3. Extracting those contents and storing them in a new, easier accessible structure...

I am quite a newcomer in the field of data-gathering and analysis, so I hope this question makes some sense.

Thank you!

flag
I don't think Wikipedia and IMDB TOS permit crawling via httrack, etc. – hyperslug Jan 8 at 13:17
hi hyperslug. Thanks for the notice.. again - I was a bit naive. IMDB's terms of service say that a license or written consent is required to crawl their pages and copy part of their content. imdb.com/help/show_article?conditions and imdb.com/licensing – V. Schreiber Jan 10 at 13:37
i read through the terms of use of IMDB again and decided to work with the downloadable plain text version of their database, which they provide via ftp. it may be used only for private and non-commercial purpose only, but at least i can use the info to prepare myself for the tests. so, since i'll write a small program to process the offline data myself, the question for web crawlers has become obsolete. thanks for your help and advice! – V. Schreiber Jan 22 at 13:27

2 Answers

vote up 1 vote down

What you want is something like a web crawler. I haven't used any often enough to know of one that accomplishes exactly what you're looking for. You might have to join the crawler (also called spider) with a grep-style searching program to find the "certain elements". Those two tools together can be used to accomplish number 3. I would also recommend looking at sed and awk to process the information.

link|flag
Thank you! I will take a look at them. – V. Schreiber Jan 9 at 21:52
vote up 1 vote down

All sites have the information protected by international copyright laws. To copy them you have to ask the permission of the copyright holder. Exception to this is when the site has a license that allows copying of the information (like Wikipedia). To download Wikipedia database follow the following link: http://en.wikipedia.org/wiki/Wikipedia_database

link|flag
Wow! I didn't think about that. Thanks for pointing it out to me. Just thought that if any person could read it online, it would also be allowed to have a machine read it (guess that was a little naive). But still, I'm not interested in copying entire sites, just scanning them for the existence of information (in case of imdb), and copying certain numbers (from wikipedia). – V. Schreiber Jan 9 at 21:51

Your Answer

Get an OpenID
or
never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.