up vote 1 down vote favorite
share [g+] share [fb]

I want to scrape an entire wiki that uses MediaWiki software. The amount of pages are pretty small, but they have plenty of revisions, and I'd like to preferably scrape revisions as well.

The wiki does not offer database dumps, unlike Wikipedia. Are there any existing software/scripts designed to scrape MediaWiki sites?

link|improve this question
50% accept rate
Try wget, Perl, or Python. – Joe Internet Feb 5 at 10:02
1  
It's "scrape". I think Wikimedia would be most disappointed if you scrapped their site. – DanH Feb 5 at 13:18
@DanH I see what you did there. Clever sir. – Raystafarian Feb 5 at 13:50
feedback

1 Answer

up vote 3 down vote accepted

If the maintainer of the wiki hasn't turned it off, you can export pages with their history through Special:Export. This will give you an XML dump similar to Wikipedia's database dumps, which you can then import into another wiki.

Another way to obtain page history from MediaWiki in XML format is to use the prop=revisions API query. However, the API results format is somewhat different from that produced by Special:Export, so you'll probably have to process the output a bit before you can feed it to standard import scripts.

link|improve this answer
Thanks, but with Special Export I can only export one page at a time? – gladoscc Feb 9 at 4:37
No, you can export as many pages as you like; just separate the names with linefeeds. That's why the user interface has this whole big <textarea> just for the page list. – Ilmari Karonen Feb 9 at 13:22
However, see the footnotes on the page I linked to: if some of the pages have a lot of revisions, you may find that you need to export them one by one anyway. You could always write a simple script to loop over the page list and send export requests for them one at a time. – Ilmari Karonen Feb 9 at 13:30
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.