Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to scrape an entire wiki that uses MediaWiki software. The amount of pages are pretty small, but they have plenty of revisions, and I'd like to preferably scrape revisions as well.

The wiki does not offer database dumps, unlike Wikipedia. Are there any existing software/scripts designed to scrape MediaWiki sites?

share|improve this question
    
Try wget, Perl, or Python. –  Joe Internet Feb 5 '12 at 10:02
2  
It's "scrape". I think Wikimedia would be most disappointed if you scrapped their site. –  Daniel R Hicks Feb 5 '12 at 13:18
    
@DanH I see what you did there. Clever sir. –  Raystafarian Feb 5 '12 at 13:50
add comment

1 Answer

up vote 3 down vote accepted

If the maintainer of the wiki hasn't turned it off, you can export pages with their history through Special:Export. This will give you an XML dump similar to Wikipedia's database dumps, which you can then import into another wiki.

Another way to obtain page history from MediaWiki in XML format is to use the prop=revisions API query. However, the API results format is somewhat different from that produced by Special:Export, so you'll probably have to process the output a bit before you can feed it to standard import scripts.

share|improve this answer
    
Thanks, but with Special Export I can only export one page at a time? –  gladoscc Feb 9 '12 at 4:37
    
No, you can export as many pages as you like; just separate the names with linefeeds. That's why the user interface has this whole big <textarea> just for the page list. –  Ilmari Karonen Feb 9 '12 at 13:22
    
However, see the footnotes on the page I linked to: if some of the pages have a lot of revisions, you may find that you need to export them one by one anyway. You could always write a simple script to loop over the page list and send export requests for them one at a time. –  Ilmari Karonen Feb 9 '12 at 13:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.