7

I want to scrape an entire wiki that uses MediaWiki software. The amount of pages are pretty small, but they have plenty of revisions, and I'd like to preferably scrape revisions as well.

The wiki does not offer database dumps, unlike Wikipedia. Are there any existing software/scripts designed to scrape MediaWiki sites?

3
  • Try wget, Perl, or Python. Feb 5, 2012 at 10:02
  • 3
    It's "scrape". I think Wikimedia would be most disappointed if you scrapped their site. Feb 5, 2012 at 13:18
  • @DanH I see what you did there. Clever sir. Feb 5, 2012 at 13:50

2 Answers 2

3

If the maintainer of the wiki hasn't turned it off, you can export pages with their history through Special:Export. This will give you an XML dump similar to Wikipedia's database dumps, which you can then import into another wiki.

Another way to obtain page history from MediaWiki in XML format is to use the prop=revisions API query. However, the API results format is somewhat different from that produced by Special:Export, so you'll probably have to process the output a bit before you can feed it to standard import scripts.

3
  • Thanks, but with Special Export I can only export one page at a time?
    – apscience
    Feb 9, 2012 at 4:37
  • No, you can export as many pages as you like; just separate the names with linefeeds. That's why the user interface has this whole big <textarea> just for the page list. Feb 9, 2012 at 13:22
  • However, see the footnotes on the page I linked to: if some of the pages have a lot of revisions, you may find that you need to export them one by one anyway. You could always write a simple script to loop over the page list and send export requests for them one at a time. Feb 9, 2012 at 13:30
6

Check out the tools available at from WikiTeam. http://archiveteam.org/index.php?title=WikiTeam

I personally use wikiteam's dumpgenerator.py which is available here: https://github.com/WikiTeam/wikiteam

It depends on python 2. You can get the software using git or download the zip from github:

git clone https://github.com/WikiTeam/wikiteam.git

The basic usage is:

python dumpgenerator.py http://wiki.domain.org --xml --images
3
  • 2
    Welcome to Super User! Can you add in the relevant parts of the link into your answer? We ask this to help the OP out, so they will not have to search through information that may not pertain to them. This is also to preserve the relevant information in case the hosting site goes down. For more information, see this meta post.
    – Cfinley
    Aug 5, 2015 at 19:48
  • FYI: Unfortunately that dumpgenerator is written in Python 2 :-(
    – Regis May
    Nov 13, 2018 at 10:01
  • They already have a lot of wikis archived here, so it may not even be necessary to do it yourself.
    – GDP2
    Aug 15, 2019 at 16:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.