Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

One of the forums that I frequent (and have added a LOT of quality content too) seems to be having problems with their server. I am not confident in their ability to sort out the problems they are having and in talking to one of the admins he mentioned that they don't back the data up.

As a complete fall back incase something goes horrifically wrong I want to download the entire forum. I am aware that I can't download the DB or the PHP files etc... I just want to make a locally browsable copy of the entire forum.

This means I could (when I have time) transfer the posts to the new site should they be starting fresh (on purpose or not).

Are there any tools that would allow this?

Side note: Obviously its really important I can browse it locally... which would be very difficult if each of the links still points to 'http://www.thesite.com/forum/specific_page.php' rather than '/forum/specific_page.php'.

share|improve this question
add comment

5 Answers

I am doing this right now. Here's the command I'm using:

wget -k -m -E -p -np -R memberlist.php*,faq.php*,viewtopic.php*p=*,posting.php*,search.php*,ucp.php*,viewonline.php*,*sid*,*view=print*,*start=0* -o log.txt http://www.example.com/forum/

I wanted to strip out those pesky session id things (sid=blahblahblah). They seem to get added automatically by the index page, and then get attached to all the links in a virus-like fashion. Except for one squirreled away somewhere - which links to a plain index.php which then continues with no sid= parameter. (Perhaps there's a way to force the recursive wget to start from index.php - I don't know).

I have also excluded some other pages that lead to a lot of cruft being saved. In particular memberlist.php and viewtopic.php where p= is specified can create thousands of files!

Due to this bug in wget http://savannah.gnu.org/bugs/?20808 it will still download an astounding number of those useless files - esepcially viewtopic.php?p= ones - before simply deleting them. So this is going to burn a lot of time and bandwidth.

share|improve this answer
    
In hindsight, I think that maybe a script that auto-incremented a wget to viewtopic?t=1 and viewforum?f=1 would work better. –  Andrew Russell Mar 5 '10 at 14:29
    
To give you an idea of how much of a problem that bug in wget is, about 92% of the HTML files that were downloaded were deleted. (Although note that far, far fewer files are downloaded than if you mirror with no reject list at all.) Also note that rejecting all these files makes links on the resulting pages virtually useless. You'd have to throw together a script to fixup links for you afterwards. –  Andrew Russell Mar 5 '10 at 14:42
add comment

Try some combination of wget flags like:

wget -m -k www.example.org/phpbb

Where -m is mirror, and -k is "convert links". You may also wish to add -p, to download images, as I can't recall whether -m does this.

share|improve this answer
    
Will this allow me to specify my username and password or a cookie somehow? The stuff I'm most interested in backing up is the stuff that is non-annon logins only. –  user28163 Mar 4 '10 at 18:53
    
Oh, of course. wget DOES have a --load-cookies argument, which apparently takes a filepath, but I have no idea how it works! –  Phoshi Mar 4 '10 at 20:01
    
@user28163; gnu.org/software/wget/manual/html_node/HTTP-Options.html has a section on --load-cookies which explains it better than I could. Sounds like you might be able to accomplish this! :) –  Phoshi Mar 4 '10 at 20:11
add comment

I recently faced a similar issue with a phpBB site I frequent facing imminent extinction (sadly, due to the admin passing away). With over 7 years of posts on the forum I didn't want to see it vanish, so I wrote a perl script to walk all the topics and save them to disk as flat HTML files. In case anyone else is facing a similar problem, the script is available here:

https://gist.github.com/2030469

It relies on a regex to extract the number of posts in a topic (needed to paginate) but other than that should generally work. Some of the regexes may need tweaking depending on your phpBB theme.

share|improve this answer
add comment

There are some Perl scripts for downloading a phpbb forum here:

http://www.derekcameron.com/phpbb-forum-download.html

share|improve this answer
    
Saddly the link is dead - any idea where the content might be? –  LordT Apr 7 at 0:15
add comment

HTTrack is a tool that might help you out. I am not sure if it will work on forums though.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.