up vote 1 down vote favorite
1
share [g+] share [fb]

I'm trying to download two sites for inclusion on a CD:

http://boinc.berkeley.edu/trac/wiki
http://www.boinc-wiki.info

The problem I'm having is that these are both wikis. So when downloading with e.g.:

wget -r -k -np -nv -R jpg,jpeg,gif,png,tif http://www.boinc-wiki.info/

I do get a lot of files because it also follows links like ...?action=edit ...?action=diff&version=...

Does somebody know a way to get around this?

I just want the current pages, without images, and without diffs etc.

P.S.: 

wget -r -k -np -nv -l 1 -R jpg,jpeg,png,gif,tif,pdf,ppt http://boinc.berkeley.edu/trac/wiki/TitleIndex

This worked for berkeley but boinc-wiki.info is still giving me trouble :/

P.P.S:

I got what appears to be the most relevant pages with:

wget -r -k -nv  -l 2 -R jpg,jpeg,png,gif,tif,pdf,ppt http://www.boinc-wiki.info
link|improve this question
No need to cross post between superuser and serverfault serverfault.com/questions/156045/… – Bryan Jun 29 '10 at 22:07
Where should I have posted it? – Tie-fighter Jun 29 '10 at 22:20
this is the right place. It's not a server question. – David Zaslavsky Jun 30 '10 at 0:42
Still I got the better answers at serverfault ;) – Tie-fighter Jun 30 '10 at 0:56
feedback

2 Answers

up vote 1 down vote

`wget -R "*?action="' This will exclude anything which contains ?action= in it's name.

link|improve this answer
1  
"Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings." – Tie-fighter Jun 29 '10 at 22:39
Hmm, I must have missed that. It looks like you can't do this with wget then if it doesn't even know that they are different files. I suggest a different program. – Daisetsu Jul 1 '10 at 16:41
feedback
up vote -1 down vote

I'd say that leeching a public wiki site is bad practice, because it puts additional load on it.

If a wiki is public and the site owners don't mind sharing the content, they usually provide a downloadable backend (database or whatever) dump. So you would just download the data pack, set up a local instance of the same wiki engine, import the data into it and have a local copy. After that, if you wish, you can do the leeching locally.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.