Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to download two sites for inclusion on a CD:

http://boinc.berkeley.edu/trac/wiki
http://www.boinc-wiki.info

The problem I'm having is that these are both wikis. So when downloading with e.g.:

wget -r -k -np -nv -R jpg,jpeg,gif,png,tif http://www.boinc-wiki.info/

I do get a lot of files because it also follows links like ...?action=edit ...?action=diff&version=...

Does somebody know a way to get around this?

I just want the current pages, without images, and without diffs etc.

P.S.:

wget -r -k -np -nv -l 1 -R jpg,jpeg,png,gif,tif,pdf,ppt http://boinc.berkeley.edu/trac/wiki/TitleIndex

This worked for berkeley but boinc-wiki.info is still giving me trouble :/

P.P.S:

I got what appears to be the most relevant pages with:

wget -r -k -nv  -l 2 -R jpg,jpeg,png,gif,tif,pdf,ppt http://www.boinc-wiki.info
share|improve this question
    
No need to cross post between superuser and serverfault serverfault.com/questions/156045/… –  Bryan Jun 29 '10 at 22:07
    
Where should I have posted it? –  Tie-fighter Jun 29 '10 at 22:20
    
this is the right place. It's not a server question. –  David Z Jun 30 '10 at 0:42
    
Still I got the better answers at serverfault ;) –  Tie-fighter Jun 30 '10 at 0:56
add comment

4 Answers

The new version of wget (v.1.14) solves all these problems.

You have to use the new option --reject-regex=.... to handle query strings.

Note that I couldn't find the new manual that includes these new options, so you have to use the help command wget --help > help.txt

share|improve this answer
add comment
wget --reject-regex '(.*)\?(.*)' http://example.com

(--reject-type posix by default). Works only for recent (>=1.14) versions of wget though, according to other comments.

Beware that it seems you can use --reject-regex only once per wget call. That is, you have to use | in a single regex if you want to select on several regex :

wget --reject-regex 'expr1|expr2|…' http://example.com
share|improve this answer
    
Could be true about the version requirement. I had v1.12 and the option was not valid. After upgrade to v1.15 it was. –  yunzen Apr 4 at 12:41
add comment
wget -R "*?action=*"

This will exclude anything which contains ?action= in its name.

share|improve this answer
2  
"Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings." –  Tie-fighter Jun 29 '10 at 22:39
    
Hmm, I must have missed that. It looks like you can't do this with wget then if it doesn't even know that they are different files. I suggest a different program. –  Daisetsu Jul 1 '10 at 16:41
add comment

I'd say that leeching a public wiki site is bad practice, because it puts additional load on it.

If a wiki is public and the site owners don't mind sharing the content, they usually provide a downloadable backend (database or whatever) dump. So you would just download the data pack, set up a local instance of the same wiki engine, import the data into it and have a local copy. After that, if you wish, you can do the leeching locally.

share|improve this answer
    
there is -w seconds. e.g. -w 5. gnu.org/software/wget/manual/html_node/… –  barlop Sep 10 '13 at 23:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.