Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

wget normally stops when it gets a HTTP error, e.g. 404 or so. Is there an option to make wget to download the page content regardless of the HTTP code?

share|improve this question
4  
You can use curl instead. –  Daniel Beck Mar 6 '11 at 8:49
add comment

2 Answers

up vote 7 down vote accepted

Use curl when wget fails.

#!/bin/bash

if ! wget $1 
    then curl -R -O $1
fi

Usage (saved as wget.sh):

404 error will be saved with curl:

$ ./wget.sh http://www.google.com/asdasdasdasd.html

Working URL will be saved with wget:

$ ./wget.sh http://www.google.com/
share|improve this answer
2  
...but then why use wget at all? –  grawity Mar 6 '11 at 14:36
1  
It depends on what you want to do. Wget can download recursively, for instance. Read more: daniel.haxx.se/docs/curl-vs-wget.html –  Fábio Perez Mar 6 '11 at 14:39
    
But not when you give it one URL, as in your script and examples. And scripts wouldn't work with recursive wget. –  grawity Mar 6 '11 at 15:34
    
@grawity I created that script just for example purposes. –  Fábio Perez Mar 6 '11 at 15:48
add comment

Parameter: --content-on-error, available from wget 1.14:

If this is set to on, wget will not skip the content when the server responds with a http status code that indicates error.

share|improve this answer
    
Thanks! Just discovered that this option got added exactly four months after this question was asked :-) –  lilydjwg Sep 22 '13 at 8:19
    
@lilydjwg interestingly the option --content-on-error is not on here gnu.org/software/wget/manual/wget.html#Wgetrc-Commands - but is on man wget ftp.connect.org.uk/hpux/Gnu/wget-1.14/wget-1.14.man.html –  barlop Dec 17 '13 at 20:52
    
@barlop that's because the first link you linked is for wget 1.13.4, and the latter is for wget 1.14. --content-on-error was added at 1.14. –  lilydjwg Dec 18 '13 at 13:29
    
@lilydjwg well, if the online manual is going to be a man page, one would think they should update it to the latest man page. –  barlop Dec 18 '13 at 13:48
    
The linked manual is now the 1.15 one, which includes the mentioned option. –  Paŭlo Ebermann Apr 17 at 16:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.