Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

wget normally stops when it gets a HTTP error, e.g. 404 or so. Is there an option to make wget to download the page content regardless of the HTTP code?

share|improve this question
4  
You can use curl instead. – Daniel Beck Mar 6 '11 at 8:49
up vote 6 down vote accepted

Use curl when wget fails.

#!/bin/bash

if ! wget $1 
    then curl -R -O $1
fi

Usage (saved as wget.sh):

404 error will be saved with curl:

$ ./wget.sh http://www.google.com/asdasdasdasd.html

Working URL will be saved with wget:

$ ./wget.sh http://www.google.com/
share|improve this answer
2  
...but then why use wget at all? – grawity Mar 6 '11 at 14:36
1  
It depends on what you want to do. Wget can download recursively, for instance. Read more: daniel.haxx.se/docs/curl-vs-wget.html – Fábio Perez Mar 6 '11 at 14:39
    
But not when you give it one URL, as in your script and examples. And scripts wouldn't work with recursive wget. – grawity Mar 6 '11 at 15:34
    
@grawity I created that script just for example purposes. – Fábio Perez Mar 6 '11 at 15:48

Parameter: --content-on-error, available from wget 1.14:

If this is set to on, wget will not skip the content when the server responds with a http status code that indicates error.

share|improve this answer
1  
Thanks! Just discovered that this option got added exactly four months after this question was asked :-) – lilydjwg Sep 22 '13 at 8:19
    
@lilydjwg interestingly the option --content-on-error is not on here gnu.org/software/wget/manual/wget.html#Wgetrc-Commands - but is on man wget ftp.connect.org.uk/hpux/Gnu/wget-1.14/wget-1.14.man.html – barlop Dec 17 '13 at 20:52
    
@barlop that's because the first link you linked is for wget 1.13.4, and the latter is for wget 1.14. --content-on-error was added at 1.14. – lilydjwg Dec 18 '13 at 13:29
    
@lilydjwg well, if the online manual is going to be a man page, one would think they should update it to the latest man page. – barlop Dec 18 '13 at 13:48
    
The linked manual is now the 1.15 one, which includes the mentioned option. – Paŭlo Ebermann Apr 17 '14 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .