If I write wget "no such address" -o "test.html" it first creates the test.html and in case of failure, leaves it empty. However, when not using -o, it will wait to see if the download succeeds and only after that, it'll write the file.

I'd like the latter behavior to be applied to -o too, is it possible?

| |

wget returns a non-zero exit status when the URL is not found, so you can append a remove command on failure:

wget "url" -O file || rm -f file

Or create a temporary file and only move it where you want on success:

wget "url" -O /tmp/wget && mv /tmp/wget file

The second has the benefit of not deleting an existing file on failure, but be sure to use unique temporary names (see man tempfile) if you're running multiple instances in parallel.

| |
  • Also, adding --retry-connrefused can help in preventing the empty file in the first place. – akom Dec 29 '17 at 15:40
  • If this is happening in an exec in a puppet manifest, changing creates => file to unless => "[ -s file ]" can make it self-healing. – akom Dec 29 '17 at 15:41

As written in the comments, wget -O is more like a shell redirection which always writes into the file regardless of errors.

You can use curl -f instead:

curl -f http://nonexistent/file.jpg -o localfile.jpg

It will not touch the local file if there is an error fetching the file.

| |

The correct syntax is

wget "url" -O file

notice the UPPERCASE O. The -o options tells wget to write a log file, that's why it's always written even on failure.

| |
  • At first I thought it was working, but then I found it didn't. try wget "host.does.not.exist" -O "emptyFile" An error is returned, yet the emptyFile is created. – akurtser Jul 22 '10 at 9:38
  • 1
    @akurtser you're right. I think there's no way to tell wget not to create the file. I found this thread: mail-archive.com/wget@sunsite.dk/msg08586.html in which they discuss the matter. The baseline is that you can have MULTIPLE downloads to the same file so it gets created because wget cannot be shure that ALL urls will fail. – Mr Shunz Jul 22 '10 at 11:41
  • Well thanks, it's a part of a bash script I'm writing, so I'll just try to first save it a temp file, which in case of successful download, will be be renamed. Not very elegant, but can't think of anything better. – akurtser Jul 22 '10 at 12:19
  • 1
    @akurtser Surely you can check the return code from wget then... it should tell you if you can delete the file "if not found". So no need for temp/renaming. – Mr Shunz Jul 22 '10 at 12:42
  • 1
    The -O option is a redirection, which redirects the downloaded content to a file, even in cases when there are no contents. Therefore, a file is always created, even if the download failed. – Quan To Sep 15 '17 at 9:13

According to the help doc(wget -h), you can use --spider option to skip download(version 1.14).

  -S,  --server-response         print server response.
       --spider                  don't download anything.

| |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.