2

I have a bash script that launches a series of wget commands and log the output. I use the following options:

wget --no-verbose --page-requisites --adjust-extension --convert-links --backup-converted --timestamping --wait=1 --random-wait --append-output="$logfile" --recursive --level=2 http://example.com

The logs should end with:

Terminé — 2014-10-09 18:06:40 —
Temps total effectif : 1m 7s
Téléchargés : 11 fichiers, 79K en 0,06s (1,22 MB/s)

But sometimes there is no such summary, all I see are the errors that happened:

http://example.com/robots.txt:
2014-10-09 18:06:41 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/index.html:
2014-10-09 18:08:27 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/folder2/default.asp.html:
2014-10-09 18:08:31 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/index.html:
2014-10-09 18:08:56 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/folder2/folder3/"../images/bullet.gif":
2014-10-09 18:09:28 erreur 403 : Action interdite.

Why doesn't wget show a summary? Is it because no new files have been downloaded, or because there have been to many server errors (404, 403)?

  • If all you see are errors that happened, then I guess that means WGET wasn't able to fetch anything. And if doesn't download anything, you won't see the statistics at the end. – Vinayak Nov 21 '14 at 9:42
3
+100

Is it because no new files have been downloaded,[...]?

Yes. That's it. There will be no summary if zero bytes have been downloaded. It's because of the second half of the if-statement below: "total_downloaded_bytes != 0"

Source from the wget Git repo:

if ((opt.recursive || opt.page_requisites
       || nurl > 1
       || (opt.input_filename && total_downloaded_bytes != 0))
      &&
      total_downloaded_bytes != 0)

How to force wget to output a summary?

There seems to be no option to force this. I can think of these alternatives:

  • Remove the two lines from the source and recompile your own private build.
  • Submit a feature request for an explicit option to the wget bug tracker.
  • Download a 1 byte dummy file each time. This should force display of the summary.
| improve this answer | |
  • is it possible to download a dummy file in the same command as current recursive download ? – Manu Nov 25 '14 at 9:44
  • I've added a simple "tail | grep" on the last line of the log ; I then show the last 3 lines if files have been donwloaded, otherwise I echo the message "no new files downloaded". – Manu Nov 25 '14 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.