2

I have a bash script that launches a series of wget commands and log the output. I use the following options:

wget --no-verbose --page-requisites --adjust-extension --convert-links --backup-converted --timestamping --wait=1 --random-wait --append-output="$logfile" --recursive --level=2 http://example.com

The logs should end with:

Terminé — 2014-10-09 18:06:40 —
Temps total effectif : 1m 7s
Téléchargés : 11 fichiers, 79K en 0,06s (1,22 MB/s)

But sometimes there is no such summary, all I see are the errors that happened:

http://example.com/robots.txt:
2014-10-09 18:06:41 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/index.html:
2014-10-09 18:08:27 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/folder2/default.asp.html:
2014-10-09 18:08:31 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/index.html:
2014-10-09 18:08:56 erreur 404 : Ce fichier n'existe pas (404).
http://example.com/folder1/folder2/folder3/"../images/bullet.gif":
2014-10-09 18:09:28 erreur 403 : Action interdite.

Why doesn't wget show a summary? Is it because no new files have been downloaded, or because there have been to many server errors (404, 403)?

1
  • If all you see are errors that happened, then I guess that means WGET wasn't able to fetch anything. And if doesn't download anything, you won't see the statistics at the end.
    – Vinayak
    Nov 21, 2014 at 9:42

1 Answer 1

3
+100

Is it because no new files have been downloaded,[...]?

Yes. That's it. There will be no summary if zero bytes have been downloaded. It's because of the second half of the if-statement below: "total_downloaded_bytes != 0"

Source from the wget Git repo:

if ((opt.recursive || opt.page_requisites
       || nurl > 1
       || (opt.input_filename && total_downloaded_bytes != 0))
      &&
      total_downloaded_bytes != 0)

How to force wget to output a summary?

There seems to be no option to force this. I can think of these alternatives:

  • Remove the two lines from the source and recompile your own private build.
  • Submit a feature request for an explicit option to the wget bug tracker.
  • Download a 1 byte dummy file each time. This should force display of the summary.
2
  • is it possible to download a dummy file in the same command as current recursive download ?
    – Manu
    Nov 25, 2014 at 9:44
  • I've added a simple "tail | grep" on the last line of the log ; I then show the last 3 lines if files have been donwloaded, otherwise I echo the message "no new files downloaded".
    – Manu
    Nov 25, 2014 at 10:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .