How to force redownload with wget?

Question

I had the bad surprise that wget doesn't redownload when a file of the same name already exist.

Is there an option to force him to redownload without deleting the file first on linux ?

Answer 1

wget -O - $url >/path/to-file

Or, if you are mirroring, you can use the -c flag. This will make it so the files are "continued", but since they're already at the size the server is telling wget, it will "continue" them from the beginning.

Answer 2

It depends a lot on what options you've given wget on the commandline. I've found that the default behavior on my systems is to re-download, but append a number to the filename to make it different.

Here's the --no-clobber option description from man wget (for wget 1.11.4) -- note the interactions with other options, particularly -r and -p. (Emphasis mine.)

-nc
--no-clobber
If a file is downloaded more than once in the same directory, Wget’s behavior depends on a few options, including -nc. In certain cases, the local file will be clobbered, or overwritten, upon repeated download. In other cases it will be preserved.

When running Wget without -N, -nc, -r, or p, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named file.1. If that file is downloaded yet again, the third copy will be named file.2, and so on. When -nc is specified, this behavior is suppressed, and Wget will refuse to download newer copies of file. Therefore, ""no-clobber"" is actually a misnomer in this mode---it’s not clobbering that’s prevented (as the numeric suffixes were already preventing clobbering), but rather the multiple version saving that’s prevented.

When running Wget with -r or -p, but without -N or -nc, re-downloading a file will result in the new copy simply overwriting the old. Adding -nc will prevent this behavior, instead causing the original version to be preserved and any newer copies on the server to be ignored.

When running Wget with -N, with or without -r or -p, the decision as to whether or not to download a newer copy of a file depends on the local and remote timestamp and size of the file. -nc may not be specified at the same time as -N.

Note that when -nc is specified, files with the suffixes .html or .htm will be loaded from the local disk and parsed as if they had been retrieved from the Web.

Answer 3

This is controlled by the -nc or --no-clobber option.

It's complicated and depends on other options, but you have the choice of everwriting or renaming redownloaded files file.1, file.2

The man page seems to say that if you don't use the 'no-clobber' option, the redownloaded files will be renumbered. Do you see files with numeric suffixes?

In any case, see the man page