55

Let's say there's an url, let's call it http://www.some-url.com/folder/

This location has directory listing enabled, therefore I can do this:

wget -r -np http://www.some-url.com/folder/

To download all its contents with all the files and subfolders and their files.

Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?

2 Answers 2

70

did you read this ? http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html

"Several days later, you would like Wget to check if the remote file has changed, and download it if it has."

wget -N http://www.some-url.com/folder/

Try this.

3
  • 9
    But -N is useless if retrieved file has not Last-Modified header.
    – unibasil
    Commented Dec 29, 2017 at 4:27
  • @unibasil but how do you know, if the retrieved file has Last-Modified header?
    – jarno
    Commented Sep 14, 2020 at 6:48
  • Also if the download was interrupted, a wget -N will reuse the downloaded index files, and assume the previous job finished. Nuke the index.html*s first, then wget -N.
    – fche
    Commented Aug 12, 2023 at 10:22
22
wget -c 

also continues partial files caused when the download has issues.....
Better yet,

 wget -c -N 

seems to do both at the same time.

3
  • 1
    I found with my version 1.11.4 Red Hat modified, these flags are not compatible and -c overrides -N. Commented Sep 21, 2016 at 14:25
  • -c option of wget helped mw the sync partially downloaded file from url. Its really a good option to use if faced any network problems while downloading.
    – mchawre
    Commented Feb 15, 2020 at 16:43
  • 2
    I can confirm Matt Williamson's observation, using the -c flag nullifies the effect of -N, meaning newer files will not get downloaded.
    – Adrian
    Commented Dec 9, 2020 at 22:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .