Let's say there's an url, let's call it http://www.some-url.com/folder/

This location has directory listing enabled, therefore I can do this:

wget -r -np http://www.some-url.com/folder/

To download all its contents with all the files and subfolders and their files.

Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?

2 Answers 2


did you read this ? http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html

"Several days later, you would like Wget to check if the remote file has changed, and download it if it has."

wget -N http://www.some-url.com/folder/

Try this.

  • 9
    But -N is useless if retrieved file has not Last-Modified header.
    – unibasil
    Dec 29, 2017 at 4:27
  • @unibasil but how do you know, if the retrieved file has Last-Modified header?
    – jarno
    Sep 14, 2020 at 6:48
wget -c 

also continues partial files caused when the download has issues.....
Better yet,

 wget -c -N 

seems to do both at the same time.

  • 1
    I found with my version 1.11.4 Red Hat modified, these flags are not compatible and -c overrides -N. Sep 21, 2016 at 14:25
  • -c option of wget helped mw the sync partially downloaded file from url. Its really a good option to use if faced any network problems while downloading.
    – mchawre
    Feb 15, 2020 at 16:43
  • 1
    I can confirm Matt Williamson's observation, using the -c flag nullifies the effect of -N, meaning newer files will not get downloaded.
    – Adrian
    Dec 9, 2020 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.