49

Let's say there's an url, let's call it http://www.some-url.com/folder/

This location has directory listing enabled, therefore I can do this:

wget -r -np http://www.some-url.com/folder/

To download all its contents with all the files and subfolders and their files.

Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?

64

did you read this ? http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html

"Several days later, you would like Wget to check if the remote file has changed, and download it if it has."

wget -N http://www.some-url.com/folder/

Try this.

2
  • 8
    But -N is useless if retrieved file has not Last-Modified header. – unibasil Dec 29 '17 at 4:27
  • @unibasil but how do you know, if the retrieved file has Last-Modified header? – jarno Sep 14 '20 at 6:48
20
wget -c 

also continues partial files caused when the download has issues.....
Better yet,

 wget -c -N 

seems to do both at the same time.

3
  • 1
    I found with my version 1.11.4 Red Hat modified, these flags are not compatible and -c overrides -N. – Matt Williamson Sep 21 '16 at 14:25
  • -c option of wget helped mw the sync partially downloaded file from url. Its really a good option to use if faced any network problems while downloading. – mchawre Feb 15 '20 at 16:43
  • 1
    I can confirm Matt Williamson's observation, using the -c flag nullifies the effect of -N, meaning newer files will not get downloaded. – Adrian Dec 9 '20 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.