I would like to use rsync
to download files from a server to my local computer. This is no problem of course:
$ rsync --progress -vr server-user@my-server:/path/to/remote/dir/ /local/dir/
receiving incremental file list
file1.txt
0 100% 10.00MB/s 0:01:00 (total: 0%) (xfer#1, to-check=n-1/n)
file2.txt
0 100% 10.00MB/s 0:01:00 (total: 1%) (xfer#2, to-check=n-2/n)
...
fileN.txt
0 100% 10.00MB/s 0:01:00 (total: 100%) (xfer#n, to-check=0/n)
While this works fine on the first download, I noticed, that when running the same rsync
command again, the existing files are re-downloaded again, even tough they have not changed:
$ rsync --progress -vr server-user@my-server:/path/to/remote/dir/ /local/dir/
receiving incremental file list
...
fileN.txt
0 100% 50.00MB/s 0:0:10 (total: 100%) (xfer#n, to-check=1/n+1) // <-- fast re-download
newFile.txt
0 100% 10.00MB/s 0:1:00 (total: 100%) (xfer#n, to-check=0/n+1) // <-- new file
While the re-download of existing files is much faster than the initial download of new files, it still takes some time. Definitly much more time than a simple comparison of timestamp and size would require. Additionally the output shows the download progress.
So, what exactly is going on here? I thought rsync
would automatically detect existing files by comparing timestamp and filesize (or using the file hash when -c
option is used) and transfer changed and new files only?
This is obviously not the case here, since re-running the command on a remote dir with several 1GB+ files takes several minutes. Thus something is transferred here, but what?
Adding the --ignore-existing
option stops the re-download of existing files and re-running the command only takes a few seconds. This shows, that comparing remote files to local files does not take very long. However, using this options does not solve the problem, because it does not recognize changed files any more...
Long story short:
How to use rsync to download only new and updated files?