15

I'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs.

for example:

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

I then verify those files are copied over... then the next time it runs it does the same thing

rsync -av /data/source/* user@host:/data/dest
sending incremental file list
source/file1.txt
source/file2.txt

any idea why it's getting stuck on these files? I've tried to wipe the whole dest directory out and start over but no luck.

thanks,

1
  • 1
    Have you done a diff on the files, and checked all the information on their ls -l list? They could, in principle, be modified at either edn without you being aware of it, and maybe rsync doesn't leave the files in the right state on the destination. Mar 3, 2010 at 13:51

2 Answers 2

11

Use --itemize-changes to get rsync to output what is actually being changed

Options that affect the decision for whether or not to skip files are:

  • -c which decides whether to skip identical files based on checksum
  • -I which ignores size and time when deciding whether to skip files.
  • -t which is used to preserve modification times (also turned on by -a flag). This option is required if you want to use modification times to decide whether or not to compare files based on modification time, and so it is required if you want to not evaluate files that have the same modification time (otherwise it will be as though you used -I).

Although rsync may be sending the files again, it shouldn't be transferring all the contents if they haven't changed - running with -v should print a summary of how much data was matched in the transfer.

For checking, the following should help:

  • md5sum of the files on either end - to show you if the contents have changed
  • ls -l should show you if the timestamps have changed.
2
  • 1
    But -t has anyway an effect on file transfer. From rsync man: Note that if this option (-t) is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing -t or -a will cause the next transfer to behave as if it used -I
    – Seki
    Apr 24, 2021 at 11:46
  • Good point, thanks Seki! I've updated the answer May 3, 2021 at 10:53
6

I remember a similar problem with two systems' clocks not quite behaving. I had to use --modify-window=60 to account for "temporal anomalies".

1
  • It helps also with --modify-window=1 when transfering on FAT filesystems as FAT stores the times with a 2-second resolution, so times can differ of +/- 1s.
    – Seki
    Apr 24, 2021 at 12:17

You must log in to answer this question.