4

I'm using the following command to copy a large number of big video files to external drives.

rsync -Ph --inplace /Volumes/Production/Prefix* Prefix

We had a power failure and the copying was abrupted. With rsync it's no problem to just restart it, but the thing is that it takes quite a while for it to get back to where it was. It goes through every file and looks like it reads through the whole file. Its speed is reported to be around 3-5 times faster than what it usually is, up to when it gets to the point it starts to copy again.

What is it doing exactly during this time? Is it reading through the whole file and comparing it with the source? Or is it doing something else fancy? Is there a way to get rsync to skip completed files faster? For example tell it to only check files that have a different file size or something?

4 Answers 4

0

The cause of your problem is not adding -t (or -a, which embeds -t and a bunch of other options) on the previous run of rsync.

-t is necessary in order to preserve the file modification times when copying, such that once copied, the destination file's last modified time will be the same as the source file's. Without this, subsequent runs of rsync will not consider any destination files as matching the source and will always update them.

Rsync's comparison behavior

By default, rsync will skip an existing destination file if its last modification time and file size match those of the source file.

Recovering from this

If you are confident no other software may have modified the destination files in the mean time and just want to resume, you can use the --size-only option. This modifies the comparison behavior such that rsync will skip an existing destination file if its file size matches, without checking the last modification time.

The problem you will have in future is that if you want to sync the directories again, you'll still need the file modification times to match, or you'll need to ignore modification times and trust the files haven't been modified by other software (in ways that don't change the size).

Explanation of options

  • Default rsync comparison: if both file size and last modification time match, skip.

  • --size-only: if the file sizes match, skip.

  • --ignore-times: never skip. File times and sizes are ignored.

  • --checksum: if the file sizes match, calculate a checksum of both files and if those match, skip.

The --size-only option makes skipping files more likely, while the --ignore-times and --checksum options make skipping files less likely.

5

rsync has an option: --size-only which does what you want.

2

Rsync checks against mod-time and size of files by default. If you add a --checksum, then it will compare against the whole file (ie: regardless of mod-time and size matching).

3
  • I guess in my case the mod-time would be wrong since I don't have the -t switch? Is there a way I can make it skip the mod-time check and only check the size?
    – Svish
    Dec 15, 2010 at 19:16
  • 2
    I think -t is default, "--size-only" will skip the mod-time check. Dec 15, 2010 at 19:19
  • This answer isn't wholly accurate. The "--checksum" option will compare by size and then by checksum if size is the same. Obvious if the size is different there is no need to calculate a checksum. By contrast the "--size-only" option will compare by size only and assume files are the same if sizes match, while the "--ignore-times" will do no comparisons at all always treat files as not matching (and therefore always copy) Aug 14, 2023 at 10:12
1

Since you didn't do -a for archive, it didn't copy over the times, so it thinks they are different files because of timestamp mismatch, and then it checksums the entire file.

If you're making copies where the time stamp could/should match: always include -a.

For now (although long ago) you can try adding --ignore-times

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .