Each day I need to copy N files from a source location to a mirror at a specific time (where N is very large). Let's say I tell multiple CPUs to each run an rsync simultaneously on a subset of the files (network and disk bandwidth are not an issue). Ideally each CPU would be responsible for a disjoint subset of the N files, but in practice this is sometimes hard to guarantee. (Some of the source files might be "claimed" by more than one CPU.) As a result, sometimes rsync I and rsync J will both try to copy file F at the same time.
rsync -avz --delete --temp-dir=/tmp remote:/path/to/source/ /path/to/dest/, let's say rsyncs I and J both see this situation to start:
/path/to/source/: FileA FileB FileC /path/to/dest/: FileA
Each rsync thinks it needs to copy files B and C, and each one starts doing so, first to
/tmp/name_of_source_file.temp_suffix. Let's say I finishes first and moves its temporary file to
/path/to/dest/FileB. Now the situation is:
/path/to/dest/: FileA FileB /tmp/: FileB.rsyncJsuffix
Now rsync J finishes copying but generates an error when it tries to move its version of FileB to
/path/to/dest/ because there's already another FileB there that it didn't see when it started.
Does one of rsync's many options somehow handle this situation? Ideally I'd like an option that tells rsync, "Believe in yourself. You can do no wrong. Feel free to overwrite anything your little heart desires." so that it wouldn't complain about the FileB that has suddenly appeared mid-execution.