Allowing multiple simultaneous rsyncs to play nice together

Question

Each day I need to copy N files from a source location to a mirror at a specific time (where N is very large). Let's say I tell multiple CPUs to each run an rsync simultaneously on a subset of the files (network and disk bandwidth are not an issue). Ideally each CPU would be responsible for a disjoint subset of the N files, but in practice this is sometimes hard to guarantee. (Some of the source files might be "claimed" by more than one CPU.) As a result, sometimes rsync I and rsync J will both try to copy file F at the same time.

Using rsync -avz --delete --temp-dir=/tmp remote:/path/to/source/ /path/to/dest/, let's say rsyncs I and J both see this situation to start:

/path/to/source/:
    FileA
    FileB
    FileC

/path/to/dest/:
    FileA

Each rsync thinks it needs to copy files B and C, and each one starts doing so, first to /tmp/name_of_source_file.temp_suffix. Let's say I finishes first and moves its temporary file to /path/to/dest/FileB. Now the situation is:

/path/to/dest/:
    FileA
    FileB

/tmp/:
    FileB.rsyncJsuffix

Now rsync J finishes copying but generates an error when it tries to move its version of FileB to /path/to/dest/ because there's already another FileB there that it didn't see when it started.

Does one of rsync's many options somehow handle this situation? Ideally I'd like an option that tells rsync, "Believe in yourself. You can do no wrong. Feel free to overwrite anything your little heart desires." so that it wouldn't complain about the FileB that has suddenly appeared mid-execution.

Thoughts?

Answer 1

I don't know why you are running rsyncs de way you are running them, but if I were you I'd seriously consider other ways to solve the problem that doesn't involve having multiple rsyncs writing to the same file tree at the same time.

This is from the rsync man page in the --temp-dir section:

If you are using this option for reasons other than a shortage of disk space, you may wish to combine it with the --delay- updates option, which will ensure that all copied files get put into subdirectories in the destination hierarchy, awaiting the end of the transfer. If you donât have enough room to duplicate all the arriving files on the destination partition, another way to tell rsync that you arenât overly concerned about disk space is to use the --partial-dir option with a relative path; because this tells rsync that it is OK to stash off a copy of a single file in a subdir in the destination hierarchy, rsync will use the partial-dir as a staging area to bring over the copied file, and then rename it into place from there. (Specifying a --par- tial-dir with an absolute path does not have this side-effect.)

Answer 2

Given you have some directory structure with some empty dirs, and some files and you want its archival copy -- what I would try is to run rsync with parallel:

1. recreate same directory structure

find /source/dir -type f|parallel mkdir -p dest/dir/{//}

2. rsync files:

find /source/dir -type f|parallel rsync -a {} /dest/dir/{}

3. then run one rsync to get empty dirs and make sure all is good

rsync -av /source/dir /dest/dir