Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Each day I need to copy N files from a source location to a mirror at a specific time (where N is very large). Let's say I tell multiple CPUs to each run an rsync simultaneously on a subset of the files (network and disk bandwidth are not an issue). Ideally each CPU would be responsible for a disjoint subset of the N files, but in practice this is sometimes hard to guarantee. (Some of the source files might be "claimed" by more than one CPU.) As a result, sometimes rsync I and rsync J will both try to copy file F at the same time.

Using rsync -avz --delete --temp-dir=/tmp remote:/path/to/source/ /path/to/dest/, let's say rsyncs I and J both see this situation to start:

/path/to/source/:
    FileA
    FileB
    FileC

/path/to/dest/:
    FileA

Each rsync thinks it needs to copy files B and C, and each one starts doing so, first to /tmp/name_of_source_file.temp_suffix. Let's say I finishes first and moves its temporary file to /path/to/dest/FileB. Now the situation is:

/path/to/dest/:
    FileA
    FileB

/tmp/:
    FileB.rsyncJsuffix

Now rsync J finishes copying but generates an error when it tries to move its version of FileB to /path/to/dest/ because there's already another FileB there that it didn't see when it started.

Does one of rsync's many options somehow handle this situation? Ideally I'd like an option that tells rsync, "Believe in yourself. You can do no wrong. Feel free to overwrite anything your little heart desires." so that it wouldn't complain about the FileB that has suddenly appeared mid-execution.

Thoughts?

share|improve this question

migrated from stackoverflow.com Jun 2 '11 at 3:48

This question came from our site for professional and enthusiast programmers.

2 Answers 2

I don't know why you are running rsyncs de way you are running them, but if I were you I'd seriously consider other ways to solve the problem that doesn't involve having multiple rsyncs writing to the same file tree at the same time.

This is from the rsync man page in the --temp-dir section:

If you are using this option for reasons other than a shortage of disk space, you may wish to combine it with the --delay- updates option, which will ensure that all copied files get put into subdirectories in the destination hierarchy, awaiting the end of the transfer. If you donât have enough room to duplicate all the arriving files on the destination partition, another way to tell rsync that you arenât overly concerned about disk space is to use the --partial-dir option with a relative path; because this tells rsync that it is OK to stash off a copy of a single file in a subdir in the destination hierarchy, rsync will use the partial-dir as a staging area to bring over the copied file, and then rename it into place from there. (Specifying a --par- tial-dir with an absolute path does not have this side-effect.)

share|improve this answer
    
Thanks for the note. Unfortunately waiting until the completion of the rsync before moving the files to their final destination still doesn't resolve the problem, since it's two separate rsyncs that are competing, and one will finish first for each file. I do recognize that this is a strange use of rsync, but unfortunately necessary in the environment I'm using ... –  dg99 Jun 2 '11 at 14:15

Given you have some directory structure with some empty dirs, and some files and you want its archival copy -- what I would try is to run rsync with parallel:

  1. recreate same directory structure

find /source/dir -type f|parallel mkdir -p dest/dir/{//}

  1. rsync files:

find /source/dir -type f|parallel rsync -a {} /dest/dir/{}

  1. then run one rsync to get empty dirs and make sure all is good

rsync -av /source/dir /dest/dir

share|improve this answer
    
This proposes an alternate approach towards the goal, rather than answering the question asked. Thanks, anyway. –  dg99 Jan 10 at 0:29
    
"Does one of rsync's many options somehow handle this situation?" No to my knowledge ;) May be you are looking for some tool like moo.nac.uci.edu/~hjm/parsync –  dimus Jan 12 at 16:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.