Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have been trying to figure out how to resume partial transfers in rsync (on Windows, using cygwin) without using the --inplace option.

I see that rsync creates "partial" files in the destination directory with names like .[filename].2dxZ1a, and so, in my understanding, if I used the --partial option, rsync would see a file like this and resume the transfer to this file. But, I'm finding that if I use the --partial option and the rsync process ends, then when I start the rsync command again, it deletes those partial files. Perhaps this option on works on a brief transfer interruption, but not when the process ends and starts over.

In addition to this option, I actually liked the --partial-dir option, expecting that if I specifiy --partial-dir=".rsync" (with or without the surrounding quotes), rsync would create the directory .rsync in the destination directory and transfer the files to [dir]/.rsync, moving them to [dir] once the transfer finished (and deleting [dir]/.rsync once it no longer needed it). But, it seems like rsync is completely ignoring that option and just creating its temporary files in the destination directory like normal (and deleting them on re-start).

So, I'm wondering: am I doing something wrong or do I just not understand those options correctly?

My command is: rsync -rt --log-file="rsync.log" --partial-dir=".rsync" --del rsync://rsync-server:port/module/ /cygdrive/local/path/

share|improve this question

2 Answers 2

up vote 3 down vote accepted

I was launching this command from the Windows Task Scheduler. When I took the command and ran it from a console (command prompt), the options worked as they should*.

The only difference between the execution of the commands happened when I tried to end them. In the console, pressing Ctrl+C, rsync ended with the error

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at ...

In the Windows Task Scheduler, right-clicking the running task and selecting End resulted in the error

rsync: read error: Connection reset by peer (104)

So, I can only assume that the way the Windows Task Scheduler ends a process differs from the way rsync expects to end*, and this makes rsync end without doing its exiting routines.


* - I found more about the way --partial and --partial-dir work.

With --partial, rsync creates a temporary file like I saw, but when rsync ends (in an expected way), it renames that temporary file to the regular filename (e.g. renames .[filename].2dxZ1a to [filename]) and then makes a copy of the partial file to a new temp file on the next run (e.g. copies [filename] to .[filename].34cae2).

With --partial-dir, rsync fills the temporary file, then on close, moves the temporary file into the directory you specify as the regular filename (e.g. renames .[filename].2dxZ1a to [partial-dir]/[filename]). Then, on the next run, it copies the file from [partial-dir] to a new temporary file in the destination directory.

share|improve this answer
    
I think Thomas had some really helpful things to say, but this answered my question. So, before you upvote my answer, upvote Thomas! –  palswim Aug 27 '10 at 23:18

Seems like you have the right ideas, but without knowing your whole command it's impossible to tell. For me these options work like expected when I kill rsync (with ctrl-C) and restart it (GNU/Linux with rsync 3.0.6). One thing to keep is that if you use --wholefile or any option that implies --wholefile (see the rsync man page), then --partial(-dir) won't work and you might see the symptoms you describe. Also, if it cannot find or create the directory specified in --partial-dir, then it will silently ignore it. Another possibility is that --partial actually works for you, but you can't tell and think it starts over, even though it does not - I would expect it to delete the temporary files once it has completed the transfer and updated the actual file. You can monitor this using --progress - it will start at 0%, but then jump very quickly to whatever percentage was already transferred and continue from there. For smaller files the progress may be too quick to see much, so try with something larger (100 MB or more, depending on connection speed to the destination). Hope this helps. Edit: I found that --partial-dir seems to create a temporary file in the destination directory during the transfer, just as with --partial. Only when the transfer dies it moves that to the partial files directory you specified, so initially it seems like it's no different than --partial.

share|improve this answer
    
I know that --partial isn't working because I'm using the logging option, and after ending rsync and restarting the process, one of the first lines is always 2010/08/01 01:00:00 [4336] *deleting .[filename].444444 –  palswim Aug 24 '10 at 20:12
    
My best guess at this point is that you are triggering the --wholefile option (implicitly?). If rsync simply ignored --partial, it would have no reason to delete the temporary file ... Since this appears to be a problem on your system/with your specific command, rather than a general issue, it might be worthwhile running some tests with a minimal command (no other options) and maybe other (newer) versions of rsync. You could try rsync from a Linux Live CD and see if you get the same ... Are you using an rsync daemon on the destination? If so, it also needs to recognize all options. –  Thomas Aug 24 '10 at 21:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.