1

Using a cloud based storage medium (Strato's HiDrive product), I employ a backup scheme using rsync. During work time, most of my local server files get backed up in the cloud by a cronjob. My backup script makes heavily use of rsync's ability not to copy unchanged files, but to hard link them instead. This looks like

rsync -av -M--perms  \
            $inp_sig $REMOTE_USER@$RSYNC_HOST:$REMOTE_BASIS/$CURRENT_SNAPSHOT \
            --exclude-from=$EXCLUDE_FILES \
            --link-dest=$link_destination/

where all of the shown variables in advance are set to the correct values by the script. All of this works very well.

During the night, another script gets started that cleans out snapshots that are no longer used. For the deletion task, I use another script which at its heart has the following rsync command:

empty_dir=`mktemp -d`
rsync -d --delete-before --inplace --perms \
            $empty_dir/  $REMOTE_USER@$RSYNC_HOST:$REMOTE_BASIS/$snap/ || \
                    log_rsync ERROR: could not delete $snap
rm $empty_dir

This one works also - almost. Problem is that all permissions on all linked files get squashed by the deleting rsync (they all end up with 644 permissions where some of them should have 444).

I've been tackling the problem for several days now, tinkering with the options of the deleting rsync in different combinations. I've tried -rd vs. -r or -a; --delete vs. --delete-before, to no avail yet. From the manpage I learned that the option --inplace should do the trick:

This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file... Hard links are not broken.

This makes me think that not using --inplace should result in broken hard links. But in our case, hard links are broken anyway (for the files with differing permission bits).

They also have a cifs interface that is perfectly usable for a small number of file operations. But an rm -rf on a snapshot directory is by no means feasible - it lasts for hours on a typical backup snapshot, and I have to perform some 16 to 20 deletions per night. The rsync variant runs about two minutes.

So am I lost due to my limited understanding or implied limitations of rsync? Or am I just unlucky because of a poor implementation of rsync on the provider's side?

  • Wow, I just stumbled across the same problem. It doesn't happen often I find bugs in tools like rsync :-) – quazgar Jul 20 '18 at 18:28
1

Problem not completely solved, but worked around.

The problem I described in my post is due to the implementation of rsync. I filed a bug report to Bugzilla and got a clear indication that rsync indeed does not handle the permission bits correctly when recursively deleting directories.

They pointed out a workaround using the --super option:

empty_dir=`mktemp -d`
rsync -d --delete --super \
        $empty_dir/  $REMOTE_USER@$RSYNC_HOST:$REMOTE_BASIS/$snap/ || \
                log_rsync ERROR: could not delete $snap
rm $empty_dir

Yes, there could be side effects with the --super option. For my application, the workaround is sufficient.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.