0

I've got a directory that changes regularly, but whose files are related. I want to rsync that to a remote location, insuring that I always have one complete set of files at the remote. That is, if I have one complete backup, I want that preserved until I have a second complete one, in order to insure that any network interruption does not destroy the integrity of the first. If you think about it, --backup doesn't work, because it won't be clear if a not-backed-up file is part of the first complete backup, or the second incomplete one.

While what I want to do is scriptable, I'm thinking there's got to be a canned solution somewhere. I gather rdiff-backup would do it, rolling back partial backups, but I can't get that installed on the target machine. Does anyone have any other leads?

2
  • Is rsync literally the only thing you can have installed on the remote machine? There are various existing backup tools which can handle snapshots in their own format; some of them would be fine with standard SFTP access while others need their specific backend to be installed Nov 5, 2021 at 8:23
  • Ya, it's managed, so if I need to be able to install something over there, it won't work.
    – Diagon
    Nov 5, 2021 at 8:30

2 Answers 2

0

If you work with opened files like a database, a direct rsync can be a bad idea, since rsync won’t read atomically all the blocks of all the files of the database. The most atomic way would need a system with snapshots enabled. This needs a dedicated lvm partition, or a file system like btrfs or zfs. When the partition is snapshoted, your rsync can work freely on a stable set of files. The --delay-updates may be used to make remote application more atomic.

1
  • Ya, thanks Frederic. The files are stable while I'm rsync'ing them. I do have btrfs and snapshots, but they are actually not needed in this case. Here, it's the network going down as I'm pusing to the remote that I'm worried about.
    – Diagon
    Nov 7, 2021 at 11:41
0

Apparently the proper term for this is that I want a backup that is "atomic." The --delay-updates switch is termed in the man page, "a little more atomic," doing all the deletions and moves, right at the end. It worked for me, like this:

HOME="/home/me"
BACKUPFOLDER="$HOME/Backup/"
DIRECTORYLOG="$BACKUPFOLDER/<directory>.log"

NICERSYNC="nice -n 19 ionice -c 3 /usr/bin/rsync"
REMOTEHOST="[email protected]"
REMOTEBACKUPFOLDER="/home/me/Backup"

# partial-dir must be relative when using it with temp-dir
RSYNCSWITCHES="-acv --itemize-changes --delay-updates --delete-delay --partial-dir=<directory>-partial --temp-dir=$REMOTEBACKUPFOLDER/<directory>-temp"

$NICERSYNC $RSYNCSWITCHES "$BACKUPFOLDER/<directory>" $REMOTEHOST:\"$REMOTEBACKUPFOLDER\" &>> "$DIRECTORYLOG"

Though I don't really understand the details, when --partial-dir is used with --temp-dir in the context of --delay-updates, it needs to be given a relative path.

There is also a script, atomic-rsync in the /usr/share/rsync/scripts directory. Quoting from the man page:

See also the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is even more atomic (it uses --link-dest and a parallel hierarchy of files).

The problem for me is, that's only for pulling to a local directory, rather than pushing to a remote. I've scripted one that should do the job properly, below. After the first successful backup, <directory> will always exist on the remote as a complete backup. Interrupted backups will appear as <directory>.1, <directory>.2, etc. Any one that rsync's to completion, will be moved to <directory>, and the other <directory>.n will be deleted.

HOME="/home/me"
BACKUPFOLDER="$HOME/Backup/<directory>"
DIRECTORYLOG="$BACKUPFOLDER/<directory>.log"

LOCALFOLDER="/home/dev/Backup"
LOCALSUBFOLDER="Restic/<directory>"
LOCALSOURCE="$LOCALFOLDER/$LOCALSUBFOLDER"

REMOTEHOST="[email protected]"
REMOTEHOME="/home/me"
REMOTEFOLDER="Backup"
REMOTESUBFOLDER="<directory>"
REMOTETARGET="$REMOTEFOLDER/$REMOTESUBFOLDER"

NICERSYNC="nice -n 19 ionice -c 3 /usr/bin/rsync"
SWITCHES="-ac --partial-dir=\"$REMOTEHOME/$REMOTETARGET-partial\""

errormsgandexit() {
    if [[ $1 -ne 0 ]]; then
        echo "$2"
        exit $1
    fi
}

ssherrormsgandexit() {
    SSHCONNECTERROR=255
    if [[ $1 -eq $SSHCONNECTERROR ]]; then
        echo "$2"
        exit $1
    fi
}

echo -e "\n$(date) Rsync <directory>\n--------------------" >> "$DIRECTORYLOG"

# If TARGET exists, we use that for links: 
ssh $REMOTEHOST "[[ -d \"$REMOTETARGET\" ]]" && SWITCHLINKS="--link-dest=\"$REMOTEHOME/$REMOTETARGET\""
ssherrormsgandexit $? "ssh connect failure while checking $REMOTETARGET directory." >> "$DIRECTORYLOG" 

# Find the smallest n such that TARGET.n doesn't exist. use TARGET.1-(n-1) for links.
declare -i n=1
while ssh $REMOTEHOST "[[ -d \"$REMOTETARGET.$n\" ]]"; ! ((RETURN=$?)); do
    SWITCHLINKS="$SWITCHLINKS --link-dest=\"$REMOTEHOME/$REMOTETARGET.$((n++))\""
done
ssherrormsgandexit $RETURN "ssh connect failure while checking $REMOTETARGET.$n directory." >> "$DIRECTORYLOG" 

SWITCHESLINKS="$SWITCHES $SWITCHLINKS"

# now I am free to copy to TARGET.n:
$NICERSYNC $SWITCHESLINKS "$LOCALSOURCE/" $REMOTEHOST:\"$REMOTETARGET.$n\" >> "$DIRECTORYLOG"
errormsgandexit $? "rsync error, code $?." # >> "$DIRECTORYLOG" 

# if rsync was successful, just keep the most recent complete TARGET
ssh $REMOTEHOST "([[ ! -d \"$REMOTETARGET\" ]] || rm -r \"$REMOTETARGET\") && \
                mv \"$REMOTETARGET.$n\" \"$REMOTETARGET\" && \
                ([[ ! -d \"$REMOTETARGET.1\" ]] || rm -r \"$REMOTETARGET.\"*)" >> "$DIRECTORYLOG"
RETURN=$?
ssherrormsgandexit $RETURN "ssh connect failure while reordering $REMOTETARGET\* directories." >> "$DIRECTORYLOG"
errormsgandexit $RETURN "error when reorganizing $REMOTETARGET* directories." >> "$DIRECTORYLOG"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .