I have to transfer large amounts of data (>80 GB) over ssh using rsync. Everything is working fine, but the DSL connection where the backup data is sent from will drop once every 24h for up to 3 minutes (switching providers is not an option).

How do I:

  1. Automatically restart the transfer when the connection is back up?

  2. Make sure there are not by accident two rsync commands running at the same time?

migrated from stackoverflow.com Jun 27 '11 at 14:23

This question came from our site for professional and enthusiast programmers.

  • Can't you check the return code? while ./run_script; do echo "Retrying..."; done; echo "Done." Make sure run_script returns 0 on success. – Kerrek SB Jun 27 '11 at 11:28
  • Possible duplicate of serverfault.com/q/98745. – tanius Jul 26 '15 at 7:28
  • Some useful info here - I just want to add that one way of getting round the repeated asking for password problem is to use the 'sshpass' command. Usually this needs to be installed with apt-get etc. – Rachael Saunders Oct 2 '17 at 9:30

The following should be helpful:

#!/bin/bash

while [ 1 ]
do
    rsync -avz --partial source dest
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "Rsync failure. Backing off and retrying..."
        sleep 180
    fi
done

When the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally.

  • 1
    Thanks, I'm trying this now... but should this: if [ "$?" = "0" ] not be: if [ "$?" == "0" ] (comparision operator)? – Andreas Jun 27 '11 at 12:55
  • 1
    No, in bash "=" is string equality (one of the many things that makes it confusing, I think!) – Peter Jun 27 '11 at 15:11
  • 6
    == is alias for = :D – bbaja42 Jun 27 '11 at 16:27
  • 8
    Ah, good to know. bash will never cease to amaze/horrify me :-P – Peter Jun 28 '11 at 23:32
  • 1
    Late to the party, however for posterity: A) just use: if rsync -avz --partial source dest; then ... B) if you want to compare integral values if use double parenthesis for arithmetic expansion: if (( $? = 0 )) then; – user18402 Mar 14 '14 at 4:44

This does much the same as Peter's answer, but gives the user the option of which remote file he wants, and where he wants to save it (as well as conducting the rsync over ssh). Replace the USER and HOST with your username and host respectively.

#!/bin/bash
echo -e "Please enter the full (escaped) file path:"
read -r path
echo "Path: $path"
echo -e "Enter the destination:"
read -r dst
echo "Destination: $dst"
while [ 1 ]
do
    rsync --progress --partial --append -vz -e ssh "USER@HOST:$path" $dst
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "rsync failure. Retrying in a minute..."
        sleep 60
    fi
done

The rsync options used here enable the progress stats during transfer, the saving of partial files upon unexpected failure, and the ability to append to partially completed files upon resume. The -v option increases verbosity, the -z option enables compression (good for a slow connection, but requires more cpu power on both ends), and the -e option enables us to conduct this transfer over ssh (encryption is always good).

Note: Use this only if you have public key login enabled with your ssh, otherwise it will ask you for a password when it restarts (killing all functionality of the script).

supervisor daemon (a process control manager) could work very well after creating rsa certificates of both sides, with a similar configuration as follows: (/etc/supervisor/ supervisord.conf is the configuration file path on debian based systems )

[program:rsync-remoteserver]
command=rsync -avz --progress root@server.com:/destination /backup-path
stdout_logfile=/out-log-path  
stderr_logfile=/errlogpath

The @Peter's answer seems be very useful, but for me it was important to use --update option. After connection has been resumed, without --update rsync was trying to sync all from the very begining. With --update, files that already exists are skipped.

rsync --partial --update --progress -r [SOURCE] [DESTINATION]

  • 1
    --update skips files that already exists... Including those that have not been completely copied to the target. I think it goes against most use cases. – durum Nov 22 '17 at 17:20

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.