Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have to transfer large amounts of data (>80 GB) over ssh using rsync. Everything is working fine, but the DSL connection where the backup data is sent from will drop once every 24h for up to 3 minutes (switching providers is not an option).

How do I:

  1. Automatically restart the transfer when the connection is back up?

  2. Make sure there are not by accident two rsync commands running at the same time?

share|improve this question

migrated from stackoverflow.com Jun 27 '11 at 14:23

This question came from our site for professional and enthusiast programmers.

    
Can't you check the return code? while ./run_script; do echo "Retrying..."; done; echo "Done." Make sure run_script returns 0 on success. –  Kerrek SB Jun 27 '11 at 11:28
add comment

3 Answers

The following should be helpful:

#!/bin/bash

while [ 1 ]
do
    rsync -avz --partial source dest
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "Rsync failure. Backing off and retrying..."
        sleep 180
    fi
done

When the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally.

share|improve this answer
    
Thanks, I'm trying this now... but should this: if [ "$?" = "0" ] not be: if [ "$?" == "0" ] (comparision operator)? –  Andreas Jun 27 '11 at 12:55
1  
No, in bash "=" is string equality (one of the many things that makes it confusing, I think!) –  Peter Jun 27 '11 at 15:11
4  
== is alias for = :D –  bbaja42 Jun 27 '11 at 16:27
3  
Ah, good to know. bash will never cease to amaze/horrify me :-P –  Peter Jun 28 '11 at 23:32
    
Late to the party, however for posterity: A) just use: if rsync -avz --partial source dest; then ... B) if you want to compare integral values if use double parenthesis for arithmetic expansion: if (( $? = 0 )) then; –  user18402 Mar 14 at 4:44
add comment

supervisor daemon (a process control manager) could work very well after creating rsa certificates of both sides, with a similar configuration as follows: (/etc/supervisor/ supervisord.conf is the configuration file path on debian based systems )

[program:rsync-remoteserver]
command=rsync -avz --progress root@server.com:/destination /backup-path
stdout_logfile=/out-log-path  
stderr_logfile=/errlogpath
share|improve this answer
add comment

This does much the same as Peter's answer, but gives the user the option of which remote file he wants, and where he wants to save it (as well as conducting the rsync over ssh). Replace the USER and HOST with your username and host respectively.

#!/bin/bash
echo -e "Please enter the full (escaped) file path:"
read -r path
echo "Path: $path"
echo -e "Enter the destination:"
read -r dst
echo "Destination: $dst"
while [ 1 ]
do
    rsync --progress --partial --append -vz -e ssh "USER@HOST:$path" $dst
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "rsync failure. Retrying in a minute..."
        sleep 60
    fi
done

The rsync options used here enable the progress stats during transfer, the saving of partial files upon unexpected failure, and the ability to append to partially completed files upon resume. The -v option increases verbosity, the -z option enables compression (good for a slow connection, but requires more cpu power on both ends), and the -e option enables us to conduct this transfer over ssh (encryption is always good).

Note: Use this only if you have public key login enabled with your ssh, otherwise it will ask you for a password when it restarts (killing all functionality of the script).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.