0

I am transferring files over the network using rsync in cron job that runs every 1 one minute, task is whenever new file come it has to start uploading on remote server:

Since I am new so doing big mistake here:

* * * * * rsync -azP /local_path user@x.x.x.x:/remote_path > /dev/null

rsync create temp file (at remote end) and on next minute it creates temp file again (at remote end) after few minutes my networks starts getting choke and no upload bandwidth left, then I kill all rsync processes and copy at at a time.

Need this automated please help.

Thanks

  • Find out how long the rsync really takes, then schedule at least the quadruple of this time. Every minute is overdoing it - your computer will do nothing else but rsync. – harrymc Aug 15 '18 at 20:01
1

Looks like this is because the command takes longer than a minute to run, so the commands pile up until your system is starved of resources.

You can ensure that only one instance of that rsync runs by using a lock file. util-linux includes a file locking wrapper called flock, which you can use in your crontab like so:

* * * * * /usr/bin/flock -n '/tmp/example.lock' -c 'rsync -azP /local_path user@x.x.x.x:/remote_path' > /dev/null

In the above example, the lock file path is /tmp/example.lock, but you can set it to be anything sensible. -n will prevent flock commands from piling up because it'll exit immediately instead of wait for the lock to be released to continue.


Alternatively, you can also prevent your cron command from piling up by making a script with a fairly simple lock that ensures only one instance of the script can be running at a time:

#!/bin/bash

PIDFILE=/tmp/example.pid

if [ -f "$PIDFILE" ]
then
        PID="$(cat "$PIDFILE")"
        ps -p $PID > /dev/zero 2>&1
        if [ $? -eq 0 ]
        then
                echo "Instance of this script is still running as PID $PID"
                exit 1
        fi
fi

echo $$ > $PIDFILE
if [ $? -ne 0 ]
then
        echo "Could not create PID file: $PIDFILE"
        exit 1
fi

# YOUR CODE GOES BELOW
rsync -az /local_path user@x.x.x.x:/remote_path
# YOUR CODE GOES ABOVE

rm -f "$PIDFILE"

Locking mechanism inspired by Preventing duplicate cron job executions by Benjamin Cane

Note that the main limitation in the script is if the PID file isn't removed for whatever reason and another process has the same PID, the script will erroneously quit.

Save this script somewhere, like in /usr/local/bin/example.sh, chmod +x /usr/local/bin/example.sh, and then call it from your crontab like this:

* * * * * /usr/local/bin/example.sh > /dev/null
  • 1
    Dangerous script. If machine would forcibly stopped, or case with power outage then saved PID file will live forever preventing rsync start. Also, rm command in such script must be executed via trap to be make sure in case for example restart or shutdown event then PID file would be removed. Also, encourage students to do at least part of their works instead of giving them free solutions – Alex Aug 15 '18 at 17:26
0

First of all remove -P option, cron really won't to see progress.

You need to implement some locking mechanism, don't call rsync directly from cron, but make some script instead that would be used in cron. In the script, check first condition if rsync already running (grep ps output for example), if it running then simply exit script, otherwise run rsync.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.