Synchronized script execution across multiple servers

Question

I'm looking to run an automated task (a shell script) on multiple Ubuntu servers at the same time with as much timing accuracy as possible. Ideally, I'd like the time discrepancy between the start of script executions to be less than a second across all of the servers. The servers are all in different locations and most are in different time zones. However, the tasks I intend to run will be at regular intervals, so time zones shouldn't be an issue.

I'm currently considering using cron to schedule the recurring task on each server, but I'm worried that it may not be accurate enough on its own. To compensate, I was considering using network time protocol to ensure that all servers have clocks that stay in sync (plus or minus time zones).

So my question is two part:

1. Will using cron and ntp be accurate enough to meet my needs?
2. Is there a better way to go about this?

Answer 1

It's difficult to answer accurately as there is different versions of cron with potentially varying accuracy.

NTP will definitively be a must for a solution like this, and use ntpd instead of ntpdate, as ntpd will constantly adjust the "length of a second" to the correct length of a second as opposed to ntpdate which just sets the time and allows it to drift again. Once a system has booted, it no longer uses the battery backed RTC clock, but will use CPU generated interrupts instead which are much more accurate, but might need to be adjusted slightly.

On to cron:

From the man page of cron:

cron then wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute. When executing commands, any output is mailed to the owner of the crontab (or to the user named in the MAILTO environment variable in the crontab, if such exists). The children copies of cron run‐ ning these processes have their name coerced to uppercase, as will be seen in the syslog and ps output.

Unless cron wakes up every minute on the minute, you have no gurantee for secondly accuracy, but I'd recommend not to take this for granted and put it to the test. Synchronize the servers using NTP, schedule a cron job to touch files over a period of time and stat them.

After you've touched enough files you should be able to get an idea of what kind of accuracy your cron has.

If your application absolutely has to synchronously within the second, or if your cron is too inaccurate, it will be better to start the scripts a few seconds early and using your NTP synchronized clock, have the scripts wait for the right moment in time to proceed with your request.

Edit: A quick bash script to sleep until the specified minute:

#!/bin/bash
minute_to_start=$1
minute_now=`date +%M`
minutes_to_wait=`echo $minute_to_start - $minute_now -1 | bc -l`
now=`date +%S.%N`
to_sleep=`echo 60 - $now + \( $minutes_to_wait \* 60 \) | bc -l`
sleep $to_sleep
$2

You can run it in cron like:

29 7 * * * /path/to/above/script.sh 30 'the real script'

And the job will start at 07:30:00

In my testing it drifted within 10-20 milliseconds, and it doesn't work if you want to start at X:00:00.00 (on the hour), so not something I'd actually use in production.

Answer 2

NTP is able is synchronize the time of your servers within a few microseconds. If the hardware clock is really bad you can toggle it hourly to prevent a bigger gap. cron is taking only hours and minutes as starting times, but will start the jobs at hh:mm:00, so from my point of view you are save with your solution.