Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I noticed that my server, an Ubuntu server 12.04, was losing time. I figured the hardware clock was off or maybe dying due to a faulty CMOS battery. I installed NTP to ensure the drift would be corrected, but to no avail. During a day it would lose 20 minutes or so.

To debug, I created a small cron job to check against a remote servers time, which I knew to be correct. The script calculates the difference in seconds between local and remote time. The NTP service was not running during these tests.

The result was interesting. It seems to be losing exactly 5 minutes several times during the day. Look at this log (difference from remote server noted in seconds):

Tue Oct 23 03:30:02 CEST 2012: 284
Tue Oct 23 03:35:02 CEST 2012: 284
Tue Oct 23 03:40:01 CEST 2012: 285
Tue Oct 23 03:45:02 CEST 2012: 285
Tue Oct 23 03:50:02 CEST 2012: 285
Tue Oct 23 03:55:02 CEST 2012: 284
Tue Oct 23 04:00:02 CEST 2012: 284
Tue Oct 23 04:05:01 CEST 2012: 285
Tue Oct 23 04:10:01 CEST 2012: 285
Tue Oct 23 04:15:02 CEST 2012: 585
Tue Oct 23 04:20:02 CEST 2012: 584
Tue Oct 23 04:25:02 CEST 2012: 584
Tue Oct 23 04:30:02 CEST 2012: 584
Tue Oct 23 04:35:01 CEST 2012: 585
Tue Oct 23 04:40:01 CEST 2012: 585
Tue Oct 23 04:45:02 CEST 2012: 585
Tue Oct 23 04:50:02 CEST 2012: 584
Tue Oct 23 04:55:02 CEST 2012: 584
Tue Oct 23 05:00:02 CEST 2012: 584
Tue Oct 23 05:05:01 CEST 2012: 585
Tue Oct 23 05:10:01 CEST 2012: 585
Tue Oct 23 05:15:02 CEST 2012: 585
Tue Oct 23 05:20:02 CEST 2012: 584
Tue Oct 23 05:25:02 CEST 2012: 584
Tue Oct 23 05:30:02 CEST 2012: 584
Tue Oct 23 05:35:01 CEST 2012: 585
Tue Oct 23 05:40:01 CEST 2012: 585
Tue Oct 23 05:45:02 CEST 2012: 584
Tue Oct 23 05:50:02 CEST 2012: 584
Tue Oct 23 05:55:02 CEST 2012: 584
Tue Oct 23 06:00:02 CEST 2012: 584
Tue Oct 23 06:05:03 CEST 2012: 584
Tue Oct 23 06:10:02 CEST 2012: 584
Tue Oct 23 06:15:01 CEST 2012: 585
Tue Oct 23 06:20:02 CEST 2012: 584
Tue Oct 23 06:25:02 CEST 2012: 584
Tue Oct 23 06:30:02 CEST 2012: 584
Tue Oct 23 06:35:02 CEST 2012: 584
Tue Oct 23 06:40:02 CEST 2012: 584
Tue Oct 23 06:45:01 CEST 2012: 585
Tue Oct 23 06:50:02 CEST 2012: 584
Tue Oct 23 06:55:01 CEST 2012: 585
Tue Oct 23 07:00:02 CEST 2012: 584
Tue Oct 23 07:05:02 CEST 2012: 584
Tue Oct 23 07:10:02 CEST 2012: 584
Tue Oct 23 07:15:02 CEST 2012: 584
Tue Oct 23 07:20:02 CEST 2012: 584
Tue Oct 23 07:25:02 CEST 2012: 584
Tue Oct 23 07:30:01 CEST 2012: 585
Tue Oct 23 07:35:02 CEST 2012: 584
Tue Oct 23 07:40:02 CEST 2012: 584
Tue Oct 23 07:45:02 CEST 2012: 584
Tue Oct 23 07:50:02 CEST 2012: 584
Tue Oct 23 07:55:02 CEST 2012: 584
Tue Oct 23 08:00:01 CEST 2012: 585
Tue Oct 23 08:05:02 CEST 2012: 584
Tue Oct 23 08:10:02 CEST 2012: 584
Tue Oct 23 08:15:02 CEST 2012: 584
Tue Oct 23 08:20:02 CEST 2012: 584
Tue Oct 23 08:25:02 CEST 2012: 584
Tue Oct 23 08:30:01 CEST 2012: 585
Tue Oct 23 08:35:02 CEST 2012: 584
Tue Oct 23 08:40:02 CEST 2012: 584
Tue Oct 23 08:45:02 CEST 2012: 584
Tue Oct 23 08:50:02 CEST 2012: 584
Tue Oct 23 08:55:02 CEST 2012: 584
Tue Oct 23 09:00:02 CEST 2012: 584
Tue Oct 23 09:05:03 CEST 2012: 584
Tue Oct 23 09:10:02 CEST 2012: 584
Tue Oct 23 09:15:02 CEST 2012: 584
Tue Oct 23 09:20:02 CEST 2012: 584
Tue Oct 23 09:25:02 CEST 2012: 584
Tue Oct 23 09:30:01 CEST 2012: 584
Tue Oct 23 09:35:02 CEST 2012: 584
Tue Oct 23 09:40:02 CEST 2012: 584
Tue Oct 23 09:45:02 CEST 2012: 584
Tue Oct 23 09:50:02 CEST 2012: 584
Tue Oct 23 09:55:02 CEST 2012: 584
Tue Oct 23 10:00:01 CEST 2012: 584
Tue Oct 23 10:05:02 CEST 2012: 584
Tue Oct 23 10:10:07 CEST 2012: 584
Tue Oct 23 10:15:02 CEST 2012: 584
Tue Oct 23 10:20:02 CEST 2012: 884
Tue Oct 23 10:25:02 CEST 2012: 884
Tue Oct 23 10:30:02 CEST 2012: 883
Tue Oct 23 10:35:01 CEST 2012: 884
Tue Oct 23 10:40:02 CEST 2012: 884
Tue Oct 23 10:45:02 CEST 2012: 884
Tue Oct 23 10:50:02 CEST 2012: 884
Tue Oct 23 10:55:02 CEST 2012: 1184
Tue Oct 23 11:00:02 CEST 2012: 1183
Tue Oct 23 11:05:01 CEST 2012: 1184
Tue Oct 23 11:10:02 CEST 2012: 1184
Tue Oct 23 11:15:02 CEST 2012: 1184
Tue Oct 23 11:20:02 CEST 2012: 1184

This does not seem to be faulty CMOS battery in my opinion. But what do you think?

EDIT:

When I enable NTP, this is the output of ntpq -p:

remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
dns02.wsrs.net  .INIT.          16 u    -   64    0    0.000    0.000   0.000
brick.steinhoff 71.40.128.157    3 u    1   64    1  144.031  1499785   0.002
chime6.surfnet. .PPS.            1 u    -   64    1   22.663  1499789   0.002
ntp0.mediamatic .INIT.          16 u    -   64    0    0.000    0.000   0.002
europium.canoni .INIT.          16 u    -   64    0    0.000    0.000   0.002

EDIT 2:

After ntpdate ntp.ubuntu.com

remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
kvm01.roethof.n 213.154.236.182  3 u   10   64    1   34.918   -1.980   0.002
ntp0.bbactive.e 193.79.237.14    2 u    9   64    1   58.378    6.956   0.002
16-164-ftth.ons 193.79.237.14    2 u    8   64    1   30.202    5.697   0.002
kameli.miuku.ne 62.237.86.238    2 u    7   64    1  106.975   -9.806   0.002
europium.canoni 193.79.237.14    2 u    6   64    1   42.010    6.381   0.002
share|improve this question
1  
But what does ntpq -p say and what NTP records are in syslog? AFAIK the CMOS clock is normally only consulted at boot. –  RedGrittyBrick Oct 23 '12 at 13:43
    
Well, I disabled NTP for the purpose of the test. I just tried enabling it and have updated my answer accordingly. –  Harold Smith Oct 23 '12 at 14:52
    
Well a) You have to correct the huge error in time (1499785 ms) before ntp can attempt to sync - try ntpdate first. b) you need to wait more than a few seconds. –  RedGrittyBrick Oct 23 '12 at 14:57
    
Well, as I've said, NTP has been running, and yet, sometimes it loses 300 seconds anyway. As this happens regularly, that will ruin NTP sync as well - right? –  Harold Smith Oct 23 '12 at 15:02
    
NTP operates not by making huge jumps but by slewing time (it can take 2000s to correct a 1s offset). Just because NTP is running doesn't mean NTP can instantly correct a huge time offset. Use ntpdate. NTP will usefully report timekeeping anomalies in syslog. –  RedGrittyBrick Oct 23 '12 at 15:15

1 Answer 1

up vote 2 down vote accepted

I figured the hardware clock was off or maybe dying due to a faulty CMOS battery.

The CMOS clock is only read when you boot the operating system.

I installed NTP to ensure the drift would be corrected, but to no avail.

NTP isn't intended to correct a huge offset. You can make that correction using ntpdate, then start ntpd and after a few hours check ntpq -p and make sure you understand what it is telling you. Then look in syslog for NTP complaints.

NTP has been running, and yet, sometimes it loses 300 seconds anyway

If so, I'd expect to see NTP comment on this in syslog - this will tell you exactly when the problem occurs. Perhaps this coincides with a cron job or some external event. I'd follow up such clues. I'm assuming you'd notice if a hardware fault was freezing the system for 5 minutes at a time.


Troubleshooting NTP

I'd read Troubleshooting NTP especially the "Known Issues" sections.


Monitoring Time

Harold already has a script, for others here's a couple of ideas:

On badserver enable RFC868 time protocol

vi /etc/xinetd.d/time-stream
# change `disable=yes` to `disable=no`
kill -HUP $(cat /var/run/xinetd.pid)

On goodserver use cron to schedule once a minute

date -u; TZ=utc rdate badserver > /tmp/badserver-time.log

Or better, write a script that only logs failures to respond or large changes in offset between goodserver and badserver.

share|improve this answer
    
I always ensure to run ntpdate before starting the NTP service, so I am sure the time is correct from the start. Whatever is going on here, it is not related to the NTP service, as it happens with or without the service running. I will keep an eye on the syslog, though. –  Harold Smith Oct 23 '12 at 15:27
    
NTP daemon is up, date is correct. I will look at syslog next time it falls behind. I am only working on the machine through SSH, and unfortunately only over wifi. Since the connection has been dropping from time to time, it could actually be happening simultaneously (in a system freeze). I will try to confirm this. –  Harold Smith Oct 23 '12 at 15:45
    
At 18:27 (UTC+2), my wifi SSH connection went out for exactly 300 seconds along with all other web services, and at 18:32 I was able to log in again. The system clock had not updated during the fallout. This is a much bigger problem, it seems... what could cause a 300 second fallout every 3 to 4 hours? –  Harold Smith Oct 23 '12 at 16:38
    
Oh yeah, and nothing was logged during the system breakdown either... –  Harold Smith Oct 23 '12 at 16:38
    
Wild wild guess: some unusual and flaky peripheral and a bad driver. I'd consider gradually stripping away non-essential peripherals and services until a culprit is found. Good luck. –  RedGrittyBrick Oct 23 '12 at 16:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.