I've noticed that on my servers and other machines the clock always drifts so it has to sync up to remain accurate.
How do the NTP server clocks not drift and always remain correct?
NTP servers rely on highly accurate clocks for precision timekeeping. A common time source for central NTP servers is atomic clocks, or GPS receivers (remember that GPS satellites have atomic clocks onboard). These clocks are defined as accurate since they provide a highly exact time reference. There's nothing magical about GPS or atomic clocks that make them tell you exactly what time it is; because of how atomic clocks work, they are simply very good at, having once been told what time it is, keep telling accurate time (since the second is defined in terms of atomic effects). In fact, it's worth noting that GPS time is distinct from the UTC that we are more used to seeing. These atomic clocks in turn are synchronized against International Atomic Time or TAI in order to not only accurately tell the passage of time but also the time.
Once you have an exact time on one system connected to a network like the Internet, it's a matter of protocol engineering enabling transfer of precise times between hosts over an unreliable network. In this regard a stratum 2 (or farther from the actual time source) NTP server is no different from your desktop system syncing against a set of NTP servers.
By the time you have a few accurate times (as obtained from NTP servers or elsewhere) and know the rate of advancement of your local clock (which is easy to determine), you can calculate your local clock's drift rate relative to the "believed accurate" passage of time. Once locked in, this value can then be used to continuously adjust the local clock to make it report values very close to the accurate passage of time, even if the local real-time clock itself is highly inaccurate; as long as your local clock is not highly erratic, this should allow keeping accurate time for some time even if your upstream time source becomes unavailable for any reason. Some NTP client implementations (probably most
ntpd daemon or system service implementations) do this, and others (like ntpd's companion
ntpdate which simply sets the clock once) do not. This is commonly referred to as a drift file because it stores persistently a measure of clock drift, but strictly speaking it doesn't have to be stored as a specific file on disk.
In NTP, stratum 0 is by definition an accurate time source. Stratum 1 is a system that uses a stratum 0 time source as its time source (and is thus slightly less accurate than the stratum 0 time source). Stratum 2 again is slightly less accurate than stratum 1 because it is syncing its time against the stratum 1 source. And so on. In practice, this loss of accuracy is so small that it is completely negligible in all but the most extreme of cases.
In network timekeeping the specification that tells you how a server gets its time source is called a Stratum Level. The lower the level, the better the time keeping of that server.
Stratum level 0 devices are not directly connected to the network. They are the actual timekeeping device itself, and must be connected to a computer to derive the actual time. This computer then becomes a Stratum level 1 NTP server.
A computer connecting to a Stratum level 1 can also become a time server, but it would then be a stratum level 2. As computers connect to time servers, the lower your stratum level, the more accurate your time keeping can be.
Stratum level 0 devices include atomic clocks taking part in TAI (international atomic time) or synchronized to it, and receivers of time signal sent by such clock. Most commonly those are GPS timekeeping receivers with an appropriate interface that includes the GPS PPS signal. The PPS signal, when the GPS has a good lock on several satellites, sends one pulse per second, and the leading edge of that pulse is within nanoseconds of the actual start of that second. Depending on the specification of the GPS receiver, the PPS signal may be more or less accurate. This is because each GPS satellite has an atomic clock. Once the GPS receiver has found its own position and the location of the GPS satellites it's listening to, it can correct for RF propagation and give you time almost as accurate as having an atomic clock right at the GPS receiver.
So Stratum level 1 servers connect to atomic clocks, or GPS receivers, and NTP servers connect to them. Even connecting to a stratum level 2 or 3 server with frequent adjustments will provide your computer with timing accuracy measured in nanoseconds. But if you need better timing, connect to a stratum level one server, or buy an appropriate timekeeping GPS receiver and become a stratum level one source yourself.
All clocks drift to some extent, it depends on the source of the timing signal and how well it is tracked. In a PC, this is the HPET these days, but the PC can lose track of how many ticks have gone by if overloaded.
The NTP servers your machine talks to are likely losing time too, however, they drift their time back to a better source.
Ultimately, the better sources are highly accurate clocks like atomic clocks. You can think of NTP as a network of machines, each one will have a number of sources it relies on for time and skews its own time to what is considered more accurate.
This is governed by a source declaring its stratum. An atomic or GPS clock is stratum 0, and the authority on what the time is. Each layer out from that is the next stratum - stratum 1, and will check a number of stratum 0 sources along with peers at the same level, in order to sanity check the time sources.
You are likely talking to a stratum 2 or 3 time source.
What the others wrote is true: a Stratum 1 server gets its time from a Stratum 0 device. I don't know in which time intervals that happens, but I think they are quite accurate there,.
A Stratum n server with n > 1 gets its time via NTP from a Stratum n-1 server. That means it synchronizes to it in regular intervals. Upon starting the NTP service, synchronization happens in quite short intervals, and by time, the intervals start increasing. Eventually, the interval is as large as 1024 s, about 17 minutes.
What hasn't been addressed is the question what happens in-between that time? Well, there is a facility called drift file. It helps NTP to monitor any drift between the local clock and the reference clock. The local clock's frequency is then adjusted according to the detected drift, so that the time is also accurate between server polls.
Other NTP implementation might use other facilities, but one thing is common: the need and the ability to adjust the clock's frequency.