Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Why Your NTP Setup Is Probably Drifting
Transcript
- Lucas: I want to start with something that sounds boring until it breaks your entire database cluster: server clock drift. Luna: Oh, I've seen that. It's one of those things you don't think about until your replication logs are full of 'clock skew detected' errors. Lucas: Exactly. And the scary part is that most people assume their NTP setup is fine because the time looks roughly correct. But roughly correct is not good enough. If you're running anything distributed — databases, message queues, certificate validation — a fifty-millisecond offset can cause real failures. Luna: Fifty milliseconds? That's less than the blink of an eye. Lucas: It is. But in distributed systems, time is used for ordering. PostgreSQL's streaming replication, for example, uses timestamps to reconcile conflicts. If two servers have clocks that drift by even a hundred milliseconds, you can end up with phantom writes or replication lag that looks like a network problem but is actually a time problem. Luna: I remember a story from a few years back — a trading firm had their entire production pipeline stall because one server was twenty milliseconds behind the others. All their automated failover logic got confused. Lucas: Right. And the fix isn't just 'install NTP.' It's about choosing the right NTP implementation, configuring multiple upstream sources, and monitoring drift over time. So let's walk through what a production-grade time sync setup actually looks like. Luna: So first question: ntpd vs chronyd vs systemd-timesyncd — which one should you use? Lucas: Short answer: chronyd is the default on modern RHEL and CentOS, and for good reason. It handles network jitter better than ntpd, it synchronizes faster after boot, and it has a built-in smoothing feature that prevents large clock jumps. Systemd-timesyncd is fine for desktops and lightly loaded servers, but it only does simple client-side sync — no server mode, no sophisticated filtering. Luna: So if you're running a database or a load balancer, you want chronyd. Lucas: Absolutely. And you want to configure at least four upstream NTP servers. The classic advice is to use the NTP pool project — pool.ntp.org — which gives you a random set of servers from a huge pool. But here's the nuance: don't just use one pool alias. Use multiple, and spread them across different zones. Luna: Right, because if all your servers end up hitting the same pool server, you lose redundancy. Lucas: Exactly. And you also need to think about security. NTP can be used for amplification attacks, but more importantly, a rogue NTP server can deliberately skew your clock. So use iburst for faster initial sync, and consider using the 'noserve' restrict directive if you're only a client. Luna: What about monitoring? How do you know your clock is actually staying in sync? Lucas: chronyc tracking gives you the current drift rate and offset. The offset field is the difference between your system clock and the reference — anything under a millisecond is good. The drift rate tells you how many parts per million your hardware clock is off. A healthy crystal might drift 10 to 50 parts per million, so you want to see that number stable over time. Luna: And if you see the drift rate climbing, that can be an early indicator of a failing hardware clock. Lucas: Exactly. I've seen servers where the drift rate went from 20 parts per million to 200 over a month, and a week later the clock was off by seconds. Replacing the hardware clock fixed it. So monitoring drift rate is like checking tire pressure — it's cheap and it prevents bigger problems. Luna: Let's talk about a concrete config for arch. What does a good chrony.conf look like? Lucas: I'll give you a minimal but production-hardened version. You start with your pool lines: pool 0.pool.ntp.org iburst, pool 1.pool.ntp.org iburst, pool 2.pool.ntp.org iburst, pool 3.pool.ntp.org iburst. Then you set 'driftfile /var/lib/chrony/drift' to persist drift. You set 'makestep 1.0 3' to allow a one-second step if the offset is more than a second, but only in the first three updates. After that, it only slews. Luna: Because a large step can confuse applications that assume monotonic time. Lucas: Right. And you can set 'rtcsync' to periodically sync the hardware clock. But the key directive is 'maxupdateskew 100.0' — that prevents chronyd from accepting updates from servers with too much jitter. You also want 'logdir /var/log/chrony' and enable tracking measurements. Luna: One thing that catches people: they forget to open UDP port 123 outbound in their firewall. Or they leave it open inbound and don't restrict. Lucas: Good catch. Outbound 123 for NTP, and inbound only if you're running a server. Also, never use the same NTP source for all your servers in a cluster. If your pool goes down, every server drifts together — which sounds okay, but if they drift at different rates, you'll see skew between them. Luna: A better approach is to have one or two internal NTP servers that sync from external pools, and then point your application servers to those internal ones. Lucas: That's the standard architecture for any sizeable deployment. It reduces external traffic, isolates you from pool changes, and lets you control the quality of time your servers see. You can even run a local stratum-1 server with a GPS receiver if you're really paranoid. Luna: But for most teams, a couple of internal chrony servers with four pool servers each is more than enough. Lucas: Now, let me tie this back to the database example I mentioned. I had a PostgreSQL cluster where the replica would randomly fall behind by a few seconds, then catch up, then fall behind again. The replication delay graph looked like a sawtooth. We spent days checking network, disk I/O, PostgreSQL parameters — nothing. Finally, someone ran chronyc tracking on the primary and the replica. The primary had a drift of 12 parts per million. The replica had a drift of -85 parts per million. Their clocks were diverging at about one millisecond every ten seconds. Luna: Wow. And PostgreSQL's replication protocol is sensitive to time because it uses the WAL position plus a timestamp for conflict resolution. Lucas: Exactly. The replica was seeing timestamps that were in the future from its perspective, so it would hold off applying them. The fix was to replace the replica's CMOS battery and reconfigure chrony with better pool servers. Once the drift stabilized, replication smoothed out completely. Luna: That's a perfect example of why this stuff matters. And it's so easy to overlook because the clock looks right to the naked eye. Lucas: One more thing: leap seconds. They're rare, but they've caused major outages. In 2012, Reddit had a thirty-minute outage because their NTP servers didn't handle the leap second correctly. Chronyd has a 'leapsecmode' directive you can set to 'slew' or 'step'. Slew is safer because it spreads the adjustment over a few hours instead of causing an instant jump. Luna: I remember that. A lot of companies just ignore leap second handling until it bites them. Lucas: Right. And if today's conversation about time sync gave you something usable — a config tweak, a monitoring idea, a debugging tip — that's exactly why this show exists. We deliberately keep these episodes ad-free because we want the information to be the value. If you'd like to support that choice, there's a link at buy me a coffee dot com slash fexingo. No pressure, no perks, just a way to keep the server lights on. Luna: Yeah, it's a small gesture that makes a big difference for an independent show like this. Lucas: So to wrap the technical side: set up chronyd, use multiple pool servers, monitor drift with chronyc tracking, and check your leap second handling. Your databases will thank you. Luna: And your future self, when you're not debugging a phantom replication lag at 2 AM. Lucas: Exactly. Next episode, I want to talk about another silent killer — TCP retransmission rates and how to spot a bad NIC before it takes down your stack. Luna: I already know a story about that. See you next time. Lucas: See you.