Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Network Time Protocol with Chrony
Transcript
- Lucas: If you manage even a handful of Linux servers, you know that time is not a nice to have. It is a hard requirement. And yet I still see production environments relying on the default ntpd setup from ten years ago, or worse, just letting the clock float. Luna: So today we are talking about Chrony. The modern replacement for the old NTP daemon that actually fixes some real pain points. Lucas: Exactly. Chrony has been the default on Red Hat Enterprise Linux since version 7 and on Ubuntu since 19.10. But adoption is still spotty. I think part of the reason is that if ntpd works well enough, people don't fix it. But the question is: does it really work well enough anymore? Luna: What does Chrony do differently that matters for a typical production server? Lucas: Two big things. First, Chrony handles network latency variation much better. If you have a server behind a flaky link or a virtual machine with intermittent network, ntpd struggles. It assumes relatively stable round-trip times. Chrony uses a different algorithm — it measures drift continuously and compensates for jitter. Second, Chrony can sync time very quickly on boot, even without a network connection, by using the hardware clock's last known drift. On a rebooted server, ntpd can take minutes to stabilize. Chrony does it in seconds. Luna: That boot speed is a bigger deal than it sounds. If you have containers or short-lived instances spinning up and down, you want them to have accurate time immediately. Lucas: Right. And it's not just containers. Think about certificate validation. If your server's clock is off by even a few minutes, TLS handshakes fail. You get errors that look like network problems but are actually time problems. I've seen production outages traced to a server that was 90 seconds behind. Ninety seconds. Luna: Let's get into the actual config. How do you set up Chrony on a standard Linux server? Lucas: On most modern distros, Chrony is either already installed or one apt or yum command away. The main config file is /etc/chrony.conf. The basic structure is simple: you define time sources — usually pool directives for public NTP servers, or server directives for internal time servers. Then you set a few tuning parameters. Luna: Walk me through a minimal production config. What would you put in there? Lucas: A solid baseline looks like this. You set iburst on your server lines to speed up initial sync. Then you add a pool like pool 2.pool.ntp.org iburst. That gives you a rotating set of servers. Then you set the driftfile location — typically /var/lib/chrony/drift. That stores the measured clock drift so it survives reboots. Then you set makestep 1.0 3. That means if the clock offset is more than one second, Chrony will step it immediately rather than slewing it slowly, but only in the first three updates after boot. After that, it slews gradually. Luna: That makestep parameter is key. The default in some old ntpd configs would let the clock drift forever without stepping. I've seen servers that were off by minutes because no one configured a step threshold. Lucas: Exactly. And then you want to set rtcsync. That tells Chrony to periodically write the system time to the hardware clock. Without it, your hardware clock drifts independently, and after a full power-off, the system time on next boot can be way off. Luna: Do you recommend using public NTP pools for all servers, or should you run your own internal time server? Lucas: If you have more than maybe ten servers, running an internal NTP tier is better. You set up one or two servers that sync from public pools, then have your other servers point to those internal ones. That reduces load on the public infrastructure and gives you more consistency inside your network. You also avoid the problem of all your servers hitting the same external servers and causing rate limiting or traffic spikes. Luna: And with Chrony, the internal server config is almost identical. You just change the pool directive to a server directive pointing to your internal host. Lucas: Right. Now let's talk about monitoring. The main tool is chronyc. You run chronyc tracking to see the current offset, drift, and last update. chronyc sources -v shows you all the time sources and their status. And chronyc sourcestats -v gives you per-source statistics like offset and delay. Luna: What do you look for in the tracking output? What numbers are red flags? Lucas: The offset value is the immediate difference between system time and the reference. Anything under 100 microseconds is fine. Between 100 microseconds and 1 millisecond, you might want to investigate. Above 1 millisecond, something is wrong — maybe a network issue or a hardware clock problem. The drift value tells you how fast your clock naturally drifts per second. A typical value might be 5 or 10 parts per million. If you see 100 ppm or higher, your hardware clock might be failing. Luna: I ran into a case where a virtual machine had an offset of 50 milliseconds constantly. Turned out the hypervisor was not properly forwarding timer interrupts. The fix was to switch the VM's clock source from kvm-clock to tsc, but that depends on the hypervisor. Lucas: That's a great point. Chrony on VMs has its own set of gotchas. The guest clock can be affected by host CPU scheduling. Chrony handles that better than ntpd, but you still need to be careful. You might want to disable the kernel's 11-minute mode, which is an old mechanism that tries to sync time periodically but can conflict with Chrony. Luna: How do you disable that? Lucas: You add the line 'rtcfile /var/lib/chrony/rtc' in chrony.conf, and it handles it automatically. But you should also check that the systemd-timesyncd service is not running if you're using Chrony. Some distros ship with timesyncd enabled by default, and they conflict. Luna: Right, you want to mask timesyncd and enable chronyd. That's a common rookie mistake. Lucas: Let's do a quick real-world scenario. Imagine a three-node Cassandra cluster. Cassandra uses timestamps for conflict resolution. If two nodes have clocks that drift more than a few hundred milliseconds apart, you can get data inconsistencies, tombstones appearing out of order, and even node rejections. Luna: That happened to a team I know. They had a clock skew of 200 milliseconds between two nodes. Cassandra started throwing 'too many tombstones' errors, and reads became slow because the wrong versions were being resolved. The root cause was that one node was using ntpd with a misconfigured pool and the other was using hardware clock only. Lucas: It's a classic failure mode. And the fix was straightforward: standardize on Chrony across all nodes, point them to the same internal NTP server, set makestep properly, and add monitoring with chronyc so they could see offsets in real time. Luna: Before we go further, I want to mention something. Episodes like this one — where we give you specific configs and commands — are exactly the kind of content we want to keep freely available. Lucas: Yeah, no paywalls, no ads. If you find this useful and want to support that approach, there is a link at buy me a coffee dot com slash fexingo. It's a simple way to help us keep the server running and the content coming. Luna: No pressure. Just if you get value from these deep dives, that's the spot. Lucas: Alright, back to it. Another feature of Chrony that people underuse is the ability to serve time to other machines. You can configure Chrony as a server by adding the 'allow' directive in chrony.conf. For example, 'allow 192.168.1.0/24' lets clients on that subnet query your server. Luna: And you can also set up authentication with symmetric keys if you want to prevent spoofing. That's in the 'keyfile' directive. Lucas: Yes. For internal networks, it's not always necessary, but if you have strict security requirements, it's good to know. Now, one more thing: how systemd integrates with Chrony. On modern systems, systemd-timesyncd is a simpler alternative, but it's much less feature-rich. If you need precision, Chrony is the way. You can check if systemd-timesyncd is active with 'timedatectl status'. Luna: And if you see 'NTP service: active', that might be timesyncd, not Chrony. You need to stop and disable timesyncd, then enable chronyd, then restart. Lucas: Exactly. A quick sanity check: run 'chronyc tracking' and see if you get a valid response. If you get '506 Cannot talk to daemon', Chrony is not running. If you get data, you're good. Luna: What about leap seconds? Does Chrony handle them gracefully? Lucas: Yes. Chrony supports leap second smearing — that's where instead of inserting an actual leap second, the system gradually adjusts the clock over a window of several hours. That avoids the one-second step that can cause applications to misbehave. You configure it with the 'leapsecmode' directive. The default is 'slew', which does a gradual adjustment. Some organizations prefer 'step' for strict accuracy, but for most services, smearing is safer. Luna: I remember the 2016 leap second where some systems actually crashed. Linux servers with older ntpd had issues. Chrony's smearing would have prevented that. Lucas: Right. So to wrap up: if you are still using ntpd, migration to Chrony is straightforward. You install chrony, stop ntpd, copy your time source list into chrony.conf, add a few key directives like makestep and rtcsync, start chronyd, and verify with chronyc. The whole thing takes ten minutes. Luna: And if you're on a system that uses timesyncd, the migration is similarly simple. Just disable timesyncd and follow the same steps. Lucas: Next time you're debugging a weird database issue or a certificate error, check the time first. It's the simplest thing that can cause the most confusing problems. Luna: And with Chrony, you get the peace of mind that your clocks are as accurate as they can be without expensive hardware. Lucas: Alright, that's it for episode 95. We'll be back with another practical deep dive soon.