Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose Linux Server DNS Resolution Issues
Transcript
- Lucas: So your application suddenly can't reach its external API endpoint. The database cluster looks fine, the network interfaces are up, ping to 8.8.8.8 works — but your curl request hangs until it times out. That smell is almost always DNS. Luna: DNS — the thing everyone assumes just works until it doesn't. I've had that moment where you stare at a hanging curl and realize the resolver is the silent culprit. Lucas: Right. And the tricky part is that modern Linux has multiple layers of DNS resolution. You've got the classic /etc/resolv.conf, but also systemd-resolved, NetworkManager, and sometimes a local caching daemon. Today we'll walk through a repeatable diagnostic process. Luna: Where do you start? Is it always the first thing you check? Lucas: First, I check the actual resolver configuration. Not just /etc/resolv.conf — because on modern distros, that file is often managed by systemd-resolved or NetworkManager. Run 'resolvectl status' to see what the active DNS servers are per interface. You might find that your primary interface is pointing to a loopback address like 127.0.0.53. Luna: That's systemd-resolved's stub resolver, right? It listens on 127.0.0.53 and forwards queries upstream. Lucas: Exactly. The stub is usually fine, but if it's misconfigured or the upstream servers in /etc/systemd/resolved.conf are wrong, you'll get timeouts. So step one: 'resolvectl query example.com'. That bypasses any application-level caching and shows you the resolved address or an error. Luna: I've had cases where resolvectl works fine, but the application still fails. What then? Lucas: Then you need to test at a lower level. Use 'dig @8.8.8.8 example.com' to bypass the local resolver entirely. If that works, the problem is in your local resolver chain. If it doesn't, you have a network-level issue — maybe firewall rules blocking UDP port 53. Luna: So dig is still the gold standard? Even with all these new tools? Lucas: Absolutely. Dig gives you the full response including the query time, the TTL, and any error codes. For example, if you get a 'SERVFAIL', that usually means the authoritative server refused the query. If you get 'NXDOMAIN', the domain genuinely doesn't exist. But the one that drives me crazy is a silent timeout — dig just hangs. Luna: That timeout — is it always a firewall blocking 53? Or could be an overloaded upstream server? Lucas: Could be either. One trick: run 'dig +trace example.com'. That follows the full delegation chain from root servers down. It pinpoints exactly which step is failing. If it stops at the root, you might have a DNS amplification attack blocking your source IP. If it stops at the TLD servers, maybe the domain's nameservers are down. Luna: I've used +trace before but it's verbose. Good for deep dives. Lucas: Exactly. Another quick test: use nslookup with a different resolver. Like 'nslookup example.com 1.1.1.1'. If that works but your default resolver fails, you know it's your resolver. Luna: What about the scenario where DNS works for some domains but not others? That happened to us once — our internal DNS worked fine, but any external domain timed out. Lucas: That's a classic split-horizon DNS issue. Usually it's because your /etc/resolv.conf points to an internal-only DNS server that doesn't have forwarders set up for external domains. Or your firewall allows DNS to internal servers but blocks outbound UDP 53. Luna: Right — and the symptom is that ping to an internal hostname works, but curl to an external domain hangs. Lucas: Exactly. So step two after checking resolvectl: run 'tcpdump -i any port 53' and then repeat your failing curl or dig. You'll see if the DNS query actually leaves the host. If you see no packets on the wire, the query is getting stuck in a local cache or stub. Luna: And if you see the query go out but no response, check iptables or nftables for DROP rules on port 53. Lucas: Right. Also check if the response is larger than 512 bytes — that can cause issues with UDP truncation. In that case, dig will show 'truncated' and the response will have the TC flag set. Then the resolver might retry over TCP, but if TCP port 53 is blocked, you'll still timeout. Luna: I've had that happen with DNSSEC responses — they can be huge. Lucas: Yes. So one fix is to ensure your firewall allows both UDP and TCP on port 53. Many people only open UDP, but DNS over TCP is required for large responses and zone transfers. Luna: Let's talk about caching. If you have a local caching resolver like dnsmasq or Unbound, how do you rule that out? Lucas: First, check if the caching service is running. Then flush its cache. For systemd-resolved, 'resolvectl flush-caches'. For dnsmasq, send a SIGHUP or restart. For Unbound, 'unbound-control flush_zone example.com'. Then retry your query. Luna: And if the cache flush doesn't help, you can temporarily stop the local resolver and point directly to 8.8.8.8 in /etc/resolv.conf. That's a quick way to see if the local resolver is the bottleneck. Lucas: Exactly. But be careful — on systems with systemd-resolved, manually editing /etc/resolv.conf might get overwritten. You need to either disable resolved or mask the symlink. For a quick test, you can run 'dig @8.8.8.8' from the start. Luna: One more thing: timeouts can also be caused by slow upstream DNS servers. If your resolver has a short timeout, it might give up before the upstream responds. Lucas: That's a good point. You can test latency with 'dig +time=5 +tries=1 @8.8.8.8 example.com' to see the response time. If it's over a second, you might want to switch to a faster resolver or add a second nameserver as fallback. Luna: A lot of this sounds basic, but I've seen production outages that boiled down to a single misconfigured resolv.conf. It's worth having a checklist. Lucas: Absolutely. And if today was actually useful to you, the way these stay ad-free is listener support. You can keep this kind of content coming by visiting buy me a coffee dot com slash fexingo. No pressure, just if you got value here. Luna: Yeah, it really helps us keep the server lights on — pun intended. Lucas: So to wrap up: start with resolvectl status, then dig +trace, then tcpdump. That three-step process will find 90 percent of DNS issues. The remaining ten percent usually involve DNSSEC validation failures or EDNS0 problems, which we can cover another time. Luna: Sounds like a plan. And always remember: DNS is the first thing to check, not the last. Lucas: Exactly. Thanks for hanging out — see you next episode.