Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose and Fix DNS Resolution Failures on Linux
Transcript
- Lucas: Alright, let me set the scene. It's a Tuesday afternoon. You've just pushed a new microservice to staging. You hit the endpoint, and it hangs. And hangs. Finally a timeout. You check the logs — connection refused. But the service is running, the port is open. Sound familiar? Luna: Oh yeah. And the first thing everyone does is blame the network team. But nine times out of ten it's something silly. Lucas: Exactly. So last month I had exactly this. A staging server running Ubuntu 24.04, systemd-resolved managing DNS. The service was trying to resolve an internal hostname, and it just failed silently. The fix was simple — but finding it took six hours because I started at the wrong layer. Luna: What was the actual symptom? Just a timeout? Lucas: Yes. Curl against an internal API endpoint timed out after 30 seconds. Tcpdump showed SYN packets going out but no syn ack. So the first assumption was a firewall issue. But the firewall team swore they hadn't changed anything. And they were right. Luna: So you went deeper. Where did you look next? Lucas: I ran dig against the hostname. And here's the thing — dig bypasses systemd-resolved. It queries the nameserver directly from /etc/resolv.conf. So dig returned the correct IP immediately. That's the gotcha. If you only use dig, you might think DNS is fine, but the actual resolver used by your application might be different. Luna: Right. So you need to test with the same resolver the system uses. On a modern systemd system, that means using resolvectl or systemd-resolve. Lucas: Exactly. When I ran resolvectl query the-hostname, it came back with a different IP — an old, decommissioned one. And it was cached. So the application was trying to connect to a server that no longer existed. Luna: Ah, stale cache. That's a classic. So was the cache stale because the TTL was too long, or because systemd-resolved was ignoring the TTL? Lucas: Great question. In this case, the upstream DNS server had a very short TTL — 60 seconds. But systemd-resolved had a positive cache entry that was hours old. Turned out the upstream DNS was returning a different record for external queries versus internal ones. The stub resolver on the server was configured to forward all queries to the corporate DNS server, but the server itself was on a different subnet. The corporate DNS was returning the wrong record based on the source IP. Luna: So it was a split-horizon DNS issue. The server's IP wasn't in the correct view. Lucas: Bingo. The fix was to configure systemd-resolved with conditional forwarding. For the internal domain, we pointed it to an internal DNS server that knew the correct subnet. For everything else, it went to the corporate DNS. Luna: Let's talk about how to do that. Because a lot of people just edit /etc/resolv.conf, but systemd-resolved will overwrite it. Lucas: Right. The proper way is to drop a config file in /etc/systemd/resolved.conf.d/. You create something like internal.conf with the following:, then DNS=10.0.0.53, then Domains=~internal.example.com. The tilde tells systemd-resolved that this DNS server is authoritative for that domain. Then you reload with systemctl restart systemd-resolved. Luna: And then you test with resolvectl query. But you also need to make sure the application is using the system resolver. Some applications hardcode Google DNS or Cloudflare. Lucas: Yes. That's another layer. In our case, the application was using glibc's gethostbyname, which checks nsswitch.conf. By default, it uses the systemd-resolved stub resolver listening on 127.0.0.53. So as long as /etc/nsswitch.conf has 'hosts: files resolve dns', it will use systemd-resolved. Luna: So the full checklist: verify nsswitch.conf, check the resolvectl status output, flush the cache with resolvectl flush-caches, then test with resolvectl query. Lucas: Exactly. And if you need to monitor DNS health, write a simple bash script that runs resolvectl query against your critical internal endpoints and alerts if the answer doesn't match an expected IP. I have one that runs every minute via a systemd timer. Luna: Let's hear the script. We love a good bash one-liner. Lucas: It's more of a three-liner. The core is: expected='10.0.1.100'; actual=$; if ]; then echo 'DNS mismatch' | systemd-cat -t dns-check; fi. That logs to journald, and you can set up a watchdog on that. Luna: Clean. And you could extend it to check multiple records, maybe even measure response time. Lucas: Yeah. But the key takeaway is: when you're debugging a connection timeout, don't stop at ping or dig. Use resolvectl. And know what your application's resolver chain looks like. Luna: I think that's a solid rule for any sysadmin. By the way, speaking of things that help keep infrastructure running smoothly — we've been doing this show for a while now, and it's entirely listener-supported. Lucas: It's true. No ads, no sponsors. Just a small group of folks who chip in through Buy Me a Coffee. If you've gotten something useful from an episode, that's the only way the show keeps going. Luna: And we really mean that. It's at buy me a coffee dot com slash fexingo. No pressure, but it helps us keep the server admin content ad-free. Lucas: Alright. Back to DNS. One more thing I want to cover: how to debug when systemd-resolved itself is the problem. Sometimes the stub resolver gets into a bad state. Luna: Like when it can't contact upstream and returns SERVFAIL? Lucas: Exactly. You can check with resolvectl statistics. If you see a high number of 'DNSSEC failures' or 'NTA failures', you might have a clock skew issue. systemd-resolved validates DNSSEC by default on some distributions. If the server's time is off, validation fails and queries return SERVFAIL. Luna: That happened to me once. The NTP service was down for three days, and suddenly every DNS query started failing. I was pulling my hair out until I checked the system time. Lucas: Right. So that's another thing to add to the checklist: timedatectl status. If the clock is off, fix NTP first, then flush the DNS cache. Luna: And you can disable DNSSEC validation with resolvectl dnssec no if you're in a controlled environment. But better to fix the time. Lucas: Agreed. So let's wrap this into a concise debugging flow. Step one: check resolution with resolvectl query. Step two: verify resolvectl status to see which DNS servers are configured. Step three: check nsswitch.conf. Step four: look at systemd-resolved logs with journalctl -u systemd-resolved. Step five: check system time. Luna: And if all else fails, you can restart systemd-resolved. But that's a sledgehammer. Better to flush the cache first. Lucas: Exactly. And remember, if you change /etc/resolv.conf manually, systemd-resolved will overwrite it unless you mark it as immutable with chattr. But that's a workaround, not a solution. Use the proper config files. Luna: Great episode. I think the next time someone complains about a timeout, we'll know exactly where to look. Lucas: For sure. And if you want to share your own DNS horror story, we're always listening. Until next time.