Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Strace for Troubleshooting
Transcript
- Lucas: If you've ever had a Linux server go slow and you had no idea why, strace is probably the single most useful tool you're not using enough. Luna: Strace — the system call tracer. I think a lot of people know it exists but maybe they've never actually needed it in a pinch. Lucas: Exactly. Strace intercepts and records the system calls made by a process — things like reading a file, opening a network socket, allocating memory. It shows you exactly what the kernel is being asked to do, and how long each call takes. Luna: So if your app is hanging, strace can tell you if it's stuck waiting on disk I/O, or a network timeout, or something else. Lucas: Right. Let me give you a concrete scenario. I had a web server running nginx that was serving pages fine under low load, but under moderate traffic, response times would spike to 10 seconds. CPU and memory looked normal. Disk latency was okay. So I attached strace to one of the worker processes. Lucas: I ran strace -p -c for a few seconds. The -c flag gives you a summary of syscall counts and time spent, instead of dumping every single call to the screen. Luna: And what did you see? Lucas: I saw that the process was spending over 70 percent of its time in the epoll_wait syscall. That's normal for an event-driven server — it's waiting for new events. But the waits were very long. Then I looked at the file descriptor activity and saw that a bunch of connections were blocked on writing to the upstream backend — a php fpm pool. Luna: So strace pointed you to the network bottleneck with the backend. Lucas: Exactly. Without strace, I might have wasted time tuning nginx buffers or swapping out the disk. But strace made it obvious: the problem was upstream. I checked the php fpm pool settings, increased the number of children, and response times dropped back to under a second. Luna: That's a perfect example. Let's talk about some practical flags. The -f flag traces child processes, right? Lucas: Yeah. If you're tracing a parent process that forks, -f follows the children. That's crucial for things like Apache with prefork — each child is a separate process. Without -f, you only see the parent's syscalls, which is mostly just waiting. Luna: And -e lets you filter by syscall name, so you don't get overwhelmed. Lucas: Right. For example, strace -e trace=read,write -p shows only read and write calls. Or you can use -e trace=network to see socket-related calls. That's super useful when you suspect a network issue. Luna: And -o writes output to a file, which is better than scrolling a terminal. Lucas: Yeah. But here's the thing — strace has a performance overhead. Every syscall gets intercepted, so attaching strace to a busy production process can slow it down significantly. You don't want to leave it running on a live server for more than a few seconds. Luna: That's a good warning. I've heard of people accidentally causing a self-inflicted denial of service by strace-ing a high-traffic process. Lucas: It happens. The rule is: use it surgically. Attach, capture a few hundred syscalls, detach. Use the -c summary mode if you can, because it collects counts and times without printing every line, which reduces overhead. Luna: Let's talk about combining strace with other tools. If you see a lot of open calls failing with ENOENT, that suggests a missing file — you might check with lsof or find. Lucas: Yeah. Or if you see read calls returning partial data, you might suspect a corrupt file. Strace gives you the raw data, but you need to interpret it in context. I often run strace and lsof together — lsof shows what file descriptors are open, and strace shows what's being done with them. Luna: And if it's a network issue, you can pair strace with tcpdump. Strace shows the syscall, tcpdump shows the actual packets. Lucas: Exactly. Another scenario: a cron job that runs a script and fails silently. You can run the script under strace -f -o /tmp/strace.log and then grep for 'ENOENT' or 'EACCES' in the log. That'll show you missing files or permission issues. Luna: I've done that. It's like having x-ray vision for your scripts. Lucas: If today's episode gave you something useful — a new angle on debugging, or a flag you didn't know — the reason we can keep these ad-free is listener support. It's at buy me a coffee dot com slash fexingo. Just a way to say this kind of deep-dive content is worth keeping around. Luna: Yeah, it really helps. We put time into these examples, and it's good to know people find them useful. Lucas: So, back to strace. One thing people often ask: how is strace different from ltrace? Luna: Ltrace traces library calls, not system calls. So if you're trying to debug a bug in a shared library, ltrace is your tool. But system-level issues like file I/O, network, signals — that's strace territory. Lucas: Right. And there's also perf, which does sampling-based profiling and has lower overhead. But perf gives you statistical profiles, not every syscall. Strace gives you exact sequence and timing. Luna: So when would you choose strace over perf? Lucas: When you need to see the specific order of operations — like 'first open the config file, then read it, then open a network connection'. Perf shows you hotspots, but not the flow. Strace shows the flow. Luna: That makes sense. Let's talk about one more advanced use case: tracing a running daemon that's in a weird state. Have you ever used strace with -p and -e trace=signal? Lucas: Yeah, that's great for debugging signal handling. If a process is ignoring SIGTERM, you can attach strace -p -e trace=signal and then send the signal. You'll see whether the process receives it and what sigaction returns. Luna: And if the process is hung in an uninterruptible sleep — the D state — strace will show you the exact syscall that's blocking. Lucas: Yes. That's one of the most valuable use cases. A process stuck in D state usually means it's waiting on kernel I/O, like a disk or NFS mount. Strace will show you the syscall, and you can figure out which mount point is failing. Luna: I've had that exact problem — an NFS export went dead, and all processes trying to access it were stuck. Strace on one of them showed the stat call on a path under /mnt/nfs. Problem found. Lucas: Exactly. And the quick fix was to force-umount the NFS share. But without strace, you'd be guessing. Luna: So let's summarize the key takeaways for someone who's never used strace before. Lucas: First, start with strace -p -c to get a summary — low overhead, high insight. Second, use -f to follow children, -e to filter syscalls, and -o to write output to a file. Third, be careful about performance in production — keep sessions short. Luna: And fourth, combine it with lsof and tcpdump for a fuller picture. Lucas: Yeah. Strace is one of those tools that you don't use every day, but when you need it, it can save you hours. It turns a black box into a transparent log of kernel interaction. Luna: I think every sysadmin should have it in their back pocket. Thanks, Lucas. Lucas: You bet. Next time, we'll talk about using perf for CPU profiling — a good complement to today's topic.