Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux BPF for Real-Time Performance Tracing
Transcript
- Lucas: So you're SSH'd into a production server, something's slow, and you have no idea where the bottleneck is. You could install perf or strace, but maybe you don't have root, or you don't want to restart anything. What do you do? Luna: If I'm being honest, I'd probably start by guessing. Maybe check iostat, then top, then give up and restart the service. Lucas: Right, and that restart might lose you the evidence. But there's a better way: BPF. Berkeley Packet Filter. It lets you attach tiny, safe programs to almost any point in the kernel — system calls, network events, file operations — and collect exactly the data you need, with zero overhead when it's not running. Luna: I've heard eBPF thrown around a lot. Is that the same thing? Lucas: Close. Classic BPF was originally for network packet filtering — think tcpdump. But eBPF — extended BPF — generalised it. Now you can attach BPF programs to tracepoints, kprobes, uprobes, you name it. And it's been in the Linux kernel since version 3.18, which means most modern servers already support it. Luna: Okay, but how do I actually use this as a sysadmin without writing C code? I'm not a kernel developer. Lucas: Great question. That's where bpftrace comes in. It's a high-level tracing language, sort of like awk for kernel events. You write one-liners that specify a probe and an action. For example, to trace all new processes being created, you can run: bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf; }' Luna: So it prints the process name every time something calls execve. That's a lot less intrusive than running strace on a running process. Lucas: Exactly. And bpftrace is just the tip. There's also BCC — BPF Compiler Collection — which gives you Python and Lua bindings to write more complex tools. But for quick diagnostics, bpftrace is your friend. Let me give you a real scenario I dealt with last month. Luna: I'm listening. Lucas: We had a web server that was serving requests fine 99% of the time, but every few minutes, one request would take ten seconds. Classic tail latency problem. I could guess it was disk I/O or network, but I wanted to know exactly which syscall was slow. So I wrote a bpftrace one-liner that traced the latency of the write syscall: bpftrace -e 'kprobe:vfs_write { @start = nsecs; } kretprobe:vfs_write /@start/ { $delta = nsecs - @start; if { printf; } delete; }' Luna: Let me parse that. You're attaching a probe at the entry of vfs_write, storing a timestamp, then on return you check if the delta is over one millisecond, and if so, print the process name and the latency. That's a one-liner? Lucas: Yeah. And it worked. I found that a background log-rotation script was issuing synchronous writes to the same disk as the application data. The writes were serialising because of a misconfigured RAID controller. Without BPF, I'd have been digging through iostat and wondering why util was high but throughput was low. Luna: So BPF gave you the specific call chain instead of just an aggregate metric. That's powerful. Lucas: Exactly. And the best part: BPF programs are verified by the kernel before they run. They can't crash your system or enter infinite loops. There's a verifier that checks for safety — no dangling pointers, no unbounded loops. So it's production-safe. Luna: That's reassuring. But what about older kernels? If I'm stuck on CentOS 7, am I out of luck? Lucas: CentOS 7 shipped with kernel 3.10, which doesn't have full eBPF support. But you can still use kprobes and tracepoints via the older perf_event interface. However, the modern tools like bpftrace and BCC really need kernel 4.1 or later. For CentOS 7, you might be able to use the 'perf' command with tracepoints, but it's more limited. Luna: So what should someone do if they want to start using BPF today? Check their kernel version first, I imagine. Lucas: Right. Run 'uname -r'. If it's 4.1 or newer, you're probably good. Then install bpftrace from your distro's repos. On Ubuntu, it's 'apt install bpftrace'. On Fedora, 'dnf install bpftrace'. Then try a simple one-liner like tracing all 'open' syscalls to a particular file. Luna: And if you want to go deeper, BCC has tools like 'biotop' for disk I/O by process, 'execsnoop' for short-lived processes, 'tcptop' for network connections. I've used 'biotop' before and it's great for spotting which process is hammering the disk. Lucas: Yeah, BCC tools are basically production-ready utilities. They're packaged in most distros as 'bcc-tools' or 'bpfcc-tools'. The 'profile' tool is a personal favourite — it samples stack traces from the CPU at a rate you specify, giving you a flame graph of where time is spent. No restart needed, no application changes. Luna: So let's say I'm on a server right now and something feels off. I don't even know what's slow. What's the first BPF command you'd recommend? Lucas: I'd start with 'execsnoop' to see if any unexpected processes are spawning. Then 'biosnoop' to see if disk latency is high. Then maybe 'tcplife' to see if connections are hanging. Each gives you a stream of events with timestamps and latencies. You'll often spot the culprit in under a minute. Luna: And you're not adding any measurable overhead unless those events are actually happening, right? Because BPF compiles to bytecode that only runs when the probe fires. Lucas: Correct. The overhead per event is tiny — on the order of microseconds. So even in production, you can leave a trace running for a few seconds without impact. That's the killer feature: observability without surgery. Luna: You know, that reminds me of something. A couple of dollars a month is genuinely what keeps these shows going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, listener support is what keeps this ad-free and focused on real tools like BPF. It's a small thing that makes a big difference. Luna: So back to BPF — one thing that trips people up is understanding the difference between kprobes and tracepoints. Can you break that down? Lucas: Sure. Kprobes let you attach to almost any kernel function, but they're not guaranteed to be stable across kernel versions. Tracepoints are stable hooks that the kernel developers maintain — they have a fixed API. So for production scripts, tracepoints are safer. For deep debugging where you need to probe an undocumented function, kprobes are your only option. Luna: So if you're writing a tool you plan to reuse, use tracepoints. If you're debugging a one-off issue, kprobes are fine. That makes sense. Lucas: Exactly. And there's also uprobes — user-space probes — that let you trace function calls in running applications. For example, you could trace calls to malloc in a Python process without modifying it. That's incredibly useful for hunting memory leaks. Luna: I've seen Brendan Gregg's flame graphs from BPF data. They're beautiful. Is it hard to generate those? Lucas: Not anymore. The 'profile' tool from BCC outputs folded stack traces that you can feed into FlameGraph scripts. The command is basically 'profile -af 49 > out.stacks', then you run the stackcollapse script and flamegraph.pl. You get a visual SVG where you can see exactly which code paths are consuming CPU. Luna: So you're sampling 49 times per second, capturing the current stack, and then collapsing them into a flame graph. That's a powerful diagnostic technique. Lucas: And again, safe in production because you're just sampling. There's no instrumentation overhead because you're not modifying the code. Luna: What about container environments? Does BPF work inside a container? Lucas: It does, with caveats. BPF programs are attached to kernel hooks, so they run in the host's kernel context. A container can load BPF programs if it has the right capabilities — CAP_BPF and CAP_TRACING — but typically you'd run your tracing tools on the host and filter by cgroup or PID namespace. BCC tools have options like '--cgroupmap' to limit tracing to a specific container's cgroup. Luna: So you don't need to exec into the container to trace its syscalls. You can do it from the host, which is much cleaner. Lucas: Exactly. And you avoid installing debug tools inside each container. One host-level bpftrace can observe all containers. Luna: That's a huge operational advantage. Let's talk about a common pitfall: someone runs a bpftrace one-liner and gets 'no probes found' or 'invalid argument'. What's that about? Lucas: Usually it means the kernel doesn't have that tracepoint or function available. Sometimes you need to check if the kernel was compiled with CONFIG_DEBUG_INFO_BTF. BTF — BPF Type Format — is needed for modern co re BPF. Without BTF, your bpftrace might fail to resolve types. You can check with 'ls /sys/kernel/btf/vmlinux'. If that file is missing, you might need to install kernel-devel or a newer kernel. Luna: So the ecosystem is evolving fast. co re means you can compile a BPF program once and run it on different kernel versions, as long as BTF is available. That's a game-changer for distributing tools. Lucas: Absolutely. Projects like libbpf are making BPF portable. And BCC has been moving towards co re as well. For a sysadmin, it means you can grab a pre-compiled BPF program from a trusted source and run it without worrying about kernel headers. Luna: Are there any security concerns? If a malicious actor gains access to a server, could they use BPF to monitor other processes or even exfiltrate data from kernel memory? Lucas: That's a real concern. BPF requires root or CAP_BPF, CAP_TRACING, and CAP_SYS_ADMIN in many cases. So if an attacker already has root, they can do a lot of damage without BPF. But there are kernel protections — like the BPF verifier limiting what programs can do, and the fact that BPF maps are isolated. Still, best practice is to restrict BPF usage to trusted users and audit its use. You can use LSM hooks like bpf syscall filtering to control who can load programs. Luna: So treat BPF like any powerful admin tool: lock it down, but don't fear it. Lucas: Exactly. And monitor for unexpected BPF programs. Tools like 'bpftool prog list' show all loaded BPF programs on the system. That's a good thing to add to your security auditing. Luna: Alright, so let's say I'm convinced. What's the one BPF one-liner that every sysadmin should memorise? Lucas: If I had to pick one: 'bpftrace -e 'kprobe:do_sys_open { printf); }'' — that traces every file open, showing the process name and the filename. It's like strace for open calls, but lighter and safer. Great for spotting rogue processes reading config files or temp directories. Luna: And remember — you can stop it with Ctrl-C anytime. It only traces while running. No lingering overhead. Lucas: Right. That's the beauty of BPF. You get surgical insight exactly when you need it, and then you disappear. No permanent monitoring, no log bloat. Just answers. Luna: So next time you're SSH'd into a slow server, don't reach for strace or restart. Reach for bpftrace. It might just save your afternoon. Lucas: And if it does, you'll know exactly what to do — and you'll have learned something about your system that no dashboard can tell you.