Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Debug a Kernel Panic in Production
Transcript
- Lucas: All right, let me paint a picture. It's three in the morning, you're on call, your phone buzzes — a production server is down. You SSH in and the screen shows a blinking cursor on a black background with a cryptic error: 'Kernel panic — not syncing'. What do you do? Luna: Panic alongside the kernel, probably. But seriously, that's one of those moments where even experienced sysadmins can freeze. Lucas: Exactly. And the wrong move is to just hit the reset button and hope it doesn't come back. I've seen teams do that repeatedly, and the panic keeps returning because they never actually diagnosed the root cause. Luna: Before we dive into the fix — and I know you've got a great story about a fintech company that had this exact problem — I want to say something quickly. If you're a regular listener and you've gotten something useful out of these episodes, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small thing that makes a real difference. We keep the show ad-free that way. And speaking of real differences — let's talk about how that fintech team finally solved their kernel panics. Lucas: So this was a fintech startup running about 200 bare-metal servers. They started seeing intermittent panics — maybe once a week, always on different nodes, always in the middle of the night. The standard response was to reboot and file a ticket. After six weeks and about a dozen panics, someone finally said, 'Let's capture the crash dump.' Luna: And that's the first lesson — you can't debug what you don't capture. Most Linux distributions have kdump pre-installed but often not enabled. Lucas: Right. kdump reserves a small chunk of memory — usually 128 megabytes or so — that a second kernel, called the crash kernel, uses to boot when the primary kernel panics. That crash kernel can then write a copy of the crashing kernel's memory to disk. The file ends up in /var/crash by default. Luna: And what did they find in their crash dumps? Lucas: That's the thing — they didn't have kdump configured. So after the first real panic where they decided to investigate, they had to enable it and wait. They set up kdump, rebooted each node to activate it, and then waited. Two weeks later, another panic hit. This time they had a dump. Luna: So they had the dump — then what? How do you even start reading a kernel memory image? Lucas: You use the 'crash' utility. It's a tool that understands kernel data structures. You load the dump file and the uncompressed vmlinux image from the same kernel version, and you get a command prompt. The first thing most people type is 'bt' — backtrace — which shows the stack trace of the panicking CPU. Luna: And in their case, what did that stack trace show? Lucas: It ended in a memory management function — specifically, something in the page allocator. But the really telling clue was in the 'dmesg' output that's preserved in the dump. They saw machine check errors logged by the kernel's mcelog facility. Those are hardware-level errors reported by the CPU when it detects a memory corruption. Luna: Ah — so it was a hardware issue, not a kernel bug. Bad memory. Lucas: Exactly. But here's where it gets interesting. They ran memtest86 on the suspected server during the next maintenance window, and it passed. No errors. So they thought, maybe it's a one-off glitch. But the panics kept happening on different nodes. That's when they realized the common denominator wasn't a specific server — it was a batch of DIMMs from a particular manufacturing run. Luna: So intermittent memory errors that only showed up under heavy load, not during a standard memtest. That's a nightmare. Lucas: Right. The fix was to replace all DIMMs from that batch across the fleet. But the process of isolating that took weeks. And it all started with having crash dumps to analyze. If they had just rebooted each time, they would never have known. Luna: What about servers that don't have kdump configured? Are there other ways to capture panic information? Lucas: Yes — netconsole. It's a kernel module that sends kernel log messages, including panic messages, over UDP to a remote logging server. You can set it up as a kernel boot parameter: netconsole=@eth0,514@192.168.1.100. That way, even if the server crashes and can't write to disk, the last messages are sent over the network. Luna: And those messages might contain the stack trace that tells you what went wrong. Lucas: Exactly. Netconsole is lightweight and doesn't need a reserved memory region, so it's a good fallback. But it's not a replacement for a full crash dump because you only get the kernel log buffer, not the entire memory state. Luna: Right. So for production, you want both — kdump for deep analysis, netconsole for when kdump fails. Lucas: And one more thing — configure the kernel to reboot automatically after a panic. There's a sysctl parameter: kernel.panic = 10. That tells the kernel to wait ten seconds after a panic and then reboot. That way, even if you don't have immediate access, the server comes back on its own. Luna: But if you do that, you risk losing the crash dump if kdump hasn't finished writing it before the reboot. Lucas: True. So you need to balance that. In their fintech environment, they set the panic timeout to 60 seconds, which gave kdump enough time to write the dump. And they also had a monitoring alert that fired if a server had been down for more than two minutes — so they knew to check for a panic. Luna: That's a solid playbook. Let's talk about preventing panics in the first place. What are the most common causes that sysadmins can actually do something about? Lucas: Number one is faulty hardware — memory, CPU, or storage. You can catch a lot of that by monitoring machine check errors from mcelog. Set up alerts so that if a server logs even one corrected machine check, you investigate. Uncorrected errors usually mean an imminent panic. Luna: What about kernel bugs or driver issues? Lucas: Those happen too. A classic example is the ext4 filesystem bug that caused panics on certain kernel versions when the journal was full. The fix was a kernel update. That's why you should always run a stable, supported kernel — not the latest mainline, but a distribution kernel with backported fixes. Luna: And sometimes it's a configuration issue, right? Like setting 'panic_on_oops' to 1, which turns a recoverable kernel bug into a forced panic? Lucas: Exactly. That's a common one. Some people set panic_on_oops because they want the system to crash and restart rather than limp along in an inconsistent state. But if you have a driver that triggers frequent oopses — say, a buggy network card driver — suddenly you're panicking multiple times a day. You need to understand what oopses are happening and fix the root cause, not just enable panic_on_oops as a band-aid. Luna: So the takeaway is: don't treat kernel panics as random events. They're symptoms. Capture the data, analyze it, and you'll often find a hardware or driver issue that can be fixed. Lucas: Exactly. And the fintech startup? After replacing those DIMMs, they went from a dozen panics in six weeks to zero in the following six months. The time invested in setting up kdump and analyzing dumps paid for itself many times over. Luna: That's a great example. Alright, next time someone's server drops into a panic, they'll know what to do — don't just reboot, capture and analyze. Lucas: And maybe set up netconsole today, before you need it.