Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Recover a Linux Server from a Kernel Panic
Transcript
- Lucas: So, you're SSH'd into a production server, everything's normal, and then — nothing. The connection drops. You try to ping, no response. You drive to the data center or hit the BMC console, and there it is: black screen, white text, a stack trace, and the words 'Kernel Panic'. Luna: The worst feeling. And the typical reflex is to just hit the power button and reboot, hoping it comes back clean. Lucas: Right. But that's exactly what you shouldn't do. If you reboot immediately, you lose all forensic context. The panic message is gone, the memory state is gone, and if the panic was caused by a corrupt filesystem or a bad driver, you might be rebooting right back into the same broken state. Luna: So what's the better move? You're stuck at a physical console or an IPMI session staring at a frozen screen. Lucas: First thing: try the SysRq key. If the kernel isn't completely locked up, the magic SysRq mechanism can still respond. On most Linux boxes, you hold down the Alt key and the SysRq key — which is often the Print Screen key — and then press another key. The most famous one is alt sysrq r, then S, then E, then I, then U, then B. People call it 'Raising Elephants Is Utterly Boring' — R-E-I-S-U-B. Luna: I've heard that mnemonic. So R takes back control of the keyboard from the X server, S syncs filesystems, E sends SIGTERM to all processes, I sends SIGKILL, U remounts filesystems read-only, and B reboots. That's the safe emergency reboot sequence. Lucas: Exactly. It forces a controlled shutdown even when the panic has taken down normal services. Now, the problem is, by the time you get to the console, the panic message might have scrolled off the screen. You need a way to capture that output before you reboot. Luna: That's where kdump comes in. If you have kdump configured, the kernel on panic loads a second 'capture kernel' that dumps the contents of memory — the vmcore — to disk before rebooting. Lucas: Yes. Kdump is the gold standard. On a CentOS 7 or RHEL 7 system, it's usually installed by default but often not enabled. You check with 'systemctl status kdump'. If it's not running, you enable it: 'systemctl enable kdump --now'. But you also need to reserve memory for the capture kernel at boot time. That's done via the kernel boot parameter 'crashkernel=auto' or a specific value like 'crashkernel=256M'. Luna: And once kdump is active, after a panic, you get a vmcore file in /var/crash. Then you can analyze it with the 'crash' utility. That's where the real detective work happens. Lucas: Let's walk through a real example. I had a file server running CentOS 7 with an ext4 filesystem. It would panic maybe once every two weeks, always at 3 AM. The panic message showed a 'kernel BUG at fs/ext4/inode.c:1234' — some internal inconsistency. The backtrace pointed to ext4_mark_inode_dirty, and the call trace had a module parameter function from a custom storage driver. Luna: So the panic was in ext4 code, but the root cause was actually in that custom driver? That's the kind of path that's hard to see without a full crash dump. Lucas: Exactly. With the vmcore loaded in crash, I could inspect the module parameters of that driver. One of them, 'max_io_size', was set to 1 megabyte — too large for the ext4 journaling layer to handle in some edge cases. I changed it to 256 kilobytes in '/etc/modprobe.d/storage-driver.conf', and the panics stopped completely. Luna: That's a perfect illustration. Without the crash dump, you'd probably just reboot and maybe swap the hardware or run fsck — which wouldn't fix the real issue. So how does someone set up kdump from scratch on a new server today? Lucas: On a modern distribution like Ubuntu 22.04 or Rocky Linux 9, the process is straightforward. First, ensure you have the kexec-tools package installed — that's what provides kdump. Then, add 'crashkernel=512M' to the kernel command line in GRUB. On Ubuntu, you edit /etc/default/grub.d/kdump-tools.cfg or just use 'dpkg-reconfigure kdump-tools'. Then update grub: 'update-grub' on Ubuntu, 'grub2-mkconfig' on Rocky. Reboot, and verify with 'kdumpctl status'. Luna: And you should test it, right? Because the last thing you want is to think kdump is configured and then discover it doesn't work when you actually need it. Lucas: Absolutely. You can trigger a fake panic using SysRq-C — that's alt sysrq c — which forces a crash dump. Do this on a test server or during a maintenance window, not on production. After the system reboots, check /var/crash for a new timestamped directory containing the vmcore. If it's there, you're good. Luna: One thing that catches people: the crashkernel reservation eats memory. On a server with 64 gigs of RAM, 512 megabytes is nothing. But on a tiny VM with 2 gigs, losing half a gig might be noticeable. You can adjust the reservation or even skip kdump on low-resource systems and rely on SysRq for a clean reboot. Lucas: Right. For low-memory VMs, you might set 'crashkernel=128M' or use the 'fadump' feature on PowerPC systems, which uses firmware-assisted dump and needs less reserved memory. But for most x86_64 production servers, 256 to 512 megabytes is fine. Luna: And if you don't have kdump and you're staring at a panic on the console, what can you do before rebooting? Is there any way to capture that screen output? Lucas: If you have console access, take a photo with your phone. Seriously. That might be your only record. Some BMCs support serial over lan, so you can capture the console output to a file. On systems with a dedicated serial console, you can use 'minicom' or 'screen' to log the session. But if you're at the physical box, a phone picture is better than nothing. Luna: And once you snap that photo, then you can do the REISUB sequence to reboot safely. Or, if the panic is total and SysRq doesn't respond, you might have to power cycle. But at least you have the panic message for post-mortem. Lucas: Exactly. Now, let's talk about a scenario where SysRq works but kdump isn't configured. You can still get useful information. After the reboot, check /var/log/messages or 'journalctl -k -b -1' — that's the previous boot's kernel log. The panic message will be there, though without the full memory snapshot. Luna: So the procedure is: 1) Don't panic — pun intended. 2) Try SysRq to do a controlled reboot. 3) Photo the screen if possible. 4) After reboot, gather logs and, if you have it, analyze the vmcore. Lucas: Right. And the long-term fix is always: set up kdump on every production server. It's one of those things that's easy to postpone until you need it, and by then it's too late. Luna: Speaking of things that are easy to put off — listener support. We put a lot of work into these deep-dive episodes, and we deliberately keep them ad-free. No sponsor breaks, no interruptive commercials. Lucas: If today's tech conversation gave you something usable — a new debugging skill, a configuration trick, or even just a clearer mental model — and you want to support that ad-free choice, there's a simple way. The link is buy me a coffee dot com slash fexingo. No pressure, just a way to keep the show exactly what it is. Luna: Yeah. It helps us stay independent and focus on topics like kernel panic recovery instead of chasing ad revenue. Appreciate everyone who's already chipped in. Lucas: Alright, back to the nuts and bolts. Once you have a vmcore, the crash utility is your scalpel. The most useful commands inside crash are 'bt' — backtrace — which shows the call stack of the panicking CPU, 'ps' to list processes, 'files' to see open files, and 'log' to dump the kernel log buffer. The log command alone often gives you the exact panic string. Luna: And 'bt -a' shows backtraces for all CPUs? That can be helpful if the panic is a spinlock deadlock across cores. Lucas: Yes. 'bt -a' is crucial for lockup issues. Also, you can use 'dis' to disassemble the instruction that caused the panic. Sometimes the offending line is obvious — like a null pointer dereference where the code tried to access offset zero. Crash can even show you the source file and line number if the kernel was compiled with debuginfo. Luna: That requires the kernel-debuginfo package, which is often not installed on production systems for space reasons. But you can install it post-mortem from the same kernel version's repository. It's well worth it for the detailed analysis. Lucas: Absolutely. Now let's talk about something that's less common but terrifying: a panic during boot. If the kernel panics before the root filesystem is mounted, kdump might not be able to save the vmcore because the crash target may not be accessible. For that, you need 'crashkernel=auto' and ensure the dump target is a raw disk partition or network location, not a filesystem. Luna: Right. The default kdump configuration on RHEL/CentOS uses a raw partition or a file on the root filesystem. If panic happens early, the root filesystem isn't mounted. So best practice is to set 'dump_to_rootfs' or use a dedicated partition like /dev/sda3 as the dump target. Lucas: And test that early-boot scenario by adding 'panic=1' to the kernel command line and triggering a panic with SysRq-C during early userspace. That's advanced, but it catches configuration issues before production crashes. Luna: One final thought: container hosts. If you're running Kubernetes nodes, a kernel panic takes down all pods on that node. Kdump is still useful, but the node will be marked NotReady. After the node reboots, kubelet will drain it eventually. But the vmcore analysis might reveal if the panic was caused by a specific workload — like a container using a particular syscall that triggers a buggy kernel module. Lucas: Great point. And that's exactly the kind of insight you can only get if you have the crash dump. So, to summarize: configure kdump, test it with SysRq-C, know your SysRq sequences for emergency controlled reboots, and always grab logs before rebooting. That's the playbook. Luna: And a phone picture is better than nothing. Got it. Lucas: Exactly. Next time we'll look at how to simulate a kernel panic in a lab to practice this workflow — without taking down production. Until then, happy debugging.