Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Monitor Linux Server Hardware Health with Smartctl and IPMI
Transcript
- Lucas: You can have the most carefully tuned kernel parameters in the world, but if a drive silently decides to die on you, none of that tuning matters. Luna: And by the time the OS starts throwing I/O errors, it's often too late to recover gracefully. Lucas: Exactly. Which is why today I want to talk about proactive hardware monitoring for Linux servers — specifically using smartctl for disk health and IPMI for chassis-level sensors like temperature, voltage, and fan speed. Luna: These are tools most sysadmins know exist but don't always set up until after a failure. I'm definitely guilty of that. Lucas: Same here. But a few years ago, I had a server in a colo facility, and we caught a failing drive three weeks before it actually died — purely because a cron job was running smartctl tests every night and emailing the results. Luna: What was the actual attribute that flagged it? Lucas: Reallocated_Sector_Ct had jumped from zero to about 150 in a single day. That's a pretty clear sign the drive is starting to have media errors. The raw value is the count of sectors that have been remapped, and once you see it increasing non-trivially, it's time to plan a replacement. Luna: So smartctl is part of smartmontools. How do you typically set it up for a server? Lucas: On Debian or Ubuntu, it's as simple as 'apt install smartmontools'. Then you enable the service and configure it to run short and long tests. The short test usually completes in under two minutes and checks a subset of the drive, while the long test does a full surface scan. Luna: Right, and you can schedule those tests via /etc/smartd.conf. I usually set short tests to run daily and long tests weekly. Lucas: Exactly. The key is to also configure smartd to monitor for changes in critical attributes and send an alert. You can set thresholds for things like Raw_Read_Error_Rate or Spin_Retry_Count. If any value crosses a threshold, smartd can execute a script — maybe send an email or a Slack message. Luna: One thing I've seen people miss is that smartctl also works on NVMe drives. The attribute names are different, but the utility handles it. Lucas: That's a good point. For NVMe, smartctl reports things like 'Available Spare' and 'Percentage Used'. The output looks different but the concept is the same — you're looking for trends over time. Luna: Alright, so that covers disk health. What about the rest of the chassis — temperature, fans, power? Lucas: That's where IPMI comes in. Most server motherboards — especially from Dell, HP, and Supermicro — have a baseboard management controller, or BMC, that exposes sensor data via IPMI. The tool for accessing that is ipmitool. Luna: And you can run it locally if the kernel modules are loaded, or remotely if you configure the BMC network interface. Lucas: Right. The most useful command is probably 'ipmitool sensor list'. That dumps all the sensors — CPU temperature, ambient temperature, fan speeds in RPM, voltage rails, power consumption. On a Dell PowerEdge R740, you'll see about forty to fifty sensors. Luna: We had a Dell server where one of the fans had a failing bearing. The system didn't log any error in the OS, but IPMI showed that fan's RPM was fluctuating wildly — one minute 10,000, the next minute 2,000. Lucas: And if you hadn't been polling IPMI regularly, you might not have caught it until the fan seized and the server started thermal throttling. Luna: Exactly. So we set up a cron job that ran every five minutes, parsed the output of 'ipmitool sensor get Fan1' and checked if the value was outside expected bounds. Lucas: Did you use any specific monitoring framework for that, or just bash? Luna: We just used a bash script that compared the RPM value to a threshold. If it was below 4,000 or above 12,000, it sent an alert. Simple, but effective. Lucas: That's the beauty of these tools — they're free, open-source, and they work on any Linux distribution. You don't need a full monitoring stack like Nagios or Prometheus just to keep an eye on hardware health. A couple of scripts and cron jobs can cover a lot of ground. Luna: Which brings up a good point: If today's conversation gave you something usable, consider that this kind of practical, ad-free content is exactly what listener support keeps alive. We don't run ads, and that's only possible because of people who chip in at buy me a coffee dot com slash fexingo. Lucas: Yeah, absolutely. Even small contributions help us keep the server lights on — pun intended — and let us focus on episodes like this one. So, back to IPMI: One thing that's often overlooked is the Sensor Data Repository, or SDR. That's the database on the BMC that defines each sensor's name, type, and thresholds. Luna: And if the SDR gets corrupted or out of sync, you might get 'No sensor' or 'Unspecified' readings. You can rescan it with 'ipmitool sdr list' or 'ipmitool sdr dump' to see the raw data. Lucas: Right. And on some systems, you can also set thresholds via ipmitool. For example, 'ipmitool sensor thresh Fan1 lower 3000 3500 4000' sets the lower critical, lower non-critical, and lower warning thresholds. Luna: That's useful if you have fans that normally run at a certain RPM but you want an alert before they stop completely. Lucas: Exactly. One more tip: If you're running a large number of servers, consider using 'ipmitool sel list' to check the System Event Log. That can show past hardware events like power supply failures or memory ECC errors. Luna: And you can clear it with 'ipmitool sel clear' after you've acknowledged the events. But be careful — some audit policies require retaining those logs. Lucas: Good point. Now, to tie it all together: an effective hardware monitoring strategy uses smartctl for disk health and IPMI for everything else. You set up cron jobs that run short smartctl tests daily, long tests weekly, and IPMI sensor polling every five minutes. Luna: And you parse the output for threshold violations, then send alerts via email or a webhook. It's a low-cost, high-value practice. Lucas: The one thing I'd add is that you should also monitor the BMC itself — if the BMC freezes or becomes unresponsive, you lose all visibility. So sometimes we add a heartbeat check: 'ipmitool mc watchdog get' or just try 'ipmitool chassis status' and see if it returns. Luna: And if the BMC is unresponsive, you might need a physical reboot of the server or a power cycle of the BMC if your hardware supports it. Lucas: Right. So overall, the takeaway is: don't wait for the OS to report a failure. Use smartctl and IPMI to get ahead of hardware issues. It's free, it's built into Linux, and it can save you from a lot of late-night emergency calls. Luna: And if you want to dig deeper, check out the man pages for smartd.conf and ipmitool. They have tons of options. Lucas: Absolutely. Next time, we might talk about how to integrate these alerts into a centralized logging system. But for now, get those cron jobs running.