Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Diagnose a Silent Disk Failure on Linux
Transcript
- Lucas: So you've got a server that's running fine from the outside. Services are up, load average looks normal, no alerts in Nagios or Prometheus. But inside — quietly — one of the disks is starting to manufacture bad sectors. And if you don't catch it early, you're looking at corrupt databases, phantom application crashes, and eventually a full recovery from backup. That's the silent disk failure scenario, and it's way more common than most people think. Luna: I've definitely had that call. 'Hey, the server's up, but our CMS keeps throwing weird errors.' Three hours later we find 300 reallocated sectors on the root volume. Lucas: Exactly. And the thing is, standard monitoring usually misses this. CPU, memory, disk space — those are easy. But disk health at the hardware level? Most setups just don't look. And the 'it's working' assumption is dangerous because a failing disk can be partially functional for weeks. Lucas: This is actually a great moment to mention something about how we make this show. We keep Linux Server Admin ad-free — no sponsor reads, no mid-roll interruptions. And that's possible because a small group of listeners chips in monthly through buy me a coffee dot com slash fexingo. It's a simple way to keep the content focused on what's useful, not on selling you something. Luna: Yeah, honestly it makes a difference. We're not chasing ad metrics, we're just trying to cover the stuff that actually matters when you're on call at 2 AM. Lucas: Alright — back to the disk. Let's talk about the concrete signals. The first thing you want to check is SMART data. Specifically the Reallocated Sector Count and Current Pending Sector attributes. Those give you a direct count of sectors the drive has moved or is trying to read. Luna: And the threshold matters, right? One reallocated sector isn't necessarily a panic, but the trend is the real signal. Lucas: Exactly. If you see Reallocated Sector Count at zero for months and then it jumps to five in a week, that drive is failing. A lot of enterprise drives define their own threshold — some can handle a few hundred before they flag. But I've seen drives that went from 10 to 2000 in a weekend. Luna: What about the raw value versus the normalized value? I've seen monitoring tools that only show the normalized score out of 100, and people think 80 is fine. Lucas: Right, that's a key gotcha. Many tools default to showing the normalized value, which is vendor-specific and can be 100 until the drive is practically dead. You want the raw value — the actual count. In smartctl, that's the RAW_VALUE column. For example, 'smartctl -a /dev/sda' will show you the raw Reallocated_Sector_Ct. That's the number you need to trend. Lucas: And while you're in there, also check Current_Pending_Sector. That's sectors the drive is having trouble reading but hasn't reallocated yet. If that number is non-zero, you have a read problem. If it stays non-zero after a write, you have a write problem — and that's worse because it means data loss is happening. Luna: So the read path failure versus write path failure — can you break that down? What do I see in the logs? Lucas: Sure. Read path: the drive can't read a sector, so it retries, maybe eventually succeeds. You'll see I/O errors in dmesg, maybe slow reads, application timeouts. Write path: the drive can't write to a sector, so it remaps it, but if the remap fails, the write fails. That's when you get filesystem corruption. A read failure might cause a crash, but a write failure can corrupt data silently if you're not using checksums. Luna: And that's where ZFS or Btrfs have an advantage — they catch checksum mismatches on read. But most production servers are still on ext4 or XFS. Lucas: Exactly. ext4 and XFS don't have block-level checksums. So if the drive lies and says 'write succeeded' but the data is bad, you won't know until you try to read that block. And by then, you may have backed up the corruption. Lucas: So how do you catch this before it bites you? The simplest approach is to set up smartd to monitor your drives and send an alert when the raw values cross a threshold. Most distros include smartmontools. The configuration is in /etc/smartd.conf. Luna: What does a good smartd.conf look like for a typical server? I've seen people just enable '-a' and call it done. Lucas: You can do better than that. Here's a baseline: for each device, specify '-H' to check overall health, '-l error' to report SMART error log, '-f' to check for failures, and then set custom thresholds with '-s' for the attributes that matter. For example, '-W 0,50,0' will warn if temperature hits 50 Celsius. For Reallocated Sector Count, you can use '-r 5' to recheck, but the real key is to set a directive like '-s ' to run short and long tests. Luna: But the default smartd email alert goes to root. If nobody reads root's mail, you're still blind. Lucas: Absolutely. That's why you want to pipe alerts to something that actually reaches you. I usually set smartd to run a script that sends a message to our incident management system. Or even just a simple systemd service that runs smartctl periodically and checks raw values. Lucas: Here's another thing: don't rely on a single check. Schedule short self-tests daily and long tests weekly. The short test checks a small portion of the disk surface. The long test reads the entire disk. Both can detect issues that idle monitoring might miss. Luna: And if you're running RAID, especially hardware RAID, the disk isn't directly visible. You have to check SMART through the RAID controller. That's a whole other layer of complexity. Lucas: Yes. With hardware RAID, the controller presents a virtual disk to the OS. The individual drives' SMART data is usually accessible via the controller's management tool — like MegaRAID's storcli or HP's hpssacli. That adds an extra step. But if you're using software RAID like mdadm, each component drive is visible and you can check SMART directly. Luna: There's also the RAID-5 rebuild risk. A lot of people still run RAID-5 with large nearline drives. If one drive fails, the rebuild stresses the remaining drives, and another failure during rebuild means total data loss. Lucas: Classic story. I've seen it happen. And if the first drive failed because of silent corruption, the other drives might be in similar shape. That's why I'm not a fan of RAID-5 with drives over 4 TB. RAID-6 or RAID-10 is much safer. But regardless of RAID level, proactive monitoring of SMART data on every physical drive is essential. Lucas: One more thing: never trust a single 'fsck -f' result. If a filesystem has a few bad blocks, fsck might mark them as bad and continue, but the underlying drive could have many more pending sectors. After a fsck, always check SMART immediately to see if the count changed. Luna: Yeah, fsck isn't a diagnostic tool. It's a repair tool. The SMART data is the diagnostic. Lucas: Exactly. So the takeaway: set up smartd with raw value thresholds, route alerts to a place you check, schedule short and long tests, and check the trend — not just a single reading. And if you're on hardware RAID, learn your controller's tools. A little effort upfront saves you from a 3 AM restore. Luna: And maybe from explaining to your VP why your 'production' server actually had 500 bad sectors for three weeks. Lucas: Right. That conversation never ends well. Alright — next episode we'll dig into network diagnostic tools. Until then, keep an eye on those SMART numbers.