Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Software Testing Fails When Engineers Ignore Human Error
Transcript
- Lucas: If today's tech conversation gave you something usable, consider supporting the show directly through buy me a coffee dot com slash fexingo. It keeps these episodes ad-free and lets us dig into cases like this without corporate interference. Luna: Yeah, and it means we can talk about messy real-world failures instead of polished press releases. Thanks for keeping the lights on. Lucas: Let's talk about the CloudScale outage from last month. It was a textbook case of perfect unit tests, passing integration suites, and a complete breakdown in production. The engineering team had ninety-eight percent test coverage on their core billing engine. By every standard metric we use in QA, they were green across the board. Luna: But the bug wasn't in the code. It was in the deployment script that an engineer ran manually because the automated pipeline timed out during a peak load window. Lucas: Exactly. And that's the blind spot. We spend millions automating the happy path, the edge cases, the data validation. But we treat human operators like variables that don't need testing. When the lead engineer rushed the rollback procedure to fix a latency spike, he used a deprecated command line flag. The system accepted it. The tests didn't catch it because no one wrote a test for that specific manual sequence. Luna: So we're essentially stress-testing our software but not our processes. That feels like leaving the front door unlocked just because the alarm system is working perfectly. Lucas: That is a brilliant analogy, Luna. And it highlights a shift happening right now in enterprise software. We're moving past the era where 'if it compiles, it ships.' We're entering the era of 'if the humans don't mess it up, it ships.' The CloudScale report showed that forty percent of their critical incidents over the last two years involved some form of manual intervention bypassing safety checks. Luna: Forty percent is staggering. You'd think they would have built a guardrail against that kind of shortcutting after the first few fires. Lucas: They tried. They added approval gates. They added read-only modes. But engineers hate friction. When production is burning, you want the fastest path to the fire extinguisher. So they created shadow workflows—unofficial Slack channels and shared drives where senior devs could execute fixes without triggering the formal change management process. Those shadow workflows were never tested. They were tribal knowledge. Luna: Tribal knowledge is just another word for undocumented debt. If it's not in the repo, it doesn't exist for the next hire or the audit. Lucas: Precisely. And here is the concrete takeaway for any engineering leader listening: you need to start testing your runbooks with the same rigor as your code. A runbook is a step-by-step guide for handling emergencies. At CloudScale, their runbook said 'execute command alpha.' But command alpha changed versions twice in the last year. The documentation hadn't been updated. The tests didn't check the documentation. So the engineer followed the book and broke the bank. Luna: So the solution isn't just better code. It's version-controlling your operations. Treating your emergency procedures like software features that get pull requests and reviews. Lucas: Yes. And I mean literally. There is a growing category of tools called 'Chaos Engineering for Ops' that actually simulates human error. They inject wrong commands into staging environments to see if the system fails gracefully or if it amplifies the mistake. CloudScale implemented this post-mortem. They started running monthly drills where junior engineers are forced to follow outdated documentation to fix a simulated crisis. The goal isn't to see if they succeed. It's to see where the documentation lies. Luna: That sounds incredibly uncomfortable. Making people fail publicly in a drill so you don't fail silently in production later. Lucas: It is uncomfortable, but it's also the only way to break the cult of competence. We assume senior engineers know what they're doing. But senior engineers get tired. They get distracted. They copy-paste old scripts. By making the failure visible and low-stakes in a simulation, you force the organization to update its truth. It turns tribal knowledge into institutional knowledge. Luna: And it exposes the gap between what we think happens and what actually happens. Most teams operate on a fantasy of perfect adherence to protocol. Lucas: Right. And that fantasy is dangerous because it creates a false sense of security. When the tests pass, leadership assumes the risk is managed. But if the risk lives in the hands of a single person making a split-second decision under pressure, then your test coverage number is meaningless. You could have ninety-nine percent coverage and still lose everything on that one percent of human action. Luna: It reminds me of aviation. Pilots have checklists, but they also do random simulator checks to ensure they haven't gotten complacent. Software teams rarely do that for their own operational habits. Lucas: The aviation comparison is apt. In fact, some larger fintech firms are borrowing heavily from aviation safety protocols. They require two-person verification for high-risk commands. Not just a digital signature, but a live voice confirmation. It adds friction, yes. But friction prevents catastrophic error. The question for business leaders is whether you value speed over resilience. Luna: Usually it's framed as speed versus quality. But this reframes it as speed versus survival. There's a difference. Lucas: There absolutely is. And the market is starting to punish the companies that ignore this. Investors are looking at operational maturity, not just product velocity. If your platform goes down because an intern clicked the wrong button, that's not a bug. That's a governance failure. And governance failures trigger regulatory scrutiny that product bugs do not. Luna: Especially in finance or healthcare. One manual error can violate compliance standards overnight. The cost isn't just downtime; it's legal liability. Lucas: Which brings us back to the economics of testing. Most QA budgets are allocated to automated tools and cloud compute for running tests. Very little goes toward human-centric testing. But investing in rigorous runbook validation, chaos drills for ops, and friction-heavy safety protocols yields a higher return on investment than adding another layer of unit tests. Unit tests catch logic errors. Operational testing catches reality errors. Luna: Reality errors are the ones that keep you up at night. Logic errors are just annoying until you patch them. Lucas: Exactly. So my advice to any engineering manager or CTO listening is this: audit your manual processes. Find the three most common emergency actions your team takes. Then write a test for each one. Simulate the failure. Watch someone else try to fix it using only your documentation. See where they stumble. Fix the doc. Repeat. It's slow work. But it builds trust. Luna: Trust in your own systems. Which is probably the most valuable asset a software company has besides its codebase. Lucas: Agreed. We often think of reliability as a technical property. But it's really a cultural one. It's about admitting that humans will make mistakes, and designing systems that survive those mistakes rather than hoping they won't happen. Luna: Hope is not a strategy. Documentation and drills are. Lucas: Well said. As we wrap up, I want to leave you with a question to ponder about your own workflow. What is the one manual task you perform regularly that you haven't tested in over six months? Maybe it's time to put it through the wrinkle.