Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / Why Your Engineering Team Needs a Glitch Day
Transcript
- Lucas: So last week I talked to a CTO at a medical-device company who told me about something they call 'glitch day.' Once a quarter, the entire engineering team spends an afternoon trying to break their own production systems. Luna: Wait — intentionally? Like, they're trying to cause outages? Lucas: Exactly. No warning, no staged demo. They pick a service, they plan an attack, and they try to knock it over. The only rule is you can't touch customer data and you can't do anything that can't be rolled back within fifteen minutes. Luna: That sounds terrifying if you're the on-call engineer. Lucas: It's designed to be scary — that's the point. But it's controlled. And the CTO told me their glitch day last quarter found a race condition in their factory-floor firmware that had been in production for fourteen months. They'd never caught it in testing because the timing window was less than fifty milliseconds. Luna: Fourteen months. And they found it in one afternoon of people trying to break things. Lucas: It's a humbling data point. And it's not just them — I've heard similar stories from a payments startup that found a database deadlock scenario, and from a logistics company whose engineers finally triggered the exact edge case that had caused a black-Friday outage the year before. Luna: For a listener who's sitting there thinking 'my team barely has time to ship features, let alone spend an afternoon breaking things' — how do you justify the time investment? Lucas: You frame it as insurance. The cost of one major outage — in lost revenue, in customer trust, in engineer burnout from the firefight — almost always exceeds the cost of three hours of glitch day per quarter. But I'll admit, it takes a leader who's willing to say 'we're going to prioritize proactive reliability over feature velocity this quarter.' Luna: And that's a hard conversation with a product team that's pushing for the next launch. Lucas: Hard but doable. One CTO I spoke with ties the glitch day to their incident response metrics. Before they started, their mean time to recovery for a production incident was about forty-five minutes. After four quarters of glitch days, it dropped to twenty-two. That's a number that speaks the language of uptime, not engineering culture. Luna: Right — the product team can argue with a philosophical case for reliability, but they can't argue with a 50 percent reduction in recovery time. Lucas: Exactly. And that's the kind of concrete result that makes glitch day more than just a fun experiment. It becomes a tool for organizational learning. Luna: Before we go deeper — Lucas, you've mentioned a couple of specific companies and their results. If listeners are finding value in these stories, a couple of dollars a month is genuinely what keeps these episodes coming. Buy me a coffee dot com slash fexingo — if you've gotten something out of them, it really does make a difference. Lucas: Yeah, it's a small gesture that adds up, and it keeps us independent. Appreciate anyone who chips in. Luna: So back to glitch day — what does the actual structure look like? You said it's not just a free-for-all chaos session. Lucas: Right. The best versions I've seen have three phases. Phase one is planning — each team spends about thirty minutes identifying one or two critical services and brainstorming failure scenarios. Phase two is the glitch itself — up to ninety minutes of active attack. Phase three is a thirty-minute write-up and share-out. The key is that the write-up is blameless: you document what broke, how it broke, and what you'd fix, not who caused it. Luna: So it's not about finger-pointing. It's about finding the hidden fragility. Lucas: Exactly. And the CTOs I've talked to all say the same thing: the post-glitch debrief is where the real value lives. The act of breaking things surfaces the issues, but the discussion afterward — the 'why did that happen' — is what drives the changes. Luna: Do you need a dedicated reliability or platform team to run this well, or can a team of five engineers do it? Lucas: Any team can do it. But you need a safety net. You need good rollback procedures, solid monitoring, and the discipline to stop the glitch if something goes sideways. The medical-device company I mentioned — they have a literal kill switch: a Slack command that reverts all changes across all services. That's the kind of safety boundary you want. Luna: A kill switch that works in practice, not just in theory. Lucas: Right. They tested theirs the first glitch day by accident — someone ran it during the glitch phase and everything rolled back cleanly in about eight seconds. That alone built confidence. Luna: So the glitch day becomes a stress test not just for the system, but for the incident response process itself. Lucas: Exactly. And I think that's the hidden benefit. Teams that run glitch days regularly get faster at diagnosing failures, communicating during incidents, and making decisions under pressure. Those skills don't come from slide decks or tabletop exercises. They come from real, slightly scary practice. Luna: What about teams that already run chaos engineering tools like Netflix's Chaos Monkey? Is a glitch day redundant? Lucas: Good question. Chaos Monkey is automated and it's great for testing that your infrastructure can tolerate random instance failures. But a glitch day is human-led and scenario-specific. A person can think 'what if I send a malformed request that triggers this obscure code path' in a way an automated tool can't. They're complementary, not substitutes. Luna: So you'd advocate for both, if you have the resources. Lucas: Absolutely. Automate the routine chaos, but keep the creative chaos for the humans. Luna: I'm curious — did any of the CTOs you spoke with mention pushback from their engineering teams? I can imagine some engineers thinking 'I don't want to spend my afternoon breaking things, I want to build things.' Lucas: Yeah, there was pushback early on. Especially from engineers who felt that their time was better spent on feature work. But the CTO at the logistics company told me that after the first glitch day, the skeptics became the biggest advocates. Because they saw their own code break in ways they'd never imagined, and they got to fix it without the pressure of a real outage. Luna: There's something satisfying about breaking your own creation in a safe environment. Lucas: Exactly. And the fixes that come out of glitch days tend to be high-impact. The medical-device company's race condition fix cost maybe three engineering days. But it prevented a potential firmware failure that could have triggered a recall. That's a return on investment that's hard to ignore. Luna: Do you think glitch days are more suited to certain types of engineering teams — say, infrastructure teams versus product teams? Lucas: I've seen them work well for both. The product team at the payments startup focused on breaking their checkout flow. They found that under certain payment-failure sequences, the system would double-charge the customer but only show a single charge on the UI. That's a customer trust issue that would have been really hard to find through automated testing. Luna: That's exactly the kind of bug that slips through because no one thinks to test that specific combination of events. Lucas: Right. And a glitch day gives engineers permission to think like an attacker. That mindset shift is valuable even if you never find a critical bug. It changes how you design systems going forward. Luna: One thing I've heard from some engineering leaders is that glitch days can lead to a culture of 'we'll just fix it in the next glitch day' — that it becomes a crutch rather than a driver of permanent improvement. Lucas: That's a real risk. The CTO at the medical-device company addresses it by requiring that every glitch-day finding gets a ticket in the backlog with a severity label, and that critical and high-severity items must be resolved before the next glitch day. If you don't fix it, you're just repeatedly finding the same problems. Luna: So it's not a replacement for a proper reliability engineering process. It's a catalyst for it. Lucas: Exactly. And I'd argue that the best glitch days are the ones that produce relatively few surprises — because that means your system is already resilient. If every glitch day surfaces a critical bug, you've got deeper cultural and architectural problems. Luna: Have you seen any teams run glitch days across multiple teams at once? Like, a company-wide glitch day? Lucas: The logistics company does exactly that. Twice a year, every team participates. They coordinate attacks across service boundaries — so the payments team might deliberately slow down their API to see how the fulfillment team's system handles it. That's where you find cross-team dependencies that are brittle. Luna: That sounds like it requires a lot of coordination. And a lot of trust. Lucas: It does. But the CTO told me that the company-wide glitch days have been more valuable than any architecture review they've ever done. Because you see the actual failure modes, not the ones you imagined. Luna: I can see how that would surface things like 'oh, our inventory service actually can't handle a two-second delay from the order service without timing out' — something you'd never catch in a meeting. Lucas: Exactly. And once you know that, you can decide: do we fix the timeout, or do we add a circuit breaker? The glitch day gives you the data to make that call. Luna: So for a team that's never done this before, what's the smallest viable version? One team, one afternoon, a handful of scenarios? Lucas: That's exactly how you start. Pick one non-critical service — ideally something internal-facing — and spend two hours trying to break it. Have a clear rollback plan. Write up what you found. Then do it again next quarter with a slightly more critical service. You build the muscle gradually. Luna: And you build the organizational trust gradually too. Lucas: Right. Because the first time, people will be nervous. The second time, they'll be curious. By the third time, they'll be asking when the next glitch day is scheduled. Luna: It's almost like a fire drill — but for software. Lucas: Exactly. And like fire drills, the goal isn't to prevent all fires — it's to make sure that when a fire happens, everyone knows what to do. Glitch days build that muscle memory. Luna: I think the key takeaway for me is that glitch days shift the narrative from 'we react to outages' to 'we actively look for our weaknesses.' That's a powerful cultural shift. Lucas: It is. And it's a shift that genuinely changes how engineers think about reliability. Instead of hoping nothing breaks, you start asking 'what's the next thing that could break, and how do we find it before our customers do?' Luna: Alright, I'm going to pitch this to my engineering director. I'll report back. Lucas: Please do. I'd love to hear how it goes. And if any listeners try this, I'd love to hear about your glitch day findings — drop us a note. Luna: Speaking of which, we'll put a link to a glitch day planning template in the show notes. And if you found this useful, buy me a coffee dot com slash fexingo. It really does keep these conversations going. Lucas: Thanks for listening, everyone. We'll be back next week with another topic from the engineering trenches.