Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Feature Flags Without Breaking Production
Transcript
- Lucas: So you've got a feature flag in production that's been sitting there for eighteen months, still toggling some old payment flow that nobody touches anymore. The tests pass, the deployment goes green, and then one Tuesday morning, a customer hits that legacy path and the whole thing falls apart. Luna: That's oddly specific. Did that happen to you or is this a hypothetical? Lucas: It's a composite, but I've seen the pattern enough times. And honestly, feature flags are one of the most under-tested parts of a modern codebase. Everyone treats them like a simple if statement, but they're actually a distributed system in miniature. You've got the flag definition, the evaluation context, the rollout percentage, the targeting rules, and then the actual code paths they gate. Luna: And most test suites just test the default path, right? The flag's on, so they never exercise the off branch. Lucas: Exactly. And that's the core problem. You're shipping a binary decision in your code, but your tests only cover one side of that coin. When the flag flips, you're effectively deploying a version of the app that has never run in your CI pipeline. Luna: So before we go deeper into the technical weeds, I want to mention something. If today's conversation gives you a practical idea for your own QA process, that's exactly the kind of thing we aim for. And listener support is what keeps this show ad-free and independent, so if you find value in these episodes, you can buy us a coffee at buy me a coffee dot com slash fexingo. It's a small gesture, but it genuinely helps us keep doing this. Lucas: Yeah, we really appreciate that. And speaking of practical ideas, let's talk about how to actually test feature flags without blowing up your test matrix. Lucas: The first rule is simple: you have to test both branches. And I don't mean just one test for each side. I mean your test suite needs to run with the flag on, and it needs to run with the flag off, for every flag that's still active in the codebase. Luna: That sounds like it would double your CI time overnight. What's the realistic way to do that? Lucas: You don't double it, you triage. The key is to separate your flags into two categories: temporary rollout flags and permanent configuration flags. Temporary flags are the ones you use for canary releases or A/B tests. Those are the ones that need full matrix testing because they're actively changing behavior. Luna: Okay, and permanent flags are more like feature toggles for, say, different pricing tiers or regional rules. Those aren't going away. Lucas: Right. For those, you don't need to run the full suite under every permutation. You need a focused set of tests that specifically exercise the boundary. But here's the thing: the moment a temporary flag has been on for more than a couple of releases, it becomes a permanent flag. And that's where the problems start. Luna: So you're saying stale flags are the enemy. How do you catch them before they bite? Lucas: You add a review step to your release checklist. Every time you deploy, you grep for flags that have been in the codebase for more than, say, three months, and you force a decision: either the flag gets removed and the old code path is deleted, or it gets promoted to a permanent configuration flag with a clear owner and a documented reason for being there. Luna: That's a discipline thing, not just a technical thing. But what about the testing itself? How do you actually run both branches without turning your CI into a nightmarish matrix? Lucas: There's a neat trick: you don't run both branches in the same test run. You run your default suite with the flag in its production state, and then you run a separate, smaller suite that toggles the flag and only runs the tests that touch that code path. You can do this by tagging your tests or using a test filter. That way, you get coverage on both sides without doubling the whole suite. Luna: So you're saying you need to know which tests are affected by a given flag. That requires some discipline in test organization. Lucas: It does, but it's worth it. And the same logic applies to rollout percentages. If your flag is set to roll out to ten percent of users, you have to test what happens when the flag is on, but you also have to test the code path that handles the case where the flag is off for the other ninety percent. That's the path that might have been forgotten. Luna: Right, and that's exactly what happened with that company I read about last month. They had a flag for a new checkout flow, rolled it out to five percent, everything looked fine, but then when they flipped it to a hundred percent, the old flow was broken because they'd stopped testing it. Lucas: That's a classic. And the fix is to have a permanent smoke test for both branches. Even if you don't run the full functional suite on the old path, you need at least a basic health check that the app boots and the main user journey works with the flag off. Luna: Okay, so what about the flag itself? How do you test the flag evaluation logic, not just the code behind it? Lucas: That's a great question, and it's often overlooked. The flag evaluation is separate from the feature code. You need to test that the flag service returns the correct value for a given user, given their attributes and the targeting rules. That's a unit test you can write without any UI or integration setup. Luna: So you're testing the flag itself like it's a piece of business logic. That makes sense. Lucas: Exactly. And you have to test the edge cases: what happens when the flag is missing, when the service times out, when the user doesn't match any rule. Because in production, those things will happen, and your code needs to degrade gracefully. Luna: And what about the kill switch? The whole point of a flag is to turn something off quickly if it goes wrong. But if you've never tested the off switch, it might not work. Lucas: That's the most important test of all. You have to have a drill where you flip the flag off in a staging environment and verify that the system returns to a stable state. And I mean a real drill, not just a unit test. You want to simulate that moment of panic and make sure the fallback actually works. Luna: So you're saying you need to practice failure. Like a fire drill for your code. Lucas: Exactly. And once you've done that, you have a much higher confidence that your feature flags are a safety net, not a trap. It's the difference between having a kill switch and having a kill switch that you know works. Luna: I love that framing. So to recap: test both branches, have a smaller focused suite for flag-specific tests, test the flag evaluation logic itself, and have a real kill-switch drill. Anything else? Lucas: One more thing: make sure your tests actually reflect the production flag configuration. I've seen teams write tests that manually set a flag to true, but in production the flag is set to false, so the test is meaningless. You need to have a way to load the real flag configuration into your test environment, or at least have a clear mapping. Luna: Yeah, that's a classic pitfall. You test in a vacuum and then the real world doesn't match. Lucas: Right. And that brings me back to the original story. The reason that payment flow broke was that the flag had been on for so long that the off branch hadn't been tested since the original rollout. The tests were green because they only ran with the flag on. It's a slow drift that you don't notice until it's too late. Luna: So the takeaway is that feature flags are not a set-and-forget thing. They require ongoing hygiene and testing. Lucas: Absolutely. Treat them like any other piece of production code. They deserve the same level of scrutiny, the same level of testing, and the same level of respect. If you do that, they'll save you a lot of pain. If you don't, they'll become the source of your next outage.