Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Feature Flags for Safe Software Rollouts
Transcript
- Lucas: You remember the global social media outage in early 2025? A feature flag—basically a simple on-off switch for a new feature—was accidentally enabled for the entire user base. The result: hours of downtime, millions in lost revenue, and a lot of embarrassed engineers. Luna: I remember that. The weird part was, the flag itself wasn't even for a major feature. It was an internal debugging toggle. Lucas: Exactly. That's the danger. Feature flags are supposed to let you roll out features incrementally, run A/B tests, or kill a buggy feature instantly. But if they're not tested properly, they become a single point of failure. Today we're talking about how to test feature flags so they don't blow up in your face. Luna: So where do we start? How do you even test a toggle? Lucas: First, you need to understand the anatomy of a feature flag. At a minimum, a flag has a name, a targeting rule—like user ID ranges, geographic regions, or random percentage—and a fallback value for when the flag service is unreachable. The fallback is critical. Too many teams default to returning the flag as 'on' because they want to avoid users experiencing the old experience. That's a recipe for disaster. Luna: Right, because if the flag service goes down, suddenly every user gets the new, half-baked feature? Lucas: Precisely. So the first test is fallback behavior. You simulate a flag service outage and verify that the application gracefully defaults to the safe experience—usually the old code path. This is basic chaos engineering for flags. Luna: That makes sense. But what about testing the flag's targeting logic? For example, if you want to roll out to only US users on iPhones running iOS 17, how do you test that combination? Lucas: That's where integration testing comes in. You need to test each targeting dimension independently and together. Write automated tests that set up mock flag evaluations for specific contexts—like user location, device type, app version—and assert that the correct code path is executed. Use a test flag service that lets you predefine evaluations so you don't hit a real production flag endpoint. Luna: So you're basically creating a fake flag environment for tests? Lucas: Yes. Many teams use a library like OpenFeature, which allows you to inject a custom provider for testing. You can then simulate any flag state. And crucially, you should test that the flag's availability matches the rollout plan. For example, if a flag is supposed to be 10% enabled, your test should verify that approximately 10% of simulated users get the flag turned on. Use statistical tests to ensure the distribution is uniform. Luna: That's a lot of scenarios. Do teams usually automate all of that? Lucas: And honestly, this level of detail is exactly what makes the difference between a smooth deployment and a headline-making outage. If you're getting practical value from this conversation, that's the reason we keep the show free of ads and sponsors. Luna: Yeah, it's a really useful deep dive. If today's tech conversation gave you something usable, you can help by buying us a coffee—just visit buy me a coffee dot com slash fexingo. Every bit helps. Lucas: Absolutely. Now, back to the automation question. Most teams do automate these tests, but they often miss one key scenario: what happens when a flag is removed? Legacy flag cleanup. Luna: Oh, that's a classic. A flag that was temporary but stays in code for months, sometimes years. Lucas: Right. You should write tests that ensure when a flag is permanently set to a new value—like after a full rollout—you can safely delete the flag logic. The test should verify that the code works correctly without the flag, and that the flag's fallback value is no longer used. Also, schedule periodic flag audits: tests that list all active flags and flag code paths that haven't been exercised recently. Luna: So it's not just about testing the flag, but testing the lifecycle. Lucas: Exactly. And there's a specific technique called 'flag toggling tests.' You run your entire test suite twice: once with the flag on, once with the flag off. Then compare results. Any test that passes in one state but fails in the other is a red flag—pun intended—that the feature couples to the flag in an unexpected way. Luna: That can be expensive, running the suite twice. But for critical flags, it's worth it. Lucas: It is. You can optimize by running only a subset: the tests that cover code paths influenced by the flag. Use code coverage to identify those test classes. Also, you should test the flag evaluation performance. If your flag service is centralized, the latency of a flag check can add up. Luna: True. I've seen cases where every button click triggered three or four flag calls. That kills performance. Lucas: So performance test with realistic flag call loads. Use a profiler to measure flag evaluation time. And test that your application handles flag service timeouts gracefully—maybe the fallback should default to a cached value rather than a network call. Luna: Let's talk about version control. How do you handle flags across different branches and deployment environments? Lucas: Great question. The flag definitions themselves should be version-controlled, ideally in a separate repository or as configuration files. Your tests should confirm that each environment—dev, staging, production—has the correct flag configuration. For example, a flag meant for internal testing should never be on in production. Automated environment validation tests can catch that. Luna: So a flag that's on in staging but accidentally on in production would be caught? Lucas: Yes. You can write a test that queries the production flag service and asserts that certain flags are off. You can even set up a webhook that triggers an alert if a flag's production value deviates from expectations. Luna: That seems like a safety net. Any final advice for teams starting to test feature flags? Lucas: Start small. Pick one critical flag, implement the tests we discussed—fallback, targeting, performance, lifecycle—and automate them. Then expand. The key is to treat flags as production code that requires the same rigor as any feature. Luna: Good advice. Thanks for the deep dive. Lucas: Thanks for listening. Now you know how to keep those switches from ruining your day.