Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses Feature Flags to Deploy Daily
Transcript
- Lucas: Let's talk about feature flags. Not just the concept, but how one CTO at a mid-size SaaS company — I'll call her Sarah — uses them to deploy to production every single day, even when the feature she's building is weeks away from being ready. Luna: I love this because so many teams still treat deployment and release as the same thing. Like, we ship code and it immediately hits users. That's terrifying for anything complex. Lucas: Exactly. Sarah's team was doing two-week sprints with a release at the end. Every merge into main was a big bang. So she moved them to trunk-based development with feature flags as the safety net. Now they merge small changes multiple times a day, but the feature stays dark until they flip the flag. Luna: Can you give me a concrete example of something they flagged? Lucas: Sure. They redesigned their entire payment flow — new UI, new third-party provider integration, new error handling. That's a high-risk change. Sarah had the team wrap the entire new flow in a feature flag called 'checkout-v2'. For six weeks, every commit went into main behind that flag. The old flow ran for all users. Luna: Six weeks is a long time. How did they avoid the flag getting stale or the code diverging too much from the old flow? Lucas: Great question. That's the classic risk with long-lived flags — flag debt. Sarah's rule is: no flag lives longer than one quarter. And she has an internal dashboard that tracks the age of every flag, who created it, and whether it's still referenced in code. Every sprint, the team spends one story point on 'flag hygiene' — removing flags that are fully rolled out or killing experiments that failed. Luna: So they treat cleanup as work, not an afterthought. That's smart. I've seen teams with hundreds of dead flags in the codebase, just rotting. Lucas: That's exactly what she wanted to avoid. Her dashboard shows 47 flags currently live. Average age is 23 days. Anything older than 60 days gets flagged in the standup. The team also uses a naming convention: prefix with the feature name, then the Jira ticket number. So 'checkout-v2-FL-342'. That way any developer can trace a flag back to the original decision. Luna: What about operational overhead? Managing a flag service, evaluating toggles at runtime — that adds latency and complexity. Lucas: Sarah uses a lightweight open-source flag evaluation library — just a couple of milliseconds per check. And they cache the flag state aggressively, so it's not a database call every time. The bigger cost is cognitive: developers have to remember to check flags in their code. But that becomes muscle memory after a few weeks. Luna: So the payoff is really about reducing release anxiety. You can deploy a broken flag, fix it in the next commit, and no one notices. Lucas: Exactly. Sarah told me that before flags, her team had three release days per month where everyone was stressed, testing manually, doing rollbacks. Now they deploy every day, often multiple times, and the only time they have a 'release event' is when they flip a flag for a percentage of users. That's a five-minute meeting, not a four-hour freeze. Luna: And flipping for a percentage — they use gradual rollouts, right? Lucas: Yes. Their flag system supports targeting by user ID, region, account tier. For the payment flow, they started with 1% of internal users, then 5% of beta customers, then 25%, 50%, 100%. Each step monitored error rates and conversion. If something went wrong at 5%, they just flipped back to 0% — no rollback, no revert commit. Luna: That's the dream. But I wonder — does this change the conversation with the business? Because now 'when does the feature ship' becomes a different question. Lucas: It does. The product team used to ask 'when will it be done?' Now they ask 'when can we turn it on?' And the answer is: as soon as the data says it's safe. That shifts the focus from calendar dates to measurable outcomes. Sarah's VP of Product actually loves it because they can do longer-running experiments without holding up other deploys. Luna: I want to talk about the dark side. What about flags that are used for A/B testing — those can live for months. And what about flags that control access to sensitive features, like billing changes? Lucas: Sarah has a separate category for 'permanent flags' — things like a toggle for a legacy API endpoint that they're slowly deprecating. Those get a different dashboard view and stricter review. Any flag that controls billing or data access requires two approvals before it can be flipped to production. And flags themselves are audited — every flip is logged with timestamp and who did it. Luna: So there's governance, not just a free-for-all. Lucas: Right. The key is that the flag system is a tool, not a magic wand. If you don't have discipline around cleanup and naming and access control, you just create a new kind of mess. But Sarah's team has been doing this for two years now, and their deploy frequency went from twice a month to 47 times a month. Their rollback rate dropped to near zero. Luna: Forty-seven deploys a month. That's basically every working day. Lucas: Yeah. And the interesting thing is that the developers report lower stress. They know that if they break something, they can fix it in the next deploy — no one has to wait two weeks for the next release train. That psychological safety is a huge win. Luna: Alright, so if a CTO is listening and wants to start using feature flags, what's the first step? Should they buy a tool or build their own? Lucas: Sarah says: start with a simple open-source library. Don't over-engineer it. Pick one low-risk feature — maybe a UI change that only internal users see — and wrap it in a flag. Prove the workflow works. Then add the dashboard and the cleanup process. The tool is the easy part; the culture change is the hard part. Luna: That feels right. And it's the kind of thing where you can't really appreciate the benefit until you've done it once. Lucas: Absolutely. And look, this whole conversation — the specifics, the framework — is exactly the kind of practical knowledge we try to share on this show. We deliberately don't run ads on these. If you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep this ad-free and focused on the stuff that actually helps you do your job better. Lucas: So back to flags — one last point. Sarah's team now treats every new feature as a flag by default. They have a template in their ticket system that requires a flag name before any code is written. That bakes the practice into the workflow. Luna: That's a great forcing function. It's like saying 'if you can't name the flag, you haven't thought about the rollout plan.' Lucas: Exactly. And it makes the rollout conversation happen at the design stage, not the night before launch. That alone probably prevents half the late-night emergencies. Luna: Alright, I'm convinced. I'm going to look at my team's flag hygiene tomorrow. Lucas: Good luck. And if you find a flag older than your youngest developer, you know what to do.