Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Uses Long-Running Experiments to Test Product Hypotheses
Transcript
- Lucas: You've probably been in a sprint planning meeting where someone says, 'We should build this feature, it's what customers want.' And you nod, because the data is thin but the intuition feels right. Luna: And six months later you've shipped something nobody uses. We've all been there. Lucas: Exactly. So today I want to talk about a countermeasure that a CTO I know has been refining for the past two years. Her name is Sarah Chen, she runs engineering at a mid-market SaaS company called FlowWorks — about 80 engineers. And she's become obsessed with what she calls long-running experiments. Luna: Define 'long-running.' Are we talking two-week A/B tests or something bigger? Lucas: Bigger. She runs experiments that last 30 to 90 days. The idea is that most product decisions are based on short bursts of data — a spike in signups after a launch, a dip in retention after a redesign. But those patterns often flip once the novelty wears off. So she designs experiments to survive the novelty effect. Luna: That is a practical insight. And if today's conversation gave you something usable, the way these episodes stay ad-free is listener support. You can buy me a coffee dot com slash fexingo. It's a small way to keep the show going without sponsors. Lucas: Yeah, that makes a real difference. Okay, back to Sarah. Her most instructive experiment ran 90 days and tested a new pricing tier — a 'team' plan that sat between the basic individual plan and the enterprise plan. The hypothesis was that small teams were churning because the individual plan was too limited but the enterprise plan was overkill. Luna: And how did she design the experiment? Because just launching a new plan and tracking revenue is not really an experiment — it's a launch. Lucas: Right. She ran it as a controlled rollout. She randomly selected 5% of customers who were on the individual plan and showed them an upgrade prompt to the team plan. Another 5% she didn't show anything — that was the control. Then she tracked not just conversion rates, but also support ticket volume, feature requests, and churn over the next three months. Luna: So she was looking for signals beyond revenue. Smart. Lucas: Exactly. And what happened? In the first two weeks, conversion looked great — about 8% of the treatment group upgraded. But by week six, that had dropped to 3%. The early adopters were the ones who would have paid anyway. The rest needed more time to see the value. Luna: So if they'd stopped the experiment at two weeks, they'd have overestimated demand by more than double. Lucas: That's the key insight. But the really interesting data came from support tickets. The team plan included a shared workspace feature, and support tickets for that feature spiked in weeks three and four — users were confused about permissions. That gave Sarah's team a clear priority: fix the permissions UX before rolling out to the full user base. Luna: So the experiment didn't just validate the pricing hypothesis, it also surfaced a critical product improvement. That's the kind of return that justifies a 90-day test. Lucas: Yeah. And there's a second example I want to touch on. She ran a 60-day experiment on a new onboarding flow that reduced the number of steps from seven to four. The short-term data was amazing — completion rates jumped 40% in the first week. But by day 45, they noticed that users who went through the new flow had a 15% higher churn rate at 60 days. Luna: Because the shorter flow skipped some education steps that users needed later. Lucas: Exactly. The quick win masked a long-term cost. Without the full 60-day window, they would have shipped a change that looked great in every sprint review but actually hurt retention. Luna: So what's her framework for deciding which experiments get the long treatment? You can't run everything for 90 days. Lucas: She has three criteria. First, the decision has to be strategic — something that would require at least a quarter of engineering work to build or change. Second, there has to be a plausible mechanism for the novelty effect — meaning you suspect early behavior might not persist. Third, the cost of being wrong is high — either in customer trust or engineering rework. Luna: That makes sense. Low-risk A/B tests on button colors still run for a week. But when you're considering a new pricing model or a fundamental UX change, you invest in the longer timeline. Lucas: Right. And she also sets kill criteria upfront. For the pricing experiment, she said if after 45 days the conversion rate was below 2% with a confidence interval that didn't overlap with the cost of building the feature, she'd kill it. That way the experiment doesn't just drift because people are invested. Luna: How does she handle the tension between engineering wanting to build and the product team wanting faster experiments? Lucas: She turns it into a shared accountability structure. The engineering team owns the experiment's infrastructure — the feature flags, the data pipeline, the dashboards. The product team owns the hypothesis and the decision criteria. And both teams review the experiment together at the halfway point. Luna: So if the data is already trending toward kill, engineers don't feel like their work is being wasted because they see the data too. Lucas: Exactly. And she budgets for experiments explicitly. About 15% of each quarter's engineering capacity is reserved for experiments that may or may not ship. That's a hard sell to the CEO, but she uses past experiments to justify it — like the time a 60-day experiment saved them from building a whole feature that only 2% of users would have used. Luna: That's a powerful number. If you can show the CEO that one experiment prevented millions in wasted engineering, the 15% becomes easier to defend. Lucas: And that's exactly what happened. That feature would have taken four engineers three months — roughly 240 thousand dollars in salary and benefits. The experiment cost maybe two weeks of a senior engineer's time to set up the feature flag and dashboard. Luna: So the ROI on the experiment itself was enormous, not to mention the ongoing value of the skill. Lucas: I think the broader lesson is about humility. Long-running experiments force you to admit that you don't know the answer in the first two weeks. And that's actually a strength, not a weakness. Luna: But doesn't it slow down the organization? If every strategic bet takes 90 days to validate, you're moving at a pace that might not work for a startup racing to market. Lucas: That's a fair tension. Sarah's take is that you only need long experiments for the bets where the cost of being wrong is high. For a startup that needs to find product-market fit, speed matters more. But once you have a product and a customer base, the cost of breaking trust or wasting engineering time goes up. Luna: So it's stage-dependent. A 20-person startup might run two-week experiments. A 200-person company probably needs longer windows. Lucas: Right. And Sarah also points out that long experiments don't have to be all-or-nothing. You can start with a low-fidelity test — like a concierge MVP where a human manually does the work — and then scale up to a full technical experiment once you have early signal. Luna: So the experiment itself is iterative. You learn, refine, then invest more. Lucas: Exactly. And I think that's the mental model that separates Sarah's approach from a typical A/B test. She's not looking for a binary ship or kill answer on day 90. She's looking for a richer understanding of user behavior that informs the whole product roadmap. Luna: One thing I'm curious about — how does she handle the data analysis? Because longer experiments mean more data points, but also more noise from external factors like seasonality. Lucas: She uses a technique called 'difference-in-differences.' You compare the change in the treatment group to the change in the control group over the same period. That cancels out things like a seasonal spike in signups that affects both groups equally. Luna: So it's not just a before and after comparison. You're isolating the treatment effect. Lucas: Right. And she also pre-registers the analysis plan — she writes down exactly which metrics she'll look at, and how she'll interpret them, before the experiment starts. That prevents cherry-picking results after the fact. Luna: That's a practice borrowed from academic science. Pre-registration keeps you honest. Lucas: It does. And her team has a template for it. They spend a day writing the hypothesis, the metrics, the minimum detectable effect, and the kill criteria. Then they share it with the whole product and engineering team before flipping the flag on. Luna: That also builds buy-in. Everyone knows what success looks like before the data starts coming in. Lucas: Exactly. And I think that's where the real value is. It's not just about the data — it's about creating a culture where decisions are made transparently, with a shared understanding of uncertainty. Luna: So if a listener wants to try this, what's the first experiment they should run? Lucas: Pick something you're already planning to build in the next quarter — a feature that has some uncertainty. Set up a feature flag that only 5% of users see. Define your success metrics and kill criteria in a one-page doc. And commit to running it for at least 30 days before making a decision. You'll learn more about your users in those 30 days than in a month of user interviews. Luna: That's solid advice. I think a lot of teams will find that the discipline of the experiment is as valuable as the result. Lucas: Yeah. And if you do run one, let us know how it goes. We'd love to hear about it.