Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Uses Small Bets to Validate Hypotheses Before Building
Transcript
- Lucas: So there's this pattern I keep hearing from CTOs who've been burned by big-bang builds — they've started using what they call 'small bets' to validate a product hypothesis before committing serious engineering resources. Luna: Small bets — like, deliberately under-investing upfront to see if something sticks? Lucas: Exactly. And the key difference from a typical prototype is that a small bet has a specific falsifiable hypothesis and a go/no-go decision tied to real customer behavior — not just internal feedback. Luna: Give me a concrete example. What does that actually look like in practice? Lucas: Sure. I spoke with the CTO of a mid-size SaaS company — about 400 employees, B2B analytics platform. They wanted to add an ai powered natural-language query feature. The product team had a fifty-page spec, six months of estimated engineering time, roughly two hundred thousand dollars in fully loaded cost. Luna: Right, the kind of project that sounds amazing in a roadmap review but might not actually solve a real problem. Lucas: Exactly. So the CTO said — hold on. Instead of building the full thing, let's run a small bet. They gave a product designer and a senior front-end engineer two weeks and a budget of five thousand dollars. The goal: build a clickable prototype that looks and feels like the real feature, but with no backend — it's all mocked data and hardcoded responses. Luna: So it's basically a Wizard of Oz prototype — the user thinks they're talking to an AI, but actually there's a human or a script behind the curtain. Lucas: Right. But in this case, the prototype was smart enough that for maybe eighty percent of common queries, the mock responses looked real. Then they put that prototype in front of existing customers. They recruited twenty customers from their user community, gave them a demo, let them play with it, and asked one question: would you pay for this? Luna: And what happened? Lucas: Seventeen out of twenty said yes — and more importantly, when the CTO pushed them to put a dollar figure on it, the average was about fifteen percent above their current monthly subscription. So they had a strong signal that this was worth building. Luna: But that's the good outcome. What if only three had said yes? Then you've saved six months of engineering time and two hundred grand. Lucas: Exactly. And the CTO told me that's actually the more common outcome — about sixty percent of their small bets lead to a no-go. But that's the whole point. You want to fail fast and cheap, not slow and expensive. Luna: So what's the structure? How do you formalize a small bet so it's not just 'let's hack something together'? Lucas: The CTO I talked to uses a one-page template. Top of the page: the hypothesis — stated as 'We believe that will cause and result in.' Then: the smallest test that would validate or invalidate that hypothesis. Then: the go/no-go criteria — specific numbers, like 'at least 40 percent of customers in the test say they'd pay.' Then: the budget — time and money. And finally: the decision date. Luna: I love that the decision date is explicit. It forces a rhythm — you're not just meandering. Lucas: Yeah. And the budget is intentionally tight. The CTO said they cap small bets at two weeks and ten thousand dollars. If you can't learn what you need within that constraint, the hypothesis is probably too vague or too big. Break it down further. Luna: That makes me think about the organizational side. How do you protect the team from feeling like their work on a small bet is wasted if it's a no-go? Lucas: Interesting question. The CTO framed it as learning, not failure. They celebrate no-go outcomes in their demo day — teams present what they learned and why the hypothesis didn't hold. And they explicitly reward that. One of their engineering leads got a promotion after running three small bets that all resulted in no-go, because the data saved the company from three bad investments. Luna: That's a cultural shift. Most orgs reward shippng, not killing bad ideas. Lucas: Right. And the CTO said the hardest part was getting the product team to write a falsifiable hypothesis. Product managers are used to saying 'customers want X' — but that's not a hypothesis, that's an opinion. A hypothesis has to be something you can prove wrong with data. Luna: So what's an example of a bad hypothesis versus a good one? Lucas: Bad: 'Customers need a natural-language query feature.' Good: 'We believe that providing natural-language query will increase the percentage of users who run a self-serve report in their first session from 12 percent to 25 percent within 30 days.' That's testable. You can measure it. Luna: And if it doesn't move the needle, you kill it — even if the feature seems cool. Lucas: Exactly. And that's the discipline. The CTO also told me they use feature flags to roll out the full build gradually after a small bet passes. They don't go from prototype straight to full release — they do a beta with 5 percent of customers, measure, then 20 percent, and so on. Luna: So the small bet mindset continues even after you decide to build. Lucas: Yeah. The idea is you never really stop validating. You just increase the fidelity of your tests as you go. And speaking of fidelity, one thing that surprised me: the CTO said the prototype doesn't need to be pretty. In fact, they deliberately keep it rough to avoid over-investing in polish before proving value. Luna: That's hard for designers and engineers who take pride in craftsmanship. Lucas: Absolutely. But the CTO's argument is: craftsmanship on the wrong thing is waste. You can always make it beautiful later. The priority is learning, not shipping. Luna: I want to come back to the broader implications for engineering leadership. If you're a CTO listening, what's the first step to introducing this practice? Lucas: Start with one bet. Pick a feature that's been on the roadmap for a while but has high uncertainty. Give a small team two weeks and a clear hypothesis. And make sure the decision criteria are written down before anyone writes code. The CTO said the first bet is usually messy, but it teaches the team the discipline. Luna: Also, it helps if the CTO models the behavior — runs a bet themselves on something they're unsure about. Lucas: Yeah, lead from the front. This CTO ran the first three small bets personally, paired with a product manager and an engineer. That sent a signal that this was a serious practice, not a side experiment. Luna: It's interesting — this approach also changes the relationship between engineering and product. Instead of product handing over a spec and saying 'build this,' they're co-designing an experiment. Lucas: Exactly. And engineers get to contribute to the hypothesis design. Sometimes they'll say, 'We could test that with a much simpler thing — we don't need to build the whole machine learning pipeline, we can fake it with a lookup table.' That kind of creativity doesn't happen when you're just executing a spec. Luna: So what's the downside? Are there cases where small bets don't work? Lucas: Sure. If the feature is highly technical infrastructure — like migrating a database — you can't really fake that. But even then, you can test the migration strategy with a small subset of data. The CTO said they've applied the small bets framework to architecture decisions too, like 'will a read-replica reduce query latency enough to justify the cost?' They set up a test with a small percentage of traffic and measured. Luna: So it's really about de-risking any decision where you have uncertainty. Lucas: Right. And the CTO emphasized that the biggest risk isn't technical — it's building something nobody wants. Small bets attack that risk directly. Luna: I've been thinking about this a lot recently, actually — how much of what we build never gets used. You know, these conversations are only possible because listeners support the show. If you've gotten something out of episodes like this, a couple of dollars a month genuinely makes a difference. It's at buy me a coffee dot com slash fexingo. Lucas: Yeah, and we keep the show ad-free because of that support. So if you've ever learned something here and wanted to give back, that's the way. Okay — let's get back to the mechanics. One thing I didn't mention: the CTO also uses a 'bet registry' — a shared document where every small bet is logged with its hypothesis, budget, outcome, and key learnings. Luna: So it becomes an organizational memory. You can look back and see what you've tested. Lucas: Exactly. And they review the registry quarterly to spot patterns. For example, they noticed that features targeting new customer segments had a much higher no-go rate than features for existing segments. So they adjusted their strategy to focus more on retention and expansion. Luna: That's a huge insight you wouldn't get from a typical roadmap review. Lucas: Right. And the CTO said the registry also helps with stakeholder management. When a VP of Sales asks for a big feature, the CTO can say, 'Great idea — let's run a small bet on it. We'll have data in two weeks.' That changes the conversation from opinion to evidence. Luna: It also buys time. Instead of saying no, you're saying 'let's learn.' Lucas: Exactly. And the VP of Sales usually appreciates that because they get a faster answer than waiting six months for a full build. Luna: So if someone wants to start this tomorrow, what's the one thing they should do? Lucas: Write a hypothesis for the next feature on your roadmap. Make it falsifiable. Share it with your team. Ask: what's the cheapest way to test this in two weeks? And set a decision date. That's the first small bet. Luna: I like that. It's a low-risk entry point. And if it works, you build from there. Lucas: Yeah. And if it doesn't, you've learned something about your process. Either way, you're better off than before. Luna: Alright, I'm going to try that with our next feature idea. Lucas: Let me know how it goes. I might run a small bet on that conversation.