Latest / MBA Training Marketing / Building an experimentation culture inside your marketing org
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:08Today's episode, Building an Experimentation Culture Inside Your Marketing Org.
- 0:12Most marketing teams say they test things.
- 0:15You're telling me that's basically worthless.
- 0:17Why?
- 0:18Because running an experiment once a quarter is like going to the gym on New Year's Day
- 0:22and calling yourself an athlete.
- 0:24The teams that actually compound their advantage run experiments constantly.
- 0:29Testing is the default state, not a special occasion someone schedules a meeting for.
- 0:34But everyone's resource constrained.
- 0:36Isn't test everything just a luxury for companies with money to burn?
- 0:40It's the opposite.
- 0:42Booking.com, the travel site, famously runs over a thousand concurrent experiments at
- 0:48any given moment.
- 0:49And they're not doing it because they're rich.
- 0:52They do it because a 1% lift on a checkout flow at their scale is millions of dollars.
- 0:57The discipline is what made them rich, not the reverse.
- 1:00Fine, but I can't be booking.com.
- 1:03What does experimentation culture even mean for a normal 30-person marketing team?
- 1:07It means two things.
- 1:10Infrastructure and incentives.
- 1:12Infrastructure so that launching a test doesn't require a two-week engineering ticket.
- 1:17Incentives so that being wrong isn't a career risk.
- 1:20If your best marketer gets punished the one time at test tanks, you've just taught
- 1:24everyone to never test anything ambitious again.
- 1:27That incentive piece sounds soft.
- 1:29How do you actually build it?
- 1:30You change what you celebrate.
- 1:32I ran a team where we tracked experiments shipped per person per month as a real number next
- 1:37to revenue.
- 1:39Not because activity equals results.
- 1:41It doesn't.
- 1:42But because if nobody's shipping tests, you have zero shots on goal.
- 1:46We went from maybe four experiments a quarter to about 40.
- 1:50Roughly a third one, a third were flat, a third lost.
- 1:53That's a healthy ratio.
- 1:55A third losing sounds like a lot of wasted work.
- 1:58If two-thirds of your tests win, your tests are boring.
- 2:01You're testing button colors instead of real hypotheses.
- 2:05The whole point of a low-cost experiment is you're allowed to be spectacularly wrong
- 2:10cheaply.
- 2:11The teams terrified of losing only ever test things they already know will work, which
- 2:16means they learn nothing.
- 2:17Give me the concrete example.
- 2:19What does one of these tests actually look like?
- 2:22At a subscription company I advised, someone had a heretical idea.
- 2:26Remove the free trial entirely and just charge day one.
- 2:29Every instinct said conversion would collapse.
- 2:32We tested it on 10% of traffic.
- 2:35Trial signups dropped, obviously.
- 2:37But paid conversions went up 18% because the people who paid immediately churned,
- 2:42whined.
- 2:43That's the share who cancel, far less.
- 2:47Nobody would have believed that in a slide deck.
- 2:50The experiment settled the argument in three weeks.
- 2:53So the experiment replaces the opinion of the highest paid person in the room.
- 2:56That's the real prize.
- 2:58In most orgs, decisions get made by whoever has the most seniority or
- 3:02the loudest voice.
- 3:03And experimentation culture is fundamentally a way of taking power away
- 3:07from hierarchy and giving it to evidence.
- 3:11That's why executives who talk about it don't always want it.
- 3:14It means their gut can be publicly proven wrong by a 26-year-old analyst.
- 3:18Isn't there a risk you optimize yourself into a local maximum?
- 3:22Endless tweaks?
- 3:23Never a big swing?
- 3:25Absolutely, and it's the most common failure.
- 3:28If all you do is small A-B tests, comparing version A against version B,
- 3:32you'll polish the same landing page for two years and
- 3:36miss that the whole channel is dying.
- 3:38The fix is deliberately reserving maybe 20% of your testing capacity for
- 3:42big risky bets.
- 3:44Kill the trial, change the pricing model, enter a channel everyone says is dead.
- 3:49What about the tools?
- 3:51Everyone's selling AI-powered experimentation platforms in 2026.
- 3:55Most of that is vendors dressing up dashboards.
- 3:58The AI can now generate variant copy and flag, which tests to prioritize,
- 4:03which genuinely saves time.
- 4:05But the tool never fixes the culture problem.
- 4:08They've seen companies buy a $200,000 platform and run six tests a year on it.
- 4:14The software isn't the bottleneck.
- 4:16The willingness to be wrong in public is.
- 4:18So if someone's driving home right now, what's the one thing they do Monday?
- 4:22Pick one decision this week that's currently being made by opinion.
- 4:26Pricing, a headline, an offer, and turn it into a live test instead of an argument.
- 4:31One.
- 4:32Then do it again next week.
- 4:34That's a wrap.
- 4:35Fresh CMO briefings drop daily at MBA-training.com.