Latest / MBA Training Marketing / Marketing mix modeling: why holdout tests beat dashboards
Transcript
- 0:00Welcome to the Leaders Insights podcast.
- 0:07Today's episode, Marketing Mix Modeling and Experimentation Culture, how CMOs build the
- 0:13evidence machine.
- 0:14Most CMOs I talk to spend more time defending their budget than growing it.
- 0:18Why is that?
- 0:19Because they walk into the finance meeting with a story instead of a machine.
- 0:24They say the brand campaign lifted awareness, and the CFO, the chief financial officer,
- 0:30the person holding the purse, nods politely and cuts 15% anyway.
- 0:35Story loses to spreadsheet every time.
- 0:38So the fix is marketing mix modeling.
- 0:40But isn't that just an expensive way to confirm what you already believe?
- 0:44That's the trap, and it's a real one.
- 0:47Marketing mix modeling, gwer, the statistical technique that estimates how each channel
- 0:51contributes to sales, is only as honest as the person feeding it.
- 0:55I've seen teams tune the model until TV comes out as the hero because the head of TV signs
- 1:01off on the budget.
- 1:02That's not evidence.
- 1:04That's astrology with a regression line.
- 1:06Strong words.
- 1:07So what separates the real thing from the astrology?
- 1:11Experiments.
- 1:12This is what everyone gets wrong.
- 1:14They treat modeling as the answer.
- 1:16Modeling tells you where the money worked, roughly.
- 1:19It cannot tell you why, and it definitely can't tell you what happens if you double the
- 1:23spend tomorrow.
- 1:24For that, you run controlled tests.
- 1:27Oh, duh-oh-co-weed.
- 1:29You hold out a region.
- 1:30You turn a channel off in 10 markets.
- 1:32You watch what breaks.
- 1:34Give me a real example.
- 1:35Not a hypothetical.
- 1:36Airbnb.
- 1:38Around 2019, their model said search advertising drove a mountain of bookings, so they ran
- 1:43a geo holdout, switched off paid search in a chunk of markets, and measured the gap.
- 1:49Find out a big share of those bookings would have happened anyway.
- 1:52People typing Airbnb were already sold.
- 1:56The model gave paid search credit for a sale it didn't cause.
- 2:00They redirected serious money to brand marketing off the back of that.
- 2:04No experiment, no discovery.
- 2:06But most companies aren't Airbnb.
- 2:09They don't have a data science army.
- 2:11You don't need one to run a holdout.
- 2:13You need the nerve to turn something off.
- 2:15That's the actual bottleneck.
- 2:16Not tooling, courage.
- 2:19Every marketer is terrified that if they pause a channel and numbers hold steady, they've
- 2:23just proven their own job was optional.
- 2:26There's the discomfort.
- 2:27Is that fear justified?
- 2:29Occasionally, yes, and that's the funny part.
- 2:32The experiment that saves the budget is also the one that might quietly fire you.
- 2:37But the CMOs who grow budgets are the ones who'd rather know.
- 2:41HubSpot puts marketers running regular experiments at meaningfully higher return on ad spend,
- 2:47though worth noting they sell marketing software.
- 2:49So cross-check that against independent work.
- 2:53And MIT Sloan Management Review has been blunt for years that firms with a genuine testing
- 2:58habit outperform the ones drowning in dashboards they never act on.
- 3:02Dashboards get a lot of love.
- 3:04You're saying they're mostly theater?
- 3:06Mostly.
- 3:07A dashboard tells you what happened.
- 3:09It has no opinion on causation.
- 3:11On to you on what actually caused what.
- 3:14You can stare at a beautiful chart of website traffic climbing and have no idea whether
- 3:18your campaign did it or a competitor imploded.
- 3:21SEMrush will happily show you traffic curves all day.
- 3:25Again, they sell the analytics tools, so treat their benchmarks as marketing, not gospel.
- 3:31The number on the screen isn't proof.
- 3:33The test is proof.
- 3:35So modeling plus experiments.
- 3:37How do they actually talk to each other?
- 3:39The model generates hypotheses.
- 3:41Podcasts look underpriced.
- 3:43We're getting more than we pay for.
- 3:46The experiment settles the bet.
- 3:51You buy more podcast slots in half your markets and see if sales follow.
- 3:56Then you feed the result back into the model.
- 3:59So next quarter's estimate is grounded in something you tested, not something you
- 4:03assumed.
- 4:04It's a loop.
- 4:06Each side covers the other's blind spot.
- 4:08If someone's building this from zero on Monday, where do they start?
- 4:12Make one channel you're convinced is working.
- 4:14Then turn it off.
- 4:15Bowl you in a handful of markets for four weeks, nowhere else.
- 4:19If sales don't move, you just found money you were setting on fire.
- 4:23If they drop, you've got the cleanest budget defense evidence you'll ever own.
- 4:27One channel, one holdout.
- 4:29That's the whole starter kit.
- 4:31Turn something off to prove it matters.
- 4:33I'll take it.
- 4:34Sources for today's episode.
- 4:36DigiDay SEMrush Vendor, SEO Analytics Tools, MIT Sloan Management Review, HubSpot Vendor,
- 4:45CRM Marketing Automation.
- 4:48That's it.
- 4:49The CMO Decision Tools are live at MBA-training.com.