Latest / The CMO Podcast with Fexingo: Marketing Leadership, Budgets, and Executive Strategy / How Incrementality Testing Saves CMOs Millions
Transcript
- Lucas: I was looking at a study from the Marketing Analytics Institute this week, and one number jumped out. Companies that run systematic incrementality tests on their ad spend report an average 20 percent improvement in return on ad spend within two quarters. Luna: But last-click attribution already tells you what converted, right? Why do CMOs need a whole new methodology? Lucas: Because last-click gives credit to whichever channel happened to be the last touchpoint before a purchase. But that says nothing about whether that ad actually caused the purchase. A customer might have been going to buy anyway. Incrementality testing uses a holdout group — a randomly selected set of users who are not exposed to the ad. Then you compare their behavior to the exposed group. The difference is the true lift. Luna: So you're basically running a controlled experiment, like a clinical trial for your marketing budget. Lucas: Exactly. And the results can be brutal for some channels. I spoke with the CMO of a Fortune 500 beverage brand — let's call them 'ThirstCo.' They ran incrementality tests on their social media campaigns and discovered that nearly 40 percent of the conversions attributed by last-click were actually coming from people who would have bought anyway. Luna: Ouch. So they were basically paying to advertise to their own loyal customers. Lucas: Exactly. The incremental lift from those ads was barely 5 percent. They shifted $20 million out of social into search and retail media, and their overall return improved by 25 percent within a quarter. Luna: That's a huge shift. But how do you design a holdout group without angering customers who don't see the ad? Lucas: Good question. You randomize at the individual level, but you don't tell anyone. It's standard A/B testing. Most brands find that the holdout group doesn't even notice. And the lift you measure is net new customers or purchases, so it's worth the temporary opacity. Luna: I've also heard that incrementality testing works best for lower-funnel tactics. What about upper-funnel brand awareness? Lucas: It's harder, but possible. You can measure brand lift through surveys or search volume changes. But the gold standard is geographic holdouts — run the campaign in some metro areas and not in others, then compare sales. A national CPG brand did that with TV ads and saw a 12 percent lift in test markets versus control. Luna: So CMOs can't just rely on one attribution model anymore. They need a measurement stack. Lucas: Right. And the smart ones are building incrementality into their quarterly planning. They run a test, learn, reallocate, then test again. It's a continuous improvement loop. Luna: What about smaller companies that can't afford a huge data science team? Can they still do this? Lucas: Absolutely. There are now tools like Facebook's Conversion Lift, Google's Brand Lift, and third-party platforms like Measured and Adloox that do the heavy lifting. You just need a clean audience split and patience. Even a two-week test on a single channel can pay for itself. Luna: Let's talk about the elephant in the room: most CMOs are judged on last-click numbers. If you switch to incrementality, your reported ROI might go down before it goes up. Lucas: That's a real challenge. The CMO at ThirstCo told me she had to educate her CFO and CEO that a lower but accurate number is more valuable than a high but inflated one. She framed it as 'we're no longer paying for false positives.' Once they saw the budget shift improve actual sales, they were sold. Luna: So it's a change management problem as much as a measurement problem. Lucas: Exactly. But the data is compelling. The same study I mentioned at the top shows brands that adopt incrementality testing see a 15 to 25 percent reduction in customer acquisition cost within a year. That's hard to argue with. Luna: And I imagine it also helps with cross-channel synergy. If you know TV drives search, not just direct response, you can plan better. Lucas: Absolutely. Incrementality testing can reveal halo effects. One auto brand found that their podcast ads had no direct conversions, but a holdout test showed a 9 percent lift in branded search and a 4 percent lift in showroom visits. Without the test, they would have killed the campaign. Luna: That's a perfect example. So for CMOs listening, what's the first step? Should they pick one channel and test? Lucas: Start with your biggest spend channel — the one where you have the least certainty. Run a four-week holdout test. Don't look at last-click at all. Just compare sales or sign-ups between the exposed and control group. That one experiment will likely pay for itself many times over. Luna: And you'll probably get some humbling results. But that's the point. Lucas: Right. And if these kinds of marketing conversations have sparked something you've actually used, you know what keeps us digging into these topics without ads? Listener support. A couple of dollars a month at buy me a coffee dot com slash fexingo genuinely makes a difference. Luna: Yeah, it helps us keep bringing you these actionable insights. Every contribution goes straight into producing more episodes like this. Lucas: So where do you see incrementality testing going next? I think we'll see it become a standard part of every marketing plan within three years. Luna: I agree. Especially as privacy regulations limit third-party data, causal measurement becomes more critical. You can't rely on probabilistic models anymore. Lucas: Exactly. And with AI coming in, I think we'll see automated incrementality tests that run continuously in the background and reallocate budget in real-time. Some platforms are already beta testing that. Luna: That could be a game-changer. But it also means CMOs need to understand the methodology, not just trust the black box. Lucas: True. The best CMOs will always ask, 'What's the counterfactual?' That question alone can save millions. Luna: And that's a good note to end on. Thanks for joining us on The CMO Podcast.