Latest / The CMO Podcast with Fexingo: Marketing Leadership, Budgets, and Executive Strategy / How CMOs Use Geo-Experiments to Prove Ad Impact
Transcript
- Lucas: So there's a stat that's been rattling around my head since I read it: Walmart ran a series of geo-experiments and found that they could reallocate up to 20 percent of their national TV budget without hurting sales. Twenty percent. That's not a rounding error. Luna: That's huge. But for anyone who hasn't heard the term, what exactly is a geo-experiment? Lucas: Right. So the basic idea is that you split your market into different geographic regions, randomly assign them to a treatment or a control, and then you run different ad spend levels in each. Because the regions are chosen randomly, you can attribute any difference in sales to the ads themselves, not to some other factor like a competitor's promotion or a viral trend. Luna: So it's like an A/B test but for entire cities or TV markets. Lucas: Exactly. And that's what makes it so powerful. The problem with most other forms of attribution, like click-based tracking or even marketing mix modeling, is that they're correlational. They tell you that sales went up when you spent more, but they can't tell you whether that's because of the ads or because of something else that just happened to move at the same time. Luna: And that's where geo-experiments give you actual causation, not just correlation. Lucas: Precisely. And it's not a new idea – geo testing has been around in some form for decades. But it used to be incredibly expensive and slow. You'd have to coordinate with broadcasters, set up panels, wait months for results. Now, with better data and automation, a lot of companies are doing this almost in real time. Luna: Can you give a concrete example? Like, what does a geo-experiment actually look like in practice? Lucas: Sure. Take a national coffee chain – I'm not going to name them, but think a large QSR with thousands of locations. They wanted to know if their TV ads were actually driving store visits or if they were just a vanity metric. So they ran a geo test across a dozen designated market areas, or DMAs. In half of them, they cut TV spend by 50 percent for eight weeks. In the other half, they kept it the same. By comparing foot traffic data from those regions, they found that the cut didn't hurt sales at all in most markets. In fact, in some areas, sales went up, probably because the ads were annoying people or were just noise. Luna: So they discovered they were wasting money on TV in certain regions. Lucas: Exactly. And they were able to redirect that money into digital channels or into the specific regions where TV actually moved the needle. That's the kind of insight that can save millions of dollars a year, not just in ad spend but in opportunity cost. Luna: And this is why CMOs are increasingly turning to geo-experiments as the gold standard for measuring incrementality. It's not just about cutting waste, it's about finding the places where your advertising is working and doubling down. Lucas: Right. And the thing is, a lot of CMOs already have the data to do this. They have store-level sales data, they have digital ad delivery data, they have weather data, they have competitor mapping. The challenge is usually not data, it's the experimental design and the analysis. Luna: So what are some of the common pitfalls? I imagine it's not as simple as just randomly splitting a map in half. Lucas: No, it's not. One of the biggest pitfalls is what statisticians call interference. That's when people in the control group are exposed to the treatment. For example, if you cut ads in one DMA but people there can still watch a national feed or see a social media ad that's running everywhere, then your control isn't really a control. Luna: So you end up comparing two groups that are both getting some of the same messaging. Lucas: Exactly. Another issue is seasonality. You have to make sure your test periods are aligned, and you may need to run the experiment for at least a full week or two, ideally longer, to capture repeat purchases and the lag effect of ads. A one-day test won't tell you much about whether a new customer became a loyal one. Luna: And then there's the analysis itself. It's not just about comparing averages, right? You need to account for variance. Lucas: Right. That's where a lot of people get tripped up. You can't just look at the difference in sales between regions and call it a day. You need to use a proper statistical model, like a regression with random effects or a Bayesian structural time series. Otherwise, you might mistake noise for a real effect. Luna: So this isn't something a marketing team can just do in a spreadsheet over a weekend. Lucas: No, but it doesn't require a PhD either. There are now platforms that handle a lot of the heavy lifting. You can input your geo spend and your outcome metric, and they'll run the experiment for you. But you still need to understand the principles to design a test that's actually valid. Luna: And that's what we're here for, right? To help CMOs and marketers understand these tools. You know, if you're getting value out of these conversations, I just want to mention that listener support is what keeps this show going. A couple of dollars a month genuinely makes a difference. You can buy us a coffee at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small thing that adds up. And we're glad to have you along. Luna: So, back to geo-experiments. We've talked about the pitfalls. What are some of the success stories beyond Walmart? Lucas: Uber is another great example. They've been using geo-experiments for years to measure the impact of their promotional offers and their brand campaigns. In one case, they wanted to know if a national brand campaign was worth the hundreds of millions they were spending. By running geo tests, they found that the campaign had a strong effect on new user acquisition, but almost no effect on existing user retention. So they were able to shift their messaging to focus on reactivation in the regions where retention was lagging. Luna: So it's not just about whether ads work, but who they work on. Lucas: Exactly. And that level of granularity is what makes geo-experiments so valuable. It's not just a yes-no answer. You can slice the results by region, by demographic, by time of day. That's where the real strategic insights come from. Luna: Now, you mentioned that geo-experiments are becoming more popular. What's driving that? Is it just the availability of better data? Lucas: Partly. But it's also the pressure on CMOs to prove ROI. The days of 'I know half my advertising is wasted, I just don't know which half' are over. CFOs want to see causal evidence, not just correlation. And geo-experiments are the closest thing we have to a causal gold standard in marketing. Luna: And with the phasing out of cookies, traditional attribution methods are getting weaker, so geo is becoming even more important. Lucas: Absolutely. When you can't track individuals across the web, you have to zoom out to the market level. Geo-experiments don't rely on cookies at all. They rely on aggregate sales data and ad delivery data. That's why they're becoming a cornerstone of the post-cookie measurement stack. Luna: So for a CMO thinking about starting with geo-experiments, what's the first step? Lucas: Start small. Pick one market where you have good data and where you have a clear question you want to answer. Maybe it's 'Does our brand campaign actually drive foot traffic?' or 'What's the ROI of our TV spend compared to digital?' Then design a test that can answer that question cleanly. You don't need to test everything at once. Luna: And you want to make sure you have a sufficient sample size, right? You can't run a geo test in a single city and expect to see a meaningful effect if your business is national. Lucas: Right. A general rule of thumb is that you need at least 20 to 30 DMAs for a reliable test. But you can start with fewer, as long as you understand the statistical limitations and you treat the results as directional rather than definitive. Luna: And there are also hybrid approaches, like geo-lift studies that combine geo-testing with customer surveys or digital measurement to get a fuller picture. Lucas: Exactly. Geo-experiments don't have to replace everything else. They can complement marketing mix modeling and even help calibrate your digital attribution. It's about building a measurement system, not just a single experiment. Luna: We've talked a lot about TV, but geo-experiments also work for digital channels, right? Lucas: Definitely. You can test different levels of Facebook ads, YouTube pre-roll, even OTT or connected TV. The key is that you need a way to control the delivery by geographic area. Platforms like Meta and Google allow you to target by DMA, so it's feasible. In fact, some of the most interesting work I've seen is using geo-experiments to compare the incrementality of social advertising against search advertising. Luna: So which one typically wins? Or is that too simplistic? Lucas: It's too simplistic, but the general finding is that search tends to be more incremental because people are already expressing intent. Social is more about creating demand, so the effect is often lower but still positive. The real insight comes from understanding the interaction – how social drives searches and how search captures that demand. Luna: That's a great point. So we're talking about a more sophisticated way of thinking about attribution, not just one number. Lucas: Right. And geo-experiments help you uncover those relationships. They're not a silver bullet, but they're a powerful tool in the CMO's arsenal. And I think we're going to see even more adoption as the tools become more user-friendly and as the pressure for accountability continues to grow. Luna: Well, I'm sold. I think every marketing team should at least run one geo-experiment this year, even if it's just a pilot. The insights are too valuable to ignore. Lucas: Couldn't agree more. And the beauty is that the knowledge you gain isn't just for one campaign. It compounds. Every experiment you run teaches you something about your market, your customers, and the real drivers of your growth. That's an asset that keeps paying dividends.