Latest / Retail Media Breakfast Club / Why iROAS Metrics Are So Inconsistent (And Why Debate Is Good For The Industry)
Transcript
- This iROAS Research Got Pushback. The Authors Respond. === [00:00:00] Kiri Masters: The only thing worse than putting research out into the world and getting pushback is putting research out and hearing crickets. And last month I covered new research from Albertson's Media Collective EVA Group and [00:00:15] Northwestern's Kellogg School of Management for my column at the drum. And the headline finding from that research was that across 42 real campaigns I OAS could vary by an average of six and a half times, depending solely on [00:00:30] how the measurement was done. [00:00:31] In 83% of those campaigns, the result could flip from positive to negative just based on methodology and that piece. Really hit a nerve. Over 13,000 impressions on [00:00:45] LinkedIn of this post, which is above average, and a comment thread that turned into a real debate about measurement methodology, the kind that you don't always see in retail media [00:01:00] where a lot of discourse stays. [00:01:02] Quite polite and surface level. So I reached out to the research team and asked if they'd respond on the record, and they said yes right away. And And that willingness to engage rather than retreat is [00:01:15] definitely worth noting on its own. [00:01:16] [00:01:17] Kiri Masters: So let's jump into what the industry said. What were the questions and pushbacks on that article? [00:01:24] Here's what stood out from Professor Cohen Powells, who is a [00:01:30] distinguished professor of marketing at Northeastern University, and a former principal research scientist at. Amazon. He raised a methodological scope question. He pointed out that the three approaches compared in the study are quite different from one another, not the [00:01:45] minor tweaks that the headline might suggest. [00:01:47] And he asked why aggregate approaches like marketing mix modeling and geo experiments weren't included. It's a fair challenge and one that the authors. Chose not to address directly. And by the way, [00:02:00] um, Dr. Powell's writes a very good Substack newsletter that's worth subscribing to. I'll link up to it in the show notes. [00:02:07] The second, uh, thread that stood out was from Vanke Reman, who is the co-founder and [00:02:15] CEO of. RIMA Labs, he argued that BSTS, which is the Bayesian structural time series approach, isn't a causal method at all. That it's a forecasting algorithm being [00:02:30] misapplied, and that the studies conclusions are built on shaky foundations. [00:02:33] His comment drew the most engagement in the thread, including a respectful rebuttal from Moody Khan, who is the VP of RMN measurement Strategy at Sana, who [00:02:45] cited peer reviewed literature supporting BSTS for causal inference. I am. Not going to adjudicate that debate myself, but the fact that it's happening publicly with named [00:03:00] practitioners staking out positions is very healthy for an industry that usually keeps these arguments behind closed doors. [00:03:08] And finally, Dan Waldman, who is the senior technical product manager for ads reporting and measurement at Chewy [00:03:15] Ads, asked a question that may matter more for day-to-day practitioners than the methodological argument, which is, what are brands actually using IO? As for, is it proving that campaigns work or optimizing and [00:03:30] allocating budget in real time? [00:03:31] Those are different jobs and I oas, which is a lagging metric that can take weeks to reach statistical significance, is better suited to the first than the second, as in it is better [00:03:45] suited to proving that campaigns actually work. Miracle Ads is the only retail media solution designed for both one P [00:04:00] and three P Marketplace brands. Why does that matter? Marketplace sellers demand a seamless advertiser experience that still offers full funnel ad formats, and retailers need a flexible solution that [00:04:15] allows you to scale your media business. [00:04:17] Learn more@miracle.com. That's M-I-R-A-K l.com. [00:04:24] [00:04:25] Kiri Masters: I shared these threads with the research team and asked them if they'd like to respond, [00:04:30] and the collective statement back from them was clear on scope. The paper was never designed to evaluate which methodology is best. [00:04:39] It was designed to show that the same campaign when measured different ways, [00:04:45] produces wildly different numbers, and that most advertisers don't know which way their results were measured [00:04:51] on the critiques specifically, the authors pointed back to that framing. They weren't testing whether BSTS is a valid causal [00:05:00] method or whether media mix modeling belongs in the comparison. They were demonstrating that variation exists between those approaches that retail media networks actually use for campaign level post-campaign reporting. [00:05:13] And these are the reports [00:05:15] that brands and agencies receive and act on every day. Neither powers nor remand critiques fall within the scope, the paper set for itself and the authors chose not to engage with them directly, but [00:05:30] what they did engage with with was the practical question, what should a mid-market brand actually do with this information? [00:05:39] Liz Roche, who is the VP of Media and measurement at Albertson's Media Collective, said that [00:05:45] right now advertisers are being asked to reverse engineer me methodology, which isn't scalable, especially for mid-market brands. The shift we're advocating for isn't one standardized method, but minimum disclosure standards. [00:05:59] She [00:06:00] said the authors went a step further. What mid-market brands need isn't internal data science capability. It's collective leverage. The more that brands can ask basic questions like, [00:06:15] was the audience filtered before matching? What is the control group size? Has the methodology changed since our last campaign? [00:06:22] The more that brands can ask these questions, the more that networks are pushed to answer consistently, and the paper's, [00:06:30] appendix gives brands those questions ready to use with no statistician required. I get pitched a lot of research and white papers. [00:06:41] Most of it tells you what you already knew, but just wrapped [00:06:45] in a new data set or, or worse new branding. But this team did something different. They picked a specific uncomfortable question. They put real numbers behind it, and when the industry pushed back they, they engaged. [00:06:59] [00:07:00] No one paper is going to unravel retail media measurement, But this team isn't trying to eat the whole elephant. They're just working through it bite by bite. They did a piece called ROAS demystified last year. And [00:07:15] IRO as demystified this year. And at each step, giving brands and agencies something concrete to take away, and that is worth more than another panel where everyone agrees that measurement is broken and then goes back to their dashboards on Monday.[00:07:30] [00:07:30] Thanks for listening, and I'll catch you tomorrow. [00:07:33]