Latest / The CMO Podcast with Fexingo: Marketing Leadership, Budgets, and Executive Strategy / Why CMOs Are Using AI Agents to Run A-B Tests
Transcript
- Lucas: So Luna, I have been digging into how CMOs are running experiments these days, and there is a shift happening that is pretty hard to ignore. Luna: Okay, I am listening. What is changing? Lucas: Well, for years, the standard was a human-led A/B testing cycle. A marketer comes up with a hypothesis, designs a test, runs it for a couple of weeks, analyzes the results, and then implements the winner. That whole loop might take a month or more. Luna: Right, that is the classic model. Painfully slow but methodical. Lucas: Exactly. But now, a growing number of CMOs are handing the wheel to autonomous AI agents. These agents can run hundreds of simultaneous A/B tests across different customer segments, pages, and even channels, all without a human in the loop for every single decision. Luna: Wait, so the AI is not just analyzing the data; it is deciding what to test next? Lucas: That is the key distinction. It is not just a dashboard that shows you results. It is an agent that proposes the test, allocates traffic, and then, based on the outcome, either scales the winning variant or drops the loser and launches a new test. Some platforms even tinker with the variants during the test using multi-armed bandit algorithms. Luna: So it is like having a tireless, super-fast experimenter on the team. What kind of results are people seeing? Lucas: I spoke with the CMO of a mid-sized e-commerce company—about $200 million in revenue—that adopted an experimentation platform from a company called Eppo. Before, they were running maybe two or three tests per month. After switching to an ai driven approach, they were running over sixty tests simultaneously. Luna: Sixty at once? How do they even manage that many? Lucas: That is the point—they don't. The AI manages it. The human role shifts to defining the guardrails: what metrics matter, which customer segments to include, and the minimum detectable effect they care about. Then the AI just goes. Luna: And did it move the needle on revenue? Lucas: Yeah, it did. Over six weeks, they saw a 12 percent lift in conversion rate. That is massive for a company that size. Though, full disclosure, that number includes some pretty low-hanging fruit they had been neglecting. But still, the speed of learning was unprecedented for them. Luna: That is impressive. But I have to ask—are there risks here? Like, what if the AI optimizes for a short-term conversion at the expense of long-term brand equity? Lucas: That is a real concern. If the AI only has a signal like 'click-through rate' or 'immediate purchase,' it could easily push you toward aggressive tactics that hurt customer lifetime value. A classic example: a pop-up that offers 20 percent off if you buy now might win on conversion, but it trains customers to wait for discounts. Luna: Exactly. So you need to give the AI the right objective. Not just any short-term metric. Lucas: Right. The smartest CMOs are including long-term metrics in the AI's objective function—like predicted lifetime value or repeat purchase rate within 90 days. Some are even using a proxy like the number of email sign-ups or app opens, but you have to be careful. Luna: So the human still sets the strategy. The AI executes the tactics. Lucas: That is the ideal state. But I have seen cases where the AI runs wild because the guardrails were too loose. For example, one company let the AI test discount amounts without a floor, and it quickly found that a 50 percent discount drove huge conversions—but the margins collapsed. Luna: Ouch. So you need to constrain the action space as well. Lucas: Exactly. Set boundaries on the variables the AI can change. You might say, 'only test discount between 10 and 25 percent,' or 'only test headline copy, not the entire value proposition.' Luna: That makes sense. And speaking of learning, if you are finding these conversations useful, maybe think about supporting the show. It is listener support that keeps us ad-free and focused on real cases like this. Lucas: Yeah, it is a small way to say this kind of content matters to you. You can find us at buy me a coffee dot com slash fexingo. No pressure, just if it has sparked something you use. Luna: Exactly. Anyway, back to those AI agents—one thing I find fascinating is how this changes the skill set for marketing teams. Do you need data scientists now to run marketing? Lucas: Not necessarily, but you do need a new kind of marketing ops person. Someone who understands experimentation design, even if they are not coding the AI. They need to know things like statistical significance, sample size, and the difference between frequentist and Bayesian approaches. Luna: Right. Because the AI might use one or the other, and that affects how you interpret the results. Lucas: Exactly. And I think the bigger shift is cultural. It is not just about the tool; it is about creating a 'test and learn' culture. The CMO has to champion that. If the CEO is impatient and wants instant results, they might kill an experiment too early. Luna: True. Patience is a luxury in most companies. But the companies that do it well—like Booking.com, which runs thousands of tests a year—they treat experimentation as a core competency. Lucas: Booking.com is the poster child. They have been at this for over a decade. They even have a dedicated experimentation platform team. And they test everything: from the color of a button to the order of search results, to the wording of cancellation policies. Luna: And they found some surprising things, right? Like, changing the default sorting from 'best match' to 'price low to high' actually decreased bookings because people felt overwhelmed by cheap options. Lucas: Exactly. That is exactly the kind of insight you only get from rigorous testing. And now, with AI, you can scale that across more dimensions. But I worry that smaller teams might feel left behind. Luna: Why? The tools are becoming more accessible. You don't need a huge data science team anymore. There are platforms like Google Optimize, Optimizely, and VWO that have AI features built in. Lucas: That is true. But the barrier is still the culture. You need a CMO who is willing to say, 'We will learn from our failures.' And that is hard in a world where everyone wants to show only wins. Luna: Agreed. So what is your prediction? In five years, will all A/B testing be ai driven? Lucas: I think most of the routine testing will be. The kind of tests where you are optimizing for a known metric. But the truly novel, high-risk hypotheses—those will still need human creativity. The AI can tell you what works, but it cannot tell you what is worth trying. Luna: So humans for the 'why,' AI for the 'what.' Lucas: Exactly. And that is a pretty good partnership, if you ask me.