Latest / Marketing Automation with Fexingo: HubSpot, Marketo, and Automated Customer Journeys / How Marketing Automation A-B Tests Hide Sample Size Traps
Transcript
- Lucas: So you run an A/B test on your email subject line. HubSpot or Marketo tells you after 48 hours that variant B has a 95 percent confidence level. You declare B the winner and send it to the rest of your list. Feels good, right? Luna: I have definitely done that. More than once. Lucas: We all have. But there's a pretty good chance that decision was wrong. Not because the platform lied — but because the test didn't have enough statistical power to begin with. Luna: Statistical power — that's the probability of detecting an effect when there actually is one, right? Not just the confidence level. Lucas: Exactly. The confidence level tells you how likely it is that the result isn't random noise. But if your sample is too small, even a 99 percent confidence level can be misleading. You're basically measuring a coin flip that happened to land heads a few times. Luna: So what's the practical trap? A marketer launches a test, sees a winner, and rolls it out to the entire list — but the initial sample wasn't big enough to detect the real effect. Lucas: That's the trap. Let me give you a concrete example. A mid-market SaaS company I worked with tested two subject lines on a 12,000-person list. After 24 hours, variant A had a 3.2 percent open rate, variant B had 3.8 percent. The platform said 94 percent confidence. They declared B the winner and sent it to the remaining 11,000 people. Luna: And what happened? Lucas: Final open rate on the full send? 3.3 percent. The difference vanished. The initial 24-hour window had just 384 opens total. That's tiny. The apparent lift was noise. Luna: So the platform gave them a false sense of certainty. Why do platforms default to that? Lucas: Short answer: they optimize for speed, not accuracy. HubSpot and Marketo both calculate confidence intervals using a standard formula — but they don't check whether your sample size is large enough to detect the effect size you're looking for. They just report whatever the math says, even if the math is unreliable at low volumes. Luna: So the marketer sees '95 percent confident' and thinks, done deal. But the underlying data is thin. Lucas: Right. And the fix isn't complicated. Before you launch any A/B test, you should calculate how many contacts you need per variant. That number depends on three things: your baseline conversion rate, the minimum effect you care about, and your desired statistical power — usually 80 percent. Luna: Walk me through the SaaS example. What would they have needed? Lucas: Their baseline open rate was about 20 percent. They wanted to detect a 10 percent relative lift — so from 20 percent to 22 percent. With 80 percent power and 95 percent confidence, they'd need roughly 7,000 contacts per variant. They had 6,000 per variant. Close, but not quite. Luna: So they were borderline. But the early results made them act too fast. Lucas: Exactly. If they'd waited until they had closer to 14,000 total opens, the result might have held. But they stopped at 384 opens. That's the sample size trap in action. Luna: You know, this reminds me of something I read about how many marketing teams treat A/B testing like a slot machine — pull the lever, see if you win, move on. No pre-test calculation. Lucas: That's the right analogy. And the fix is genuinely easy. There are free online calculators — just Google 'A/B test sample size calculator'. Plug in your baseline, your minimum detectable effect, and it tells you the sample size you need. Then you know whether to wait or whether you can call it. Luna: One thing I wonder: do platforms ever warn you? Like, does Marketo say 'hey, your sample is too small'? Lucas: Not really. HubSpot's A/B test report shows a confidence level, but it doesn't flag low power. Marketo's is similar. They assume the user knows what they're doing. Which, to be fair, most marketers don't — because statistics isn't their job. Their job is to get the email out. Luna: So the responsibility falls on the practitioner. What's the one rule you'd give listeners to avoid this trap? Lucas: Calculate required sample size before you launch. If you can't hit that number within a reasonable timeframe, don't run the test. Or, lower your minimum detectable effect. Or accept that the result is directional, not conclusive. Luna: Directional meaning you use it as a signal, but you don't bet the whole campaign on it. Lucas: Exactly. You send variant B to 10 percent of your list, see a lift, and roll it out — but you keep tracking to confirm. That's a reasonable approach. The problem is when you treat a 48-hour result as gospel. Luna: I want to bring up a related issue: multiple comparisons. If you're testing three subject lines at once, the chance of a false positive goes up. Lucas: That's a whole other episode. But briefly — if you test three variants, your effective confidence level drops. A simple correction is to divide your alpha by the number of comparisons. So if you want 95 percent confidence across three tests, each test needs 98.3 percent. That's a high bar. Luna: And most platforms don't do that adjustment either. Lucas: They don't. So you have to be the one who knows. Luna: Okay, so we've covered pre-test planning and the multiple comparison problem. What about after the test? Any pitfalls there? Lucas: One big one: peeking. Stopping the test early because the results look good. That inflates false positives dramatically. There's a classic paper by Simmons, Nelson, and Simonsohn — they showed that if you peek after every 50 observations, your false positive rate can hit 30 percent. Luna: So the discipline is: decide your sample size upfront, then don't look until you hit it. Lucas: Exactly. Or use a sequential testing method that accounts for peeking. Some platforms like Optimizely offer that, but most email automation tools don't. Luna: It feels like the industry has a long way to go in making statistical rigor accessible. Most marketers I know are overwhelmed just keeping up with campaign calendars. Lucas: That's fair. And honestly, if today's conversation helped you think differently about your next test — maybe saved you from rolling out a false winner — that's exactly what we're here for. Luna: Yeah, if it sparked something you've actually used, that's great. Lucas: And if today was worth a coffee to you, there's a link — buy me a coffee dot com slash fexingo. No pressure. Just a small way to keep these conversations ad-free. Luna: Totally. And we really mean that. The tiniest gesture goes a long way. Lucas: Alright — back to the numbers. One more practical tip: if you're using HubSpot, you can export the test data and run a chi-squared test yourself. It's a simple formula. That gives you a p-value independent of the platform's calculation. Luna: Is that easy for a non-statistician? Lucas: There are online calculators. You just plug in the counts — how many sent, how many opened for each variant. It takes 30 seconds. I do it for every A/B test I run. Luna: I'm going to start doing that. I usually trust the platform's number, but now I know better. Lucas: And the other thing: always report the confidence interval, not just the point estimate. If the interval is wide — say 1 percent to 6 percent — you know you have low precision. Even if the confidence level is high. Luna: That's a good heuristic. Wide interval = weak evidence. Lucas: Exactly. So to wrap up: plan your sample size before you test, account for multiple comparisons, don't peek, and double-check with a simple chi-squared test. Do that, and you'll avoid the trap that tripped up that SaaS company. Luna: And if you do all that, you can actually trust your results. Which, in marketing, is rare and valuable. Lucas: Rare and valuable — I'll take that as a tagline. Next episode, we're going to talk about a related issue: how personalization tokens can actually hurt your click-through rates. There's some surprising data there. Luna: Looking forward to it. Until then, happy testing.