Latest / The Email Marketing Podcast with Fexingo: Newsletters, Open Rates, and Subscriber Growth / How Email List A B Testing Boosts Performance
Transcript
- Lucas: So we talk a lot on this show about building a list and keeping subscribers happy, but I realized we've never done an episode on the one thing that separates good email marketing from great email marketing: actually testing your assumptions. Luna: You mean A/B testing. And I bet most people listening have heard the term, but maybe they're not doing it right. Lucas: Exactly. And the data backs that up — I was looking at a Litmus study from late last year, and it found that only about a third of email marketers run tests on a regular basis. And of that third, a lot of them are testing things that barely move the needle, like button color or font size. Luna: Button color tests — those are the classic 'we need to test something, let's test the red button vs the blue button' experiments. And yeah, sometimes they matter, but usually the lift is tiny. Lucas: Right. The big wins come from testing things that actually change the decision to open or click. Subject lines, preview text, send time, personalization tokens. The variables that hit the core of the email's value proposition. Luna: So let's start with subject lines. What's a good way to approach that? Lucas: I think the biggest mistake is testing two completely different subject lines. Like, 'Your weekly roundup is here' versus 'Don't miss these 5 deals'. You get a winner, but you don't know why it won. Was it the length? The personalization? The urgency? Luna: So you want to isolate one variable at a time. Change one word, or add one emoji, or flip the order of a sentence. Lucas: Precisely. I love the example from a mid-size outdoor gear company — they tested a subject line that said 'Last chance for 20% off' versus 'Your 20% off expires tonight'. Same offer, same length, just a shift in framing from 'last chance' to 'expires tonight'. The second one lifted open rates by twelve percent. Luna: That's a real number. And that's the kind of insight you can only get from a controlled test. But there's a pitfall here — sample size, right? Lucas: Huge pitfall. People test on tiny segments and call a result statistically significant when it's really just noise. You need enough subscribers in each variant to be confident the difference isn't random. A good rule of thumb: you want at least a few hundred opens per variant for open rate tests, and even more for click tests. Luna: And you have to let the test run long enough to capture different days of the week, different times of day. If you only send on Tuesday mornings, you're testing Tuesday morning behavior, not general preference. Lucas: Right. And that's where confirmation bias creeps in. Marketers often stop a test as soon as one variant looks like it's winning, even before the stats are solid. They want the test to be over so they can declare a winner and move on. Luna: Yeah, I've definitely been guilty of that. You see a lead after an hour and you think 'great, the A version worked', but by the end of the day the B version catches up. You have to be disciplined. Lucas: Which brings me to a framework I like: the 'test, learn, deploy' cycle. You test something small, measure the result with a clear threshold — say, ninety percent confidence — then you implement the winner for your full list, and immediately start planning the next test. Luna: So it's a continuous improvement loop. And it doesn't have to be complicated. Even a single test per month can compound over a year. Lucas: Absolutely. And if you're thinking, 'I'd love to test more but I don't have the tools,' most email service providers offer A/B testing built in. Mailchimp, Klaviyo, ConvertKit — they all have it. There's really no excuse not to start. Luna: And you know, we talk a lot on this show about practical tactics you can use. If these conversations have sparked something you've actually implemented in your own email marketing — even just one idea — that's exactly why we do this ad-free. And listener support, through things like buy me a coffee dot com slash fexingo, is what keeps it going. Lucas: Yeah, it's a small way to help us keep producing episodes that go deep on topics like this. No pressure, but if the show's been valuable, that's where you can find us. Luna: Okay, so back to testing. Let's talk about a common scenario: you're testing send time. What's the right way to set that up? Lucas: Good question. Testing send time is trickier because you're not testing a single element in the email — you're testing the timing of the entire send. The best approach is to split your list into two random halves and send the same email at two different times. But you have to account for the fact that the second group gets the email later, so you're comparing apples to apples only if you measure opens and clicks within a fixed window after each send. Luna: So like, measure open rate within 24 hours of send for both groups, not just raw open rate, because the group that got it later might not have had as much time to open. Lucas: Exactly. And some platforms handle this automatically, but it's good to understand what the metric actually means. Another thing I see people mess up is testing multiple variables at once — like a new subject line and a new send time in the same test. Then you don't know which change caused the lift. Luna: Right, that's a classic confound. So you test one variable at a time, and you keep everything else identical. Lucas: Yes. And then there's the question of which tests to prioritize. I like to use a simple impact-effort matrix. High impact, low effort tests go first. Changing a few words in your subject line is low effort. Redesigning your entire email template is high effort. So start with the low-hanging fruit. Luna: And you'll build momentum. Once you see a win, it's easier to get buy-in for bigger tests. Lucas: Exactly. So here's my challenge to listeners: pick one variable — just one — and design a simple A/B test for your next campaign. Run it for at least 48 hours, or until you have at least 500 opens per variant. Then look at the result with a critical eye. Did it actually move the needle? And then, learn from it. Luna: And if you're not sure where to start, subject lines are the classic entry point. But even something as simple as testing a personalization token — like using the subscriber's first name versus not — can reveal a lot about your audience. Lucas: Great point. Actually, there was a study from Experian that found personalized subject lines lifted open rates by about 26 percent on average. But that's an average — your list might respond differently. That's why you test. Luna: Yeah, averages can be misleading. Your audience might be the one where personalization doesn't work because it feels spammy. You need your own data. Lucas: Exactly. And that's the core message of this episode: don't guess, test. And when you do test, do it correctly. One variable, adequate sample size, clear success metric, and enough time. Luna: I think that's a solid takeaway. And maybe we'll follow up in a future episode with some advanced testing techniques — like multivariate tests or sequential testing. Lucas: I'd love that. But for now, if you run one test this week, you're already ahead of the majority of email marketers. Luna: And if you get a surprising result, share it with us. We love hearing about real-world experiments. Lucas: Absolutely. That's a wrap for today's episode. We'll see you next time.