Latest / The Newsletter Podcast with Fexingo: Substack, Beehiiv, and Email Newsletter Business / How Newsletters Use A-B Testing to Double Click-Through Rates
Transcript
- Lucas: I want to talk about something that sounds boring on paper but can literally double your click-through rate if you do it right. A-B testing for newsletters. Luna: Yeah, it's one of those things everyone says they should do but most people half-ass it. They test one subject line on a Tuesday and call it science. Lucas: Right. And the thing is, the platforms make it so easy now. Substack has built-in A-B testing for subject lines. Beehiiv has it for subject lines, preview text, even send times. But the tool is only as good as the person setting up the test. Luna: You know, I was reading a case study from a writer on Substack — she runs a weekly culture newsletter, about 8,000 subscribers. She tested two subject line styles for a month: one was a straightforward headline, like 'The Problem With Streaming Bundles,' and the other was a provocative question, like 'Are Streaming Bundles a Trap?' Lucas: And the question format won by a mile — 12 percent higher open rate, and more importantly, click-through rate nearly doubled. But here's the kicker: she ran the test for 30 days, not just one send. That's the discipline most people skip. Luna: Exactly. You need enough volume to get statistical significance. If you only have a small list, testing one subject line against another on a single send can give you a false winner. Lucas: And this is where I see a lot of newsletter operators make a mistake. They look at the A-B test results after one send, see that version A got a 5 percent higher open rate, and they declare victory. But if your list is only 2,000 people, and you split it into two groups of a thousand, that 5 percent difference might not be statistically significant. You could be making decisions on noise. Luna: Right. So what's the minimum sample size you'd recommend for a subject line test? Lucas: I'd say at least 5,000 people per variant, but ideally more. If you have a list of 10,000, you can split it 50-50 and get decent confidence after a few sends. But if you're just starting out and have 500 subscribers, honestly, don't bother with A-B testing yet. Focus on writing better content and growing the list. The statistical power just isn't there. Luna: That's a good point. I think a lot of small creators feel like they have to do everything at once — A-B testing, segmentation, automation — but the fundamentals come first. Lucas: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes ad-free. If these newsletter marketing conversations have sparked something you've actually used, that's the way to keep it going. Luna: Yeah, it's a small group that makes it possible for us to dive deep into topics like this without worrying about sponsors. Lucas: So back to the testing itself. Once you have enough subscribers, the next question is: what should you test? Subject lines are the obvious starting point, but I think the real gains come from testing content format. Luna: Can you give an example of a content format test that worked? Lucas: Sure. There's a finance newsletter — I think it's called 'The Daily Upside' — that tested sending a long-form essay versus a bullet-point summary of the same stories. They found that the bullet-point version had a 30 percent higher click-through rate because readers could scan quickly and click on what interested them. So they switched to that format full-time. Luna: And that's not just about subject lines. That's about respecting your reader's time. A lot of newsletter writers think they need to write long, thoughtful pieces, but sometimes the audience just wants the key takeaways. Lucas: Exactly. And you can test that with a simple A-B split on Beehiiv. Send half your list the long version, half the short version, and measure not just clicks but also unsubscribe rate and reply rate. Those secondary metrics matter. Luna: What about send time? I know some tools let you test that too. Lucas: Beehiiv has a feature called 'Send Time Optimization' that automatically figures out the best time for each subscriber based on their past behavior. But if you want to run your own test, you can split your list into two groups: one gets the newsletter at 8 a.m., the other at noon. Run it for two weeks, and see which group has higher open rates. But again, you need a big enough list. Luna: I've also seen people test preview text — that little snippet that shows up next to the subject line in most email clients. It can make a big difference. Lucas: Huge difference. I've seen tests where changing the preview text from a generic 'View this email in your browser' to a compelling teaser like 'Why Netflix is losing the streaming war' improved open rates by 8 percent. And it's zero extra effort once you set it up. Luna: So what's the biggest mistake you see people make with A-B testing? Lucas: Stopping the test too early. The classic error is: you send to 10 percent of your list, see that version A is winning, and then send version A to the remaining 90 percent. That's called 'peeking' and it invalidates the test. You should decide your sample size and your duration beforehand, and stick to it. Luna: That's a good rule. I also think people test too many variables at once. If you change the subject line, the preview text, and the content format all in one test, you won't know what caused the change. Lucas: Exactly. Test one thing at a time. That's the scientific method. And document your results. I know a newsletter operator who keeps a simple spreadsheet with the date, the variant, the open rate, click rate, and whether the result was statistically significant. After six months, she has a playbook of what works for her audience. Luna: That's smart. And you can even share those insights with your audience. I've seen newsletters that say 'I tested two subject lines this week, and here's what I learned.' It builds trust and shows you're data-driven. Lucas: Yeah, transparency is a great content strategy in itself. So to wrap up: if you have a list of at least 5,000 subscribers, start with subject line tests. Run them for at least a week, don't peek, and track everything. Once you have that down, move to content format tests. And if you're smaller, focus on writing great content and growing your list. Luna: And if you want to support the show and keep these deep dives coming, you know where to find us. We'll be back next week with another angle. Lucas: Happy testing.