Latest / The Newsletter Podcast with Fexingo: Substack, Beehiiv, and Email Newsletter Business / How Newsletter Writers Use A-B Testing Subject Lines to Boost Open Rates
Transcript
- Lucas: If these marketing conversations have sparked something you've actually used — a better subject line, a smarter segmentation trick — there's a reason for that. These shows are built on the principle that good tactics beat hype every time. Luna: And the reason we can spend ten minutes on something as specific as, say, A/B testing subject lines is because we're not trying to sell you anything in between. Lucas: Exactly. So let's jump right into that. Today I want to talk about the single most impactful lever a newsletter writer can pull without changing a single word of the content inside: the subject line. Luna: And not just any subject line, but the practice of systematically testing them — A/B testing, where you send two versions to a subset of your list and let the winner go to the rest. Lucas: Right. It sounds obvious, but a shocking number of newsletter writers — especially on Substack and Beehiiv — still just write one subject line per issue and hit send. They're leaving opens on the table. Luna: What kind of lift are we talking about? Like, is it worth the extra five minutes? Lucas: It absolutely is. I looked at a case study from a marketing newsletter called 'The Growth Report' — about 200,000 subscribers on Beehiiv. They tested two subject lines for the same issue. Version A was '5 Growth Hacks for Q3'. Version B was '5 Growth Hacks That Actually Worked in Q2'. Luna: That's a pretty subtle change — just adding 'that actually worked' and shifting the timeframe. Lucas: Exactly. Version B beat Version A by 22 percent in open rate. That's not a small bump. On a 200k list, that's tens of thousands of additional opens. And the only difference was a few words that signaled social proof and recency. Luna: So the test itself is straightforward — but how do you set it up properly? Because I think a lot of writers just split their list 50/50 and call it a day. Lucas: That's the bare minimum, but there are a few pitfalls. First: sample size. If you only send the test to 100 people, the result might not be statistically significant. Most platforms — including Beehiiv — will tell you the confidence level. You want at least 95 percent confidence before you declare a winner. Luna: So you need a big enough test group. Beehiiv's A/B testing tool lets you choose the percentage — I think the default is 20 percent of the list. Lucas: Right. And Substack doesn't have a built-in A/B test feature at all. So writers on Substack have to do manual splits — like sending to a segment, waiting, then sending the winner to the rest. It's clunky, but doable. Luna: I've seen some writers use two separate campaigns and then compare opens after a few hours — but that introduces time bias because the second batch goes out later. Lucas: That's a real issue. The ideal is simultaneous send, which Beehiiv handles natively. Substack is working on it, I've heard, but as of July 2026 it's not there yet. Luna: What about the psychology behind the subject lines themselves? Because the 'that actually worked' phrase in your example is clearly tapping into something. Lucas: It's the curiosity gap, combined with social proof. 'Actually worked' implies that others have tried it and it's validated. That's more compelling than just listing generic hacks. Another classic test is specificity versus benefit. For example, 'How I Grew My List to 10k in 30 Days' versus 'A Simple Growth Strategy That Works'. Luna: I've seen data that specificity almost always wins. Numbers, timeframes, concrete outcomes. Lucas: Yeah, there's a well-known test from the email marketing world — not a newsletter, but an e-commerce brand — where they tested '50% Off Everything' versus '50% Off Everything — Today Only'. The second one outperformed by 35 percent because of urgency. Urgency works, but you have to use it honestly. Luna: So for a newsletter that's not selling anything, urgency might not apply as often. But curiosity — like 'What I Learned From...' or 'The One Thing That...' — that's evergreen. Lucas: Exactly. The key is to test one variable at a time. If you change both the phrasing and the length and the emoji usage, you won't know what caused the lift. Test one element — like adding a number, or using a question versus a statement, or including an emoji or not. Luna: Which brings up emojis. Love 'em or hate 'em, they can boost open rates in certain niches. A newsletter about pop culture might see a lift from a 📰 or 🎬, but a B2B finance newsletter might not. Lucas: Right. And you only know by testing. I saw a test from a tech newsletter — they tested a subject line with a rocket emoji versus the same line without. The emoji version won by 8 percent. But that's not universal. Luna: So the process is: pick one variable, split your list, let it run for a few hours or until you hit significance, then send the winner. How long do you wait before declaring a winner? Lucas: Most opens happen within the first hour. So I'd say wait at least two hours, but no more than six, because after that the late opens can muddy the results. Beehiiv's tool actually auto-selects the winner after a set time and sends it to the remainder automatically. That's the ideal. Luna: Let's talk about a mistake I see a lot: testing too many variations at once. Some writers will test four or five subject lines, but with a small list, that splits the sample too thin. Lucas: Yeah, stick to two. A/B, not A/B/C/D. Unless you have a massive list — like over 100,000 — two is plenty. And always have a control — your usual style — so you can see if the new approach actually beats your baseline. Luna: One thing I find interesting is that some writers don't test at all because they think their gut feeling is enough. But the data often surprises them. Lucas: 100 percent. I remember a writer who was convinced that short subject lines — under 30 characters — would win. She tested a short one against a longer, more descriptive one, and the longer one won by 15 percent. She was shocked. That's why you test. Luna: So the takeaway is: test consistently, one variable at a time, and use the platform tools when available. Lucas: Right. And keep a log of what you test and the results. Over time, you'll build a personal playbook of what resonates with your specific audience. That's worth more than any generic best-practice list. Luna: That kind of knowledge — the kind you generate yourself through testing — is exactly the sort of thing that makes these shows valuable. And it's why we keep the episodes free of sponsor interruptions. If you want to support that approach, the link is buy me a coffee dot com slash fexingo. No pressure, just if it's useful to you. Lucas: Yeah, that's right. Every coffee helps us keep the research deep and the ads out. Now, back to subject lines — one more tactic I want to mention is personalization. Including the subscriber's first name in the subject line can boost opens, but it's become so common that it might actually hurt if overused. Luna: I've seen studies where personalized subject lines outperform generic ones by about 10 to 15 percent, but only if the list is well-segmented. If you're a B2B newsletter and you use first name personalization, it can feel like a mass email. Lucas: Exactly. So test that too. Some audiences love it, some hate it. The only way to know is to run the test. Luna: Alright, so to wrap up: start with one test per week. Pick a variable — specificity, curiosity, emoji, length — and let the data guide you. Lucas: And remember: the subject line is the first impression. It deserves more than a afterthought. Give it five minutes, test it, and you'll see the payoff in your open rate within a month.