Latest / The Email Marketing Podcast with Fexingo: Newsletters, Open Rates, and Subscriber Growth / How Email List A-B Testing Cut Churn by 30 Percent for One Retailer
Transcript
- Lucas: So there's this assumption in email marketing that if you want to keep people subscribed, you should send them exactly what they signed up for — no surprises, no experiments, just consistent value. And that's not wrong, exactly. But it might be incomplete. Luna: Because consistency without curiosity can actually bore people into unsubscribing? Lucas: Exactly. And that's where A-B testing comes in — not just for subject lines or send times, but for the fundamental content strategy of the email itself. I want to talk about a case that really drove this home for me. Luna: I'm all ears. Who are we looking at? Lucas: An outdoor gear retailer — let's call them Trailblazer Supply — around 150,000 active subscribers, average open rate around 22 percent, churn rate of about 1.8 percent per month. That's a little above the e-commerce average, so they were already thinking about retention. Luna: And they decided to test their weekly newsletter format? Lucas: Right. They had two camps internally. The product team wanted the newsletter to be heavily product-focused — new arrivals, best sellers, gear comparisons. The content team wanted storytelling — customer trip reports, user-generated photos, longer-form tips. So instead of arguing, they A-B tested both approaches for three months. Luna: Three months is a long test. Most tests I see run for a week, maybe two. Lucas: And that's exactly why most tests are misleading. Churn is a lagging indicator. Someone doesn't unsubscribe because of one email — they unsubscribe because of a pattern. You need enough exposure for the pattern to sink in. Trailblazer ran it for 12 weeks, randomly splitting their list into two groups of about 60,000 each. Luna: What did the two versions actually look like? Walk me through it. Lucas: Version A — let's call it the 'product-first' version — opened with a hero image of a new tent or backpack, then three product blocks with prices and 'shop now' buttons, and a small editorial blurb at the bottom. Version B — the 'story-first' version — opened with a customer's photo of a mountain summit, a two-paragraph trip story with a gear mention woven in, then a single product recommendation, then more community photos. Luna: So version B was basically saying 'we're a community of people who happen to sell gear' instead of 'we're a store that sells gear'. Lucas: Exactly that. And the results were pretty stark. The product-first version had slightly higher click-through rates on the product links — about 3.1 percent versus 2.7 percent. But the story-first version had a 30 percent lower unsubscribe rate over the full 12 weeks. Luna: So the product version was better at driving immediate clicks, but it was slowly bleeding subscribers. And the story version kept people around longer, even if they clicked less per email. Lucas: Which is a classic long-term versus short-term trade-off. If you only look at click-through rate, you'd pick version A. But if you model out lifetime value, version B wins. Trailblazer calculated that the lower churn from version B translated into roughly 5,400 retained subscribers per year that they would have lost — at an average order value of around $85, that's significant. Luna: And that's the kind of math that makes a CFO pay attention. But I'm curious — was there any difference in open rates? Lucas: Surprisingly, no. Both versions averaged around 22 to 23 percent open rate. The subject lines were the same — they only changed the body content. So the open rate wasn't the lever. It was entirely about what happened after people opened. Luna: That's a great reminder that open rates don't tell you everything. You can have great open rates and still be driving people away with the content inside. Lucas: Right. And this is where I think most email marketers underinvest. They test subject lines endlessly, they test send times, they test from names. But they rarely test the core content thesis — the ratio of selling to storytelling. Luna: Because it's harder. You can't just swap a line of copy. You have to design two completely different email templates, write different copy, curate different assets. It's a bigger lift. Lucas: It is. But the payoff is also bigger. A subject line test might lift open rates by 2 or 3 percent. A content thesis test can change your churn trajectory. And Trailblazer didn't stop there. After the 12-week test, they ran a follow-up test to optimize the story-first format. Luna: What did they test in round two? Lucas: They tested how many product recommendations to include in the story-first format. The original version had one product link. They tested versions with two and three product links. And they found that two product links actually had the lowest churn — better than one or three. Luna: So there's a Goldilocks zone. One felt too sparse, like they weren't really shopping. Three felt too pushy, overriding the community vibe. Lucas: Exactly. And that's the beauty of sequential testing — you don't just find one winner, you refine it. Trailblazer ended up with a weekly newsletter that opened with a customer story, had exactly two product recommendations in the middle, and closed with a community photo gallery. Their monthly churn dropped from 1.8 percent to 1.2 percent over six months. Luna: Which is a 33 percent reduction in churn. That's massive for a list of 150,000. Lucas: Huge. And it cost them nothing but design and copy time. No new tools, no paid acquisition. Just smarter testing. Luna: So what's the framework for a listener who wants to try this? How should they set up a content thesis test? Lucas: First, pick one dimension to test — not three. Trailblazer tested content focus only. They kept subject lines, send day, send time, and frequency constant. If you change too many things at once, you won't know what drove the result. Luna: And how big should the sample be? Lucas: You want at least a few thousand subscribers per variant to get statistically significant results on churn, because unsubscribes are rare events. Trailblazer used 60,000 per group, which is more than enough, but even 10,000 per group can work if you run it long enough. I'd say minimum 12 weeks — but at least 8 if you're impatient. Luna: And what metrics should you look at besides unsubscribes? Lucas: Look at click-through rate, but also at 'email read time' if your ESP offers it, and at the rate of spam complaints. Trailblazer saw no difference in spam complaints, which is good. Also look at downstream metrics — website traffic from email, conversion rate, average order value. But churn is the north star for this type of test. Luna: One thing I really like about this case is that they didn't just test 'storytelling versus selling' as a binary. They refined the winning concept. That's where the compounding improvement comes from. Lucas: Yeah, and that's the part most people skip. They run one test, declare a winner, and move on. But the real gains come from iterative testing — taking the winning concept and optimizing it further. Luna: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making these deep dives possible. No ads, no sponsors, just people who find value in the show and want to keep it going. Lucas: And we really appreciate that. It lets us spend time on cases like Trailblazer instead of scrambling for ad revenue. So if these episodes have helped you rethink your email strategy, that's the place to support. Luna: Exactly. Okay, back to testing. One thing I want to emphasize — when you run a test like this, you have to commit to not peeking at results early. The temptation to check after two weeks and declare a winner is huge, but you need the full time horizon. Lucas: Absolutely. Early peeking is the enemy of reliable results. Trailblazer committed to 12 weeks and stuck to it. They had weekly check-ins but didn't make any changes until the full period was over. Luna: And what about the control group? Should you always keep a control? Lucas: Ideally yes. Trailblazer actually kept a small holdout group — about 10 percent of the list — that continued receiving the original product-first format throughout both test phases. That gave them a baseline to measure against. By the end, the optimized story-first format had 30 percent lower churn than the control. Luna: That's a powerful number. And it's the kind of result that can justify a permanent shift in content strategy. Lucas: Right. And that's what Trailblazer did. They rolled out the story-first format to their entire list. Their churn stayed lower, and interestingly, their click-through rate on product links actually improved over the next six months — from 2.7 percent to 3.4 percent — as they got better at weaving recommendations into stories. Luna: So the short-term click trade-off disappeared as they learned. Lucas: Exactly. And that's the final lesson — testing isn't just about picking a winner. It's about building a learning loop. You test, you refine, you learn, and your emails get better over time. Luna: And you don't need a massive budget or a fancy tool. Just a hypothesis, a split test, and the discipline to let it run. Lucas: Couldn't agree more. And if you try a content thesis test, I'd love to hear what you find. Tweet at us or drop a note — we might feature your results in a future episode. Luna: For now, that's what we've got on Trailblazer Supply and the surprising power of testing your email's soul, not just its subject line. Lucas: Next time, we'll look at how one nonprofit used behavioral triggers to reengage donors who had gone silent for over a year. Spoiler: it involved a very specific email sent exactly 11 days after their last donation.