Latest / The Email Marketing Podcast with Fexingo: Newsletters, Open Rates, and Subscriber Growth / How Email List Sentiment Analysis Can Predict Churn
Transcript
- Lucas: So I was looking at churn data from a daily newsletter that sends every morning at 6 AM. They had a 60-day churn rate of about 8 percent, which is actually pretty normal for that space. Luna: Right, and most people just accept that as inevitable. Lucas: Exactly. But then they started doing something that very few email marketers do — they analyzed the sentiment of every single reply email that came in from subscribers. Not just the ones that said 'unsubscribe,' but all replies. Luna: What did they find? Lucas: They found that subscribers who sent even one reply with negative sentiment in the previous 14 days were six times more likely to unsubscribe in the following 30 days. And the most predictive single sentence was not 'stop emailing me' — it was 'this isn't relevant.' Luna: Wow. So they could actually see churn coming weeks before it happened. Lucas: Exactly. And once they saw it, they had a window to intervene. They started sending a one-time re-engagement email to anyone flagged by the sentiment model — just a short message asking, 'What topics would you like more of?' And they cut that 60-day churn from 8 percent to 4.2 percent. Luna: So basically, they halved their churn by listening to the replies they were already getting. Lucas: Right. The replies were already there. Most people just never read them systematically. So today, I want to talk about how you can set up a simple sentiment pipeline for your own list, even without a data science team. Luna: Alright, let's do it. Where do you even start? Lucas: First, you need to make sure your email provider is actually storing replies. A lot of services like Mailchimp or ConvertKit will show you replies in the interface, but they don't give you easy API access to the text. So you need to either use a provider that exposes reply content, or set up a forwarding rule to a dedicated inbox. Luna: So you're basically collecting every reply into one place. Lucas: Exactly. Then you run sentiment analysis on each reply. You can use a simple tool like the VADER lexicon from the NLTK library — it's free, it's open-source, and it's tuned for social media and short text, so email replies work really well. Or if you want something more turnkey, there are SaaS tools like MonkeyLearn or Google Cloud Natural Language that will give you a sentiment score. Luna: And what score do you use as a threshold? Lucas: The newsletter I mentioned used a compound score below negative 0.5 on VADER's scale. But they also found that a single reply with very strong language was actually less predictive than a series of mildly negative replies. So they built a rolling window: if a subscriber had two or more replies in a 14-day period with a compound score below negative 0.3, that was their trigger. Luna: So it's not just one angry person blowing off steam. It's a pattern. Lucas: Right. And that pattern is what gives you the signal. The angry person might just be having a bad day. But the person who keeps saying 'this is not useful' or 'too salesy' over two weeks — that person is about to leave. The data backed that up. Luna: What about false positives? I imagine some people send negative replies but still stay subscribed. Lucas: Absolutely. And that's the most important caveat. In that newsletter's test, about 30 percent of the flagged subscribers did not unsubscribe in the next 60 days. So you have to be careful not to overcorrect. They didn't change the content for those people — they just sent the re-engagement email and waited. If the subscriber didn't reply or their sentiment improved, they were left alone. Luna: So the intervention itself was pretty low-touch. Lucas: Very low-touch. One extra email. And the reply rate on that email was about 12 percent, which is high for a re-engagement message. Most of the replies were positive — people saying 'more case studies' or 'shorter pieces' — and that gave the team actual content signals. Luna: So they not only reduced churn, they got direct feedback on what to write. Lucas: Exactly. And that's the real win. Sentiment analysis becomes a feedback loop for your content strategy. If a certain topic consistently generates replies with positive sentiment, you double down. If another topic generates negative sentiment, you rethink it. The churn prediction is almost a side effect. Luna: I want to ask about scale. If you have a list of 10,000 subscribers, you might get 50 replies a day. That's manageable. But what about a list of 500,000? Lucas: At that scale, you need automation. But the beauty is that sentiment analysis is cheap to run. VADER can process hundreds of replies per second on a single server. And the SaaS tools charge per query, usually a fraction of a cent. So even at 500,000 subscribers, the cost is negligible compared to the churn you're preventing. Luna: So the ROI is pretty clear. Lucas: I think so. Let me give you a rough number. If you have 100,000 subscribers and a 5 percent annual churn rate, you're losing 5,000 subscribers per year. If you can cut that churn by half using sentiment analysis, you retain 2,500 subscribers. At a conservative lifetime value of $50 per subscriber, that's $125,000 in retained revenue. And the cost of setting up the pipeline? Maybe a few hundred dollars and a weekend of work. Luna: That's a no-brainer. But I'm wondering — are there any situations where you should ignore sentiment data? Lucas: Yes. If your list is very small — under 1,000 subscribers — the signal to noise ratio is too low. A single angry reply from one person could feel like a trend, but it's probably just that one person. Also, if you send emails that are inherently transactional, like order confirmations or password resets, sentiment analysis on replies is mostly noise. People don't reply to those, and if they do, it's usually a support issue, not a content issue. Luna: So this is really for newsletters and content-driven email. Lucas: Exactly. If you're sending a daily or weekly newsletter with editorial content, this is for you. If you're sending purely transactional emails, it's not worth the effort. Luna: One thing I'm curious about — how do you handle replies that are negative but actually constructive? Like someone saying 'I love your content but the font is too small.' That's negative on the font, but positive on the content. Lucas: That's a great point. Standard sentiment analysis would flag that as negative, but the real signal is mixed. The newsletter I mentioned actually used a second layer: they classified replies into topic categories using a simple keyword list. So if a reply contained words like 'font,' 'design,' or 'layout,' they'd route it to the design team, not the churn model. They only triggered the re-engagement email if the negative sentiment was about content relevance or frequency. Luna: So you need to understand the context of the negativity. Lucas: Right. And that's where a human-in-the-loop can help. In the early days, the founder of that newsletter actually read every flagged reply himself. He said it gave him a much better intuition for what the numbers meant. After a few months, he automated it. Luna: I like that. Start manual, then automate once you know what to look for. Lucas: Exactly. And that's the approach I'd recommend for anyone starting out. Don't try to build the perfect pipeline on day one. Just start collecting replies, reading them, and noting which ones seem to precede unsubscribes. After you have 50 or 100 data points, you'll start to see patterns. Luna: Alright, I'm convinced. But let me play devil's advocate. Some people might say, 'If I'm already tracking open rates and click-through rates, isn't that enough to predict engagement?' Lucas: Great question. Open rates and click-through rates tell you about behavior, but they don't tell you about sentiment. You can have a subscriber who opens every email but is getting increasingly annoyed by the content. They might not click because the links aren't relevant, but they're still opening out of habit. Sentiment analysis catches that frustration before it turns into a behavior change. Luna: So it's a leading indicator, while opens and clicks are lagging. Lucas: Exactly. Opens and clicks are trailing signals. By the time your open rate drops, the subscriber has already mentally checked out. Sentiment analysis gives you the chance to intervene while they're still engaged enough to reply. Luna: That's a powerful distinction. And it makes me think — if more email marketers did this, the average list quality would probably be much higher. Lucas: I think so too. And it doesn't have to be complicated. The newsletter I mentioned started with a simple spreadsheet where they manually copy-pasted replies and assigned a green, yellow, or red sentiment. They did that for three months before they automated anything. Luna: So the barrier to entry is really just the willingness to read your replies. Lucas: That's it. And if these marketing conversations have sparked something you've actually used, whether it's a sentiment pipeline or something else, we'd love to hear about it. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes possible. Luna: Yeah, it's a small group that keeps it ad-free, and every bit helps keep the conversations going. Lucas: Exactly. Back to sentiment — the key takeaway is that your replies are a goldmine of predictive data. Most email marketers ignore them. The ones who read them can predict churn, improve content, and build a stronger relationship with their subscribers. Luna: And it starts with just one reply. So maybe go check your inbox after this. Lucas: I'd recommend it. Next time, I want to talk about something related — how to use reply data to segment your list by content preference, not just engagement level. Luna: I'm already thinking about how that would work. See you then.