Latest / The Email Marketing Podcast with Fexingo: Newsletters, Open Rates, and Subscriber Growth / Why Email Analytics Can Mislead You and What to Measure Instead
Transcript
- Lucas: So there's this phrase that's been rattling around in my head for a while now: 'What gets measured gets managed.' It's practically a mantra in email marketing. But what if you're measuring the wrong things? Luna: That's a pretty provocative starting point. Because the industry loves its dashboards—open rates, click rates, bounce rates. You're saying those aren't the right things? Lucas: Not exactly. I'm saying they can be actively misleading if you're not careful. And I've got a concrete case that illustrates why. There's a mid-size B2B SaaS company I've been following—let's call them ProjectFlow, not their real name. They run a weekly newsletter and a product update email. Last year, they noticed something weird. Luna: Okay, I'm curious. What happened? Lucas: Their open rates jumped from around 22 percent to 35 percent over three months. The marketing team was thrilled. They put out a case study internally about how their new subject line strategy was working. But here's the catch—their actual conversion rate from email, meaning sign-ups for demos and trials, dropped 12 percent over the same period. Luna: Wait, so they were getting more opens but fewer people actually taking the next step? That sounds like a classic vanity metric trap. Lucas: Exactly. And it gets more interesting. The team initially thought maybe their call to action wasn't strong enough, or their landing page had issues. They spent weeks A/B testing buttons and copy. But the real culprit was that their open rate was being inflated by Apple's Mail Privacy Protection, which launched in late 2021. By 2024, the vast majority of Apple Mail users had it enabled, and by now, in mid-2026, it's essentially standard. Luna: Right—MPP preloads images in the background, which forces an 'open' even if the subscriber never actually looked at the email. So their 35 percent open rate might have been more like 20 percent real human opens. Lucas: And maybe even lower if bots or security scanners were hitting their emails. So their subject line 'improvements' weren't actually improving anything. They were just optimising for a metric that had become unreliable. Meanwhile, the actual engagement—clicks, replies, conversions—was declining. But nobody noticed because they were so focused on the open rate going up. Luna: And I imagine the same could happen with click rates if you're not careful. If you have one link at the bottom that says 'Click here to unsubscribe', that counts as a click, but it's not a positive signal. Lucas: Right. So the lesson here isn't that open rate is useless—it's that no single metric tells the full story. The ProjectFlow team eventually shifted their focus to what they called 'subscriber-level lifetime value.' They started tracking whether a person who opened three emails in a row was more likely to convert than someone who opened sporadically. And they found that open frequency was actually a weak predictor. The stronger predictor was whether someone replied to an email—even just a 'thanks' or a question. Luna: So replies become a proxy for genuine interest. That makes sense. But most email platforms don't surface that as a primary metric. You have to dig into the inbox to see it. Lucas: And if these marketing conversations have sparked something you've actually used, it's worth mentioning that we keep this show free from ads. We think it keeps the advice cleaner. If that kind of independence matters to you, you can support it at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small way to keep the conversation going without any sponsor pressure. We appreciate it. Lucas: Back to the analytics issue. So they started measuring 'engaged subscriber rate'—the percentage of subscribers who had opened or clicked within the last 30 days, adjusted for MPP by excluding Apple Mail opens that seemed automated. That gave them a much clearer picture. They also started tracking 'list health score,' which combined engagement, bounce rate, and spam complaint rate into a single number. Luna: That sounds more actionable. What happened when they started using that new score? Lucas: They noticed that a segment of subscribers who had been on the list for over a year but never engaged—they called them 'zombies'—was dragging down the health score. So they did a re-engagement campaign, and for those who didn't respond, they removed them. Their list shrank by about 8 percent, but their conversion rate from email went up by 15 percent over the next quarter. Luna: So a smaller, more engaged list outperformed a larger, less engaged one. That's a pretty common finding, but the key is having the right metrics to spot it early. Lucas: Exactly. And this is where I think many marketers get tripped up. They're so focused on month-over-month growth in list size or open rate that they miss the underlying quality degradation. I've seen companies celebrate hitting 100,000 subscribers, only to find that 40 percent of them haven't opened an email in six months. That list isn't worth much. Luna: What about click to open rate? That's often touted as a better metric because it measures the proportion of people who opened who also clicked. Doesn't that filter out MPP inflation? Lucas: It helps, but it's not a silver bullet. If your open rate is artificially inflated by MPP, your click to open rate will be artificially depressed because the denominator is bigger than it should be. So you might think your content isn't resonating when actually it's fine. The better approach is to either use a proxy for real opens—like tracking the number of times a unique subscriber has loaded images after the first open, which is harder to fake—or just focus on click rate and conversion rate directly. Luna: So you're saying ignore open rate almost entirely? That feels radical for an email marketing podcast. Lucas: Not ignore, but deprioritise. Use it as a directional signal, not a key performance indicator. For example, if your open rate drops suddenly, it might signal a deliverability issue. But day-to-day, I'd rather watch click rate, conversion rate, and reply rate. Those are closer to business outcomes. And for B2B especially, reply rate is gold—it means someone actually read your email and cared enough to type a response. Luna: I've heard some marketers say that the best metric is 'revenue per email sent.' That's pretty bottom-line. Lucas: That's a great one, especially for e-commerce or SaaS where you can attribute revenue to specific campaigns. But it requires good tracking. You need to tie email clicks to a purchase or signup event, and then divide total revenue by number of emails sent. That gives you a dollar figure per email. If you're sending to a million people and making $10,000, that's one cent per email. If you can double that to two cents, you've effectively doubled your ROI without sending more emails. Luna: And that forces you to think about list quality, because sending to unengaged subscribers lowers the average. It's a natural incentive to clean your list. Lucas: Right. And it also encourages you to segment. Because if you know that your top-tier segment generates five cents per email and your bottom tier generates half a cent, you might decide to send less frequently to the bottom tier or try a different strategy. The metric drives the behaviour. Luna: So what would you say is the single most underused metric in email marketing today? Lucas: I'd say 'subscriber lifetime value' by cohort. Most marketers look at campaign-level metrics: this week's open rate, last month's click rate. But they don't track how a subscriber's value changes over time. If you sign up in January, what is your expected revenue by December? And how does that compare to someone who signed up in July? Cohorts let you see if your welcome sequence is improving or if your newer subscribers are less engaged than older ones. Luna: That's a longer-term view, and it requires more data infrastructure. But it's probably the most honest picture of whether your email program is actually working. Lucas: Exactly. And it's not that hard to set up. Most email service providers allow you to export data by signup date. Then you can map that against revenue data in your CRM. Even a simple spreadsheet analysis can reveal surprising insights. For ProjectFlow, they found that subscribers who joined through a specific webinar had a 30 percent higher lifetime value than those who joined through a blog opt-in. That changed how they allocated their marketing budget. Luna: So the takeaway is: don't just measure what's easy to measure. Measure what matters to your business, even if it takes more work. Lucas: And be suspicious of any metric that goes up while your business results go down. That's the sign of a vanity metric. It might feel good, but it's leading you astray.