Latest / Your Mic / The Metric You're Ignoring Is Killing Your Podcast

Transcript

The Metric You're Ignoring Is Killing Your Podcast · Your Mic

0:00
Your download numbers look great, but your retention, on the other hand, is a flaming dumpster fire.
0:06
And you have no idea why.
0:08
That's the problem my friend Nile Sullivan built pod critic to solve.
0:12
It's an AI-driven editorial tool that doesn't just tell you what is happening to your show, it tells you why.
0:19
And today we're getting into the analytics blind spot that's costing podcasters listeners they never knew they were losing.
0:27
Nile, welcome to show by the end of this episode what will listeners take away?
0:33
Yeah, no fakes for having me.
0:35
Freddie and Heather, hello everyone, out there.
0:38
I think the main thing is that there's more to reviewing podcasters and your standard analytics tools.
0:45
What we're going to talk about today is, you know, really dig into the editorial side and how you can bring the two together to really understand what's happening across all of your shows
0:55
and across all your episodes and it's editorial.
0:58
Yeah, let's go back into your standard analytics tools because for the longest time and it only seems...
1:05
And I like to know your thoughts here Nile, is that it seems like the tail end of 2025 and definitely this year that the consensus among podcas people like us is that downloads are trash, that there are nothing more than vanity metrics.
1:23
So why don't we talk about from your standpoint what that means to you.
1:27
Standard analytics, not really meaning as much as we as a collective industry or people who are looking at KPIs might give them credit for.
1:37
Yeah, and I understand why people look at them and spend so much time looking at them.
1:41
I mean, I've been podcasting for them for years and I could be understand, you know, the reasons look at them.
1:46
But what I've discovered particularly over the past some of those six months I've been talking to various different podcasting people like yourself ready and many other studios agencies and networks is that analytics only tell you part of the story.
1:59
So I only tell you what's happening with an episode, but they don't necessarily tell you why it's happening.
2:05
So why is there a drop off at certain point? Why did that episode do better than the other?
2:10
Why, you know, and all these questions seem unanswered and then so that's really what I'm trying to bridge with Paul critic is actually bringing the editorial side and so people can try and understand what is happening.
2:22
The various points there.
2:24
There's shows.
2:26
Let's say you're working with the show that has.
2:30
And we're going to just use downloads as a reference, but it's a it's a popular show.
2:34
So 100,000 downloads per episode per month.
2:38
So we're looking at over 400,000 downloads per month and they dig into pod critic.
2:45
They're hiring you guys and you're like, whoa.
2:49
They're retention to trash.
2:51
How do you break that news to them?
2:53
Or what are you what are you working on when when someone gets that kind of that kind of visceral feedback?
2:59
Wow, my audio baby actually is kind of ugly.
3:03
That's an interesting, interesting way to put it, definitely.
3:07
But I mean, the first thing I'd say first was really what we're looking at here, you know, with pod critic particularly it's when you're managing sort of multiple shows like lots and lots.
3:17
You could be a podcast network with hundreds of shows.
3:19
You could be a studio managing multiple clients.
3:22
So it's really that that point where you've got.
3:25
But I mean, you can bring it back to one show.
3:28
Because if you got one show, you've got lots of episodes all performing differently, all with different retention rates.
3:33
One maybe doing brilliant one episode does brilliantly the other one doesn't.
3:37
So what what we do or what you know what pod critic does is actually bring those some of those metrics.
3:44
It's basically bringing to three things together.
3:46
It's an editorial assessment of the episode using AI and then bringing in the context of the show.
3:53
So so it knows, you know, who who who is it for?
3:57
You know, is it edited in a certain way?
4:00
You know, for a certain reason, maybe the intro is going to be very short and punchy for a reason you can have that answer all the context around the show is very important.
4:08
And then finally, you bring in the performance context as well.
4:11
And once you have those three elements, it can then provide some reasoning of why editorially.
4:17
Like I said, but the example you go, why it might be drifting in a certain way.
4:22
And the value really comes from the more episodes you add uploads.
4:26
The more shows you add is the better and clearer picture you can get editorially across all of your sort of portfolio.
4:34
It doesn't seem because it is more valuable to somebody who has a sponsor show because you don't know what you don't know.
4:43
And if all you know are your standard standard dashboard analytics tools and you're not really paying attention to retention all that much and you go through and you're seeing what seeing this common drop off point.
4:55
And it just so happens to be before an ad plays.
4:59
Well, that I would say it behooves you as the host to do something about it.
5:05
Thanks. I mean, ads is also an interesting, interesting thing as well.
5:10
I mean, I could talk more personally because of course, as well as building a tool and, you know, working with podcasts and things like that.
5:18
I also listen to a lot of podcasts as well.
5:21
So I also have my own perspective on ads and what ads work from my perspective.
5:27
Of course, it will be different for different people.
5:31
But ones that generally cross over whatever the show's topic is, generally work better.
5:36
I've listened to some podcasts where it's very jarring.
5:38
It goes from, you know, talking about sport and then, you know, advertising something completely different.
5:43
I want to shoehorn it in the whenever it relates closely to the show and it's subjective.
5:47
That's that's really where, you know, for me personally is where that kind of thing works best.
5:52
It sounds like it's one of those dynamic ads.
5:55
So if it's a sports show and then you hear something for Home Depot or something that really doesn't have anything to do with football or hockey or MMA or whatever.
6:05
It's just whatever happens to be the ad that the network or the host puts in the podcast host, the hosting site.
6:16
Yeah, and I think I also, I mean, I know we've gone down a bit of a rabbit hole here about ads.
6:22
I like rabbit holes.
6:23
It's great for a podcast, isn't it?
6:25
So you can come back out again and go somewhere.
6:29
No, I mean, other examples I've heard is where I mean, I listen to a, I'll say football, but I'll say soccer because we're, you know, maybe a American audience, but soccer saw podcasts.
6:39
There's only one real football, but yeah, I don't exactly exactly.
6:42
But then soccer show I listen to in the UK and what they try and do, they say, you know, these sections powered by whatever the advertiser is.
6:51
And then they'll do like a short like quiz with the host because it's a very sort of conversational.
6:56
It's all about the latest, you know, soccer news and kind of thing.
6:59
And that'll ask some questions and they have obviously free former footballers.
7:02
So they ask stuff, who's your favorite player you've played with in your career and that kind of thing.
7:08
So that kind of thing is, so it's content, but it's also advertising as well.
7:12
It's been more subtle.
7:14
And again, this is really my opinion at this point because over people might flig nuts.
7:19
I don't know a bit.
7:21
I don't know.
7:22
Yeah, I mean, who loves ads?
7:23
I don't know.
7:24
Do people love ads out there?
7:25
I don't know.
7:26
But I mean,
7:27
Yeah.
7:28
The podcast,
7:30
podcast industry likes to tell the staff that people are more likely to buy host red from host red ads than a traditionally played commercial,
7:40
which is what we were talking about in one of our examples that we had just covered.
7:45
But you bring up something and that is your opinion, your opinion aisle.
7:50
So how much influence did you have personally as the brains of pod critic?
7:58
Because it is an AI driven software.
8:01
So it is, you know, how does that work?
8:04
And we're really getting into the rabbit hole of how you really devise this kind of program.
8:09
Is it, is it based on people that you talked to?
8:14
And it's like a, I guess like a collaborative effort or like a consensus, so to speak, or is it like, well,
8:22
it's, I saw this problem and I'm creating a, I'm creating a product to help hosts solve it.
8:29
And I'll recruit some buddies.
8:32
Yeah, that's a really good question.
8:34
I guess I'm going to take the story back.
8:36
So when we started speaking about it, the product was a completely different type of product, right?
8:42
And it's evolved.
8:43
I think it was about August last year.
8:45
I had the original thought about it.
8:46
And then I started talking to, like I said, various different people in the podcasting world,
8:50
whether B2B podcast or studios networks, et cetera.
8:53
And originally the idea was to do a, and you probably remember this,
8:57
like an in-depth analysis on one episode.
9:00
And it was very detailed, very, every aspect of it was covered from, you know,
9:06
speaker ballots to tone and everything.
9:09
And I thought it was great.
9:10
But when I spoke to people, it, to me, it seemed quite overwhelming.
9:14
You know, it's too much information for one episode.
9:18
And I was, you know, maybe you weren't actually saying that to me,
9:21
but in the terms of the people using it and that kind of thing, the people I've spoken to.
9:25
So what I did, I actually made quite a radical decision to flip it on its head,
9:30
rebuild it from scratch.
9:32
And actually build in a much simpler way.
9:34
So, and you've seen this as well.
9:36
It's built very simply that it analyzes a transcript,
9:39
either by connecting an RSS feed or uploading a file or pasting a transcript,
9:44
but analyzes a transcript, produces one editorial reading of the episode,
9:48
so what's working, what's not, and the evidence why it's suggesting that,
9:54
and things like that, which is incredibly valuable,
9:57
but then the value starts to compound once you upload more episodes
10:02
and analyze more episodes.
10:03
It starts to pick up patterns that are very hard to spot.
10:06
Even if you reviewed every second of every episode you've ever recorded,
10:09
you've found it very hard to spot it.
10:11
So then you begin to spot patterns across the episodes of the show,
10:16
then across, as you add more shows, you get,
10:19
then get to see it across your portfolio as well.
10:21
And that's where you really get to see very subtle, often very subtle things, right?
10:25
So even if you could listen to every episode and have the time to do that,
10:28
then it, you know, there'd be things that we would miss.
10:31
So it's trying to spot those things before the show starts to drift.
10:36
And then the other side of it, like I mentioned before,
10:39
is it's not just an AI analyzing it and that's it.
10:42
We're bringing the context of the show into the loop,
10:46
and all those things I mentioned before about creative principles
10:49
and target audience, and whether it's a solo or interview type show,
10:52
all these things are important.
10:54
So it knows the context of why the show exists and what it's there for.
10:58
And then the final layer is the performance side.
11:01
So bringing in some of the analytics, like downloads,
11:04
so it can correlate everything together, you know, how...
11:08
And one thing I guess I didn't mention is the AI itself is actually
11:12
that original version that used for Friday,
11:16
all of the different aspects it analyzed,
11:19
so speak at balance and tone, performance,
11:22
and guest feedback.
11:23
And all I can think it's still doing that analysis,
11:25
but it's doing it in the background,
11:27
and it's just surfacing a very simple insight.
11:29
So I hope that makes sense.
11:30
That was a very long answer.
11:31
Well, I agree with that.
11:33
So that makes a lot of sense.
11:35
And this is all really good stuff because, you know,
11:38
it's one of those things where, as creatives,
11:42
we sort of like to get, at the risk of generalizing us,
11:46
high on our own supply.
11:48
And if we hit a plateau, or if we start a new series,
11:53
a new season, or go with a new sort of angle,
11:57
like if you're a solo host and you're trying podcast interviews
12:01
for 10 or 12 episodes in a row, or vice versa,
12:05
you're doing interviews and now you're trying solos,
12:07
and you're having a lot of fun.
12:10
But you don't know why your podcast isn't growing.
12:15
This is, I think, to me, a high-leveraged way
12:19
of getting feedback on multiple, multiple episodes.
12:24
And it's not going to be, oh, well, Freddie,
12:29
you're such a nice guy.
12:31
Let me just give, no, it's very real and very blunt.
12:35
It's a lot of say it's fairly balanced as well,
12:38
because there is lots of places where it talks about the strengths,
12:42
as well, because obviously, it really wants to learn
12:44
from what's not working, and that's clearly important.
12:46
But clearly, it shows a lot of strengths,
12:49
and no one's taking that away.
12:52
And what I should add, as well, of course,
12:54
is it's really a supplementary to the human aspect.
12:57
You know, experts like yourself, Freddie, have worked in podcasting for many years.
13:01
It's not about replacing anyone, but it's actually trying to standardize
13:04
some of that feedback.
13:06
And sort of work together with you.
13:08
So the stuff you're noticing, yourself as a human,
13:10
and bringing empathetic as the sort of editorial assistant
13:15
to flag things that maybe you haven't considered.
13:17
And it's actually all right to disagree with some of it as well.
13:20
You don't have to agree with every word it says.
13:22
It's not going to be perfection.
13:24
It's, but it's an indicator.
13:26
It's like a decision support to say, maybe you should look at this a bit closer.
13:32
That's, that's probably what I was trying to save you time.
13:35
Right.
13:36
And that's what I love about this program,
13:39
is that it helps to augment you as the creator, not turn you into a robot,
13:45
turn you into something that's emotionless.
13:47
It's not trying to take away your personality,
13:49
because as you said, these are suggestions.
13:51
The ones, the episodes that I had ran through,
13:55
Pog Critic, were already published episodes.
13:58
So there was no chance of me going back and editing.
14:00
But it was really good to know that like, okay, this work, this work,
14:03
this definitely didn't work.
14:05
And so these are things that, you know, it's giving,
14:08
it was giving me that 360 degree view of my show,
14:12
things that I had no idea was happening,
14:16
was going on underneath the proverbial hood.
14:19
What do you, what do you see as, what's the podcast host
14:24
that would, that would likely get the most benefit?
14:28
I know this is kind of asking you to pick a favorite kind of a podcast format,
14:32
but like, is it more like the host who does interviews?
14:36
Is it more the solo host?
14:38
What's the, what's the feedback ban or what, what are you seeing?
14:42
Pog Critic could be used for any type of format, right?
14:46
From, you know, solo to interview shows.
14:49
But like I said, one that's an emerging format is also sort of,
14:53
you know, where you're scripting episodes or pre-publishing episodes.
14:57
So like the narrative, the agency obviously,
15:00
we're doing a six part sort of narrative based podcast
15:05
and we'll be working with them, hopefully, in May.
15:08
So they can pre, as they can analyse the scripts in Pog Critic.
15:12
And then that will give them indication, obviously,
15:15
what's working, what's not, what to change across those different episodes.
15:19
And then with the idea that it will save them time, both in the edit,
15:22
but also when presenting it to the client,
15:24
who, you know, to reduce the back and forth and the changes
15:27
and things like that.
15:28
Hopefully it spots things before that and speeds up.
15:31
That's what we're hoping to see anyway.
15:34
So it's quite an interesting piece.
15:35
So it can be pre-recorded, can be scripted, can be interview
15:39
or solo or panel discussion.
15:42
It's for any type of format, I'd say, yeah.
15:47
Pog Critic.ai is the place to go to get connected with Nile
15:50
and the team at Pog Critic.
15:52
I've had the pleasure of sampling the product
15:55
and I love it.
15:56
It's one of those things, Nile,
15:58
where with whatever someone's favourite chatbot is,
16:04
I tend to use perplexity.
16:06
A lot of people use chat GPT, maybe clot or some other model or whatever.
16:11
And it gets to a point where the chatbot gets to be a yes man.
16:17
And that's where, that kind of goes back to what I was saying
16:21
that as a creative we sort of get high on our own supply.
16:24
And Pog Critic provides the checks and balances
16:29
where it's like, hey, this is good.
16:32
Also, here's a pattern that we're noticing
16:35
and you may want to work on it.
16:37
And so it's up to the human as the host to decipher what it is.
16:43
Well, maybe they like that part and they want to keep it.
16:47
Okay, well, Pog Critic flagged it,
16:50
so proceed at your own risk.
16:52
Or it's one of those things where Pog Critic picks up something
16:55
in the first 30 seconds.
16:56
And it's like, I had no idea that I was doing that.
16:59
So I'm going to stop.
17:01
So it's all really, really great contacts, great feedback
17:05
that this program provides.
17:08
And I wish you much success
17:11
with the within what is this here first year
17:14
within the first year of launching this.
17:16
So congratulations and best of luck to you, my man.
17:19
Hi, thank you.
17:20
Thank you.
17:21
And yeah, if anyone's interested,
17:23
you can Pog Critic dot AI
17:25
or connect with me on LinkedIn.
17:27
So it's another way to do that.
17:30
But yeah, but now I'm thank you to you as well.
17:32
Freddie, she's been very open and to testing and discussing
17:37
and sort of getting feedback
17:38
and to the whole podcasting community.
17:40
There's been lots of names, like I mentioned,
17:42
like Rina, for example, Frubert as well.
17:44
He's been fantastic.
17:45
So just to give her a shout out as well.
17:47
It's been great to connect with the podcasting community
17:49
and just learn and evolve and it's still evolving now.
17:51
So yeah, positive for the future.
17:54
All right, thank you so much now.
17:55
No, thank you.