Latest / eCommerce Podcast / Is It Time We Talked About Turning Off Your Ads? (And Yes, You Read That Right)

Transcript

Is It Time We Talked About Turning Off Your Ads? (And Yes, You Read That Right) · eCommerce Podcast

0:03
So hello and welcome to the eCommerce Podcast. My name is Matt Edmundson and it
0:08
is great to be with you on this exceptionally hot day here from sunny England.
0:13
As we are recording this, we are in the midst of another heatwave. Yes, we are.
0:17
So I'm calling you from my non-air-conditioned studio, but I'm not bitter in any
0:22
way. No, I'm not. It's great to be with you. For those of you who've never been
0:26
to the show before, we talk about ecommerce, I myself run ecommerce businesses,
0:31
have done since 2002. So I've been around a while and I just love to get guests
0:36
on the show who are experts in their field, just like Brendan today, and just
0:40
grill them for an hour on their topic of choice and learn a lot. So I'm here to
0:45
learn just as much as you. If you're like me, you're gonna have a notebook,
0:49
you're gonna have a pen, you're gonna make lots of notes as you go through. But
0:54
if you don't have a notebook and pen, don't panic. We've still got you covered
0:59
because there'll be an in-depth blog post about today's day's show with all the
1:03
notes, the links from the guests and all that sort of stuff. We put that on the
1:08
website at ecommercepodcast.net. You will find that there. It's all free. And
1:12
you'll also find all kinds of other things at ecommercepodcast.net. So go check
1:17
out the website and find out all the amazing tools and archives and everything
1:21
that's just on that website, which will help you run your e-com business. But
1:26
yes, and of course, if you're welcome, if you're coming back to the show, if
1:31
you're a regular subscriber, Don't let me, don't let me neglect saying a very
1:35
warm welcome to you. You are absolutely a legend. Thank you for subscribing.
1:40
Thank you for keeping listening. Thank you for all your support. It keeps us
1:44
doing what we're doing. So really appreciate you guys tuning in week in, week
1:49
out. And hopefully wherever you are in the world, you have air conditioning
1:54
right now. Now, that said, let's get Brenden on. Brenden, how are we doing?
1:59
I'm doing well. I have AC, so I'm actually pretty comfortable right now.
2:03
This is where you get to rub it in, right? That's okay. That's okay. We were
2:07
talking about air conditioning before we hit the record button about whether we
2:11
should start a transatlantic air conditioning company.
2:15
I think we should. I think there's a big market for it right now. I bet every
2:18
single person in the UK would buy it.
2:21
Yeah, I think we could do pre-sales, pre-orders. We would clean up.
2:26
I mean, you're the guy to do it, right? Like, we can sell e-com, we can, you
2:29
know what I mean? You could buy it all online.
2:32
Yeah, yeah, yeah, absolutely. Well, I tell you, I'll do the website, you do the
2:34
marketing, we've got a business.
2:36
If we want the marketing to be really bad, yeah, sure, I'll do it. I'll do it,
2:40
whatever you want.
2:43
Brilliant. Absolutely brilliant. So, Brendan, for those of you who, uh, for
2:46
those that don't know you, as well as air conditioning, uh, just give us a quick
2:50
sort of, uh, 30-second elevator pitch about who Brendan is and what you get up
2:53
to.
2:56
Yeah, so it all started with my mom and my dad. No, I won't go that far. I won't
3:02
go that far. No, I've been— yeah, I'll give a serious intro. I've been in paid
3:07
media for the last 11 years working with a lot of different brands from B2B to
3:12
e-com to lead gen to pretty much everything under the sun. I've worked for small
3:17
mom-and-pop shops to large Fortune 500 companies at large agencies. But most
3:22
recently I've left my last agency almost 3 years ago to pursue a causal
3:27
measurement company. So like measuring what's causing sales for larger e-com
3:32
marketers in a company called Stella. Yeah, I don't know how in-depth I wanna
3:37
be, but Stella does incrementality testing, media mix models, and then always-on
3:42
incrementality to help brands truly measure what's causing growth. invest more
3:47
in what's actually, you know, improving the P&L.
3:52
Yeah, yeah, yeah. Well, we're going to get into all of that. I love this causal
3:56
measurement phrase. I think that's quite— I'm gonna use it a lot.
4:01
Yeah, I made it up just right now. I just made it up.
4:04
Brilliant. Well, let's just capitalize on that. So you've been doing this 4
4:08
years, you say? Right? You did, you sort of left the agency and started this 4
4:13
years ago.
4:17
A little under 3 years. But yeah, we started Stella. We were doing causal
4:22
measurement at the agency. It kind of came like, I tell this story a lot because
4:27
it's like, well, one, it's a true story, but it's like the only story. It's like
4:32
basically when I was a media buyer, maybe like 2 or 3 years into my journey as
4:36
like a full-time media buyer, I had a pretty large account and they, you know,
4:41
The marketing leaders there wanted me to hit a ROAS target. So across Meta,
4:46
Google, Criteo, and some other platforms at the time, I can't remember what we
4:51
were all on, maybe like Simplify or something. But I hit all the ROAS targets. I
4:56
like smashed them. I even exceeded them. And when that QBR came for me to
5:00
present the results, I presented like record-breaking ROAS. Like I hit the ROAS
5:05
goal, but I got, it was, I always say on calls I got pressed, but really it's
5:10
just the CFO was curious as to like, Why is revenue down? Like, if you're
5:15
showing us that like ROAS is so high, like, why am I not seeing that in our
5:19
actual bank account? And, you know, that sat with me for a while. I took time to
5:24
think through it. And what I— the conclusion I came to, which a lot of marketers
5:29
are starting to come to, is ROAS is not correlated with profit, even though a
5:34
lot of people think ROAS and accounts are. So that's how I got started. So at
5:39
the agency, while I was still managing accounts, I was really getting into
5:43
causal inference or causal impact, geo holdouts, media mix models, because I
5:48
started being able to lean on that type of measurement for truth. And obviously
5:53
when you're working on client relationships, it helps when you're speaking their
5:58
language. Like you don't care what the platforms say, you just really want
6:02
contribution margin or net profit to go up and to the right, which is exactly
6:07
what causal measurement does if you do it right.
6:12
Okay. Well, I, okay. I mean, I'm, I'm, before we get too headlong into this,
6:16
because I have a lot of questions already, um, you obviously worked with clients
6:20
varying sizes over the years. What's the biggest mistake there? I mean, maybe
6:25
you've covered this already with your You comment about ROAS not being
6:29
correlated to profit, but what's the biggest mistake we're making with paid
6:33
media that when someone comes to you, you're like, 9 times out of 10, this is
6:37
the first problem we solve?
6:42
I think it's, yeah, I, well, I think it starts with the understanding of what
6:46
you're looking at in an ad platform. Like what is ROAS? ROAS is a metric based
6:51
on correlation. Someone saw or clicked an ad and then later converted. So that's
6:56
why it deprioritizes certain channels like CTV or audio because you can't really
7:01
click, right? Like you try to launch a billboard campaign and you grade it based
7:05
on ROAS, like you're going to have a hard time doing that. And then as a
7:10
marketing leader, your goal, the way like you keep your job or your client is by
7:15
providing that ROAS that the client is asking for. So you start to, you know,
7:19
maybe invest more in retargeting or branded search because that's a quick win
7:24
where we can increase ROAS and the client doesn't know any better. But
7:29
eventually it'll catch up to you where the business is not growing, which
7:33
ultimately is kind of the path that I took a long time ago. And that's how I
7:38
fell into that mistake. Like if you want a high ROAS, I can get you a high ROAS,
7:43
right? But ultimately what I think it is, is business owners or people kind of
7:48
give, giving the orders downstream. Don't understand how ROAS or CPA or CPL is a
7:52
flawed metric. Because ultimately what every single business wants, which is
7:57
what I'm finding whether they say it or not, is net profit. Like if you have a
8:02
lead gen client, I know this is an e-com podcast, so don't, don't get mad at me,
8:06
but like you might, you might get a target of like MQLs, marketing qualified
8:11
leads, or SQLs, sales qualified leads. But ultimately the goalpost will keep
8:16
moving until you get down to bookings and customers and actual net revenue or
8:20
net profit. Like that is the goalpost will always move with e-com. It's much
8:25
closer because someone purchases and you see that sale come in. You have the
8:30
click-based attribution, you have the ROAS, but you don't know what caused that
8:34
sale to happen. And that's kind of the difference. So I think it's the biggest
8:39
mistake is like not fully understanding what multi-touch attribution is actually
8:44
doing behind the scenes.
8:50
Okay, so if this is the biggest mistake, I guess that we're making in not
8:55
understanding this multi-touch attribution behind the scenes and moving away. I
9:00
think people go with ROAS for 2 reasons. One is what everyone talks about and 2,
9:06
it's one simple metric, right? It's something that every platform tells you.
9:11
It's an easy thing to look at. So where do we start, Brendan? I mean, lead us
9:16
through the process of where we start to think differently about this.
9:22
Yeah, yeah. I think every e-com brand has launched like before the ads have
9:26
started, right? And then you launch ads, you launch Meta ads is usually the
9:30
first place people go and you immediately see lift. You immediately see like
9:34
sales coming in. So you, that's, those are causal sales. Those sales are caused
9:38
because you turned on Meta. And for smaller brands, it's not as much of an
9:42
issue, even though it's still there as like a what's causing sales to happen,
9:46
right?
9:50
Mm-hmm.
9:50
You launch Meta, you start to see immediate impact. Then you launch Google and
9:53
you see even more impact too. And I mean, there's a lot to be said about like in
9:57
e-com that maybe Meta is teeing up demand that Google's just taking credit for
10:00
and capturing.
10:04
Mm-hmm.
10:05
But typically when you're small enough, every new channel you launch, you start
10:10
to see this like Jump in revenue, jump in revenue, jump in revenue. But then
10:15
where incrementality really comes handy is for brands that are like making over
10:20
10 mil annually and they're running on like 2 to 3 or more platforms. And the
10:25
idea is for e-com businesses, usually the biggest expense are the ads. And
10:30
imagine if you turned off Meta and nothing happened, or if you spend $2 million
10:36
a month on Meta. Imagine if you spent, or imagine if you pulled back to 1.5
10:41
million instead, and that was actually optimal spend. Like the last 500,000 is
10:46
just completely wasted. You're past that point of saturation. So every business
10:51
deals with this issue. It just becomes a larger business or a larger issue at
10:56
scale, right? And starting small and understanding what's actually causing sales
11:01
to happen. is a great practice or habit to get into to allow you to scale. The
11:06
idea is we're trying to find what's causing sales to happen so we can invest
11:12
more in those regions, those areas, right?
11:17
Those channels. Okay. It's an interesting idea, isn't it? I suppose, does it—
11:23
does this idea of incrementality— you mentioned you should be turning over $10
11:29
million a year. What happens if I'm doing $3 million or $2 million a year?
11:36
Yeah, so the concept of incrementality, knowing what's causing sales, it still
11:40
applies. The methodology of measuring it is a little bit trickier, right?
11:45
Typically what I recommend, and a lot of people, because I talk about causality,
11:49
they think I'm against multi-touch attribution. I'm not. It's just, it's good to
11:54
understand what multi-touch attribution is, but for smaller brands around $3
11:58
mil, What I would recommend is making sure you have a proper multi-touch
12:03
attribution tool and then also setting up post-purchase surveys. So getting the
12:07
qualitative data and having your users tell you where they, where they came
12:12
from, also why they converted. I love asking the question on post-purchase
12:16
surveys of what almost stopped you from converting because that gives you tons
12:21
of insights as to like in your marketing messaging or targeting. But the
12:26
post-purchase surveys really tell you the why. And even like a brand new
12:30
business can start a post-purchase survey and a multi-touch attribution tool.
12:35
You don't have to be a huge spender to be able to start getting 2 different data
12:39
points. For example, I have a client I run ads for that is running heavily on
12:44
YouTube ads on Google.
12:48
Yeah.
12:49
And YouTube is typically infamous for having a terrible click-to-convert ratio.
12:53
Yeah.
12:53
So the in-platform ROAS is low. So if you're really scaling YouTube, your
12:58
account ROAS might be very low, right? Like if YouTube is the majority of your
13:03
spend on Google, your account ROAS is going to suffer. So we were having a
13:07
conversation around like, why are we spending so much on YouTube if YouTube has
13:12
like a 0.2 ROAS? But when I pull up the post-purchase survey data from no
13:16
commerce, Which shout out NoCommerce, I love their tool. You can see like, and I
13:21
forget the exact numbers, but it was like 7% of all purchasers, or at least that
13:25
filled out the survey, said they came from YouTube. And with that proportion of
13:30
revenue, 7% of revenue compared to how much we're spending on YouTube was
13:35
actually super profitable.
13:39
Yeah.
13:39
It was like a 3x return based on what we can take out of the qualitative data.
13:43
So now we have 2 different touchpoints, right? So we have the click-based
13:47
multi-touch attribution, and now we have qualitative data. And then once you get
13:51
to a level where you're getting enough volume of sales, you can start doing
13:54
proper holdouts.
13:59
Okay, that, that makes sense. I love the YouTube story. That's quite
14:03
fascinating. There's a couple of things. If I can dig into this a little bit,
14:07
Brendan, multi-touch attribution tools, what would you recommend and why?
14:15
Yeah. There's some I recommend. I feel like the owners of the companies don't
14:20
like me because I talk about causal measurement so much. Yeah, we've had some
14:24
exchanges in the DMs. I'll just say it, for e-com tools, I love Triple Whale and
14:29
I love Northbeam for multi-touch attribution.
14:33
Yep.
14:37
Yeah, I will say at Stella, we are building a multi-touch attribution tool as
14:41
well. However, we're not building it to be a big player in the multi-touch
14:46
attribution space. We just want to get closer to the full customer journey so we
14:51
can start assigning incremental values to every touchpoint. But yeah, I'd like
14:56
to plug Stella, but I honestly, I don't, the product is not there yet for the
15:00
multi-touch attribution side, but it is Coming, coming along.
15:07
As a little side point then, uh, as I'm busy, uh, writing notes here, I'm really
15:13
curious, uh, in your, so you're doing this multipoint attribution tool, um, or
15:19
MAT, uh, which might just be an easier way to say it. The, so you're thinking
15:24
about building one.
15:30
Yeah.
15:31
So my question to you is, is, is why? What is it that the current tools don't
15:35
have that you want to see in your system? What is the scratch that they're—
15:39
that's not getting— what's the itch that's not getting scratched is the right
15:43
phrase.
15:47
As to why we're building one?
15:50
Yeah. What is it? Yes. Why you're building it. What's it going to have that the
15:54
current tools don't have? If I can be as bold to ask.
15:58
Sure. I mean, multi-touch attribution is interesting. A lot of people think they
16:03
can vibe code a multi-touch attribution tool. So what goes into a multi-touch
16:07
attribution tool is a lot of pixels and working with like the DNS or your domain
16:12
provider to capture a lot of different signals and store those signals. So for
16:17
example, if I don't know who you are, I don't know you're Matt and you come to
16:21
our AC e-com website. to get AC in the UK. I don't know you're Matt until you
16:26
convert, right? But I can see your IP address, I can see your device ID, I can
16:31
see your UTMs, I can see your click ID. So I need to be able to anonymize you.
16:35
So you are now like an anonymous ID. Every time you visit the site, we can ping
16:40
the same anonymous ID.
16:44
Yeah.
16:45
We capture all of your touchpoints. So if you came in from TikTok, we pull in
16:50
Your TikTok UTMs and your TikTok click ID. You came in from Microsoft, we pull
16:55
that, Google, whatever, each interaction. And then once you convert and we find
17:00
out your name's Matt, we can stitch all that back and say, okay, Matt converted,
17:05
but his first time ever on the site was 3 months ago and he came in from a
17:10
TikTok ad. Or the last conversion was a branded search ad for Microsoft, right?
17:15
Mm-hmm.
17:15
So we can see like first touch or last touch. Or we could look at it like what's
17:18
called a linear attribution model where we fairly credit every touchpoint,
17:20
right?
17:23
Yeah.
17:23
So that's just the basics of any multi-touch attribution tool. What we're trying
17:28
to do at Stella is, and you'll find out with me rambling on this podcast,
17:33
causality is so complex. And I talk to marketers all the time that think they
17:37
know incrementality or like to trick themselves into like, yeah, yeah, we do
17:42
holdouts. But they really don't. And even if they do, they don't know how to
17:47
operationalize it. So it's not just about measuring, is something incremental,
17:52
yes or no? It's about how do we invest more in that? How do we change? What do
17:57
we, what levers do we actually pull as operators in Meta, Google, TikTok,
18:01
wherever to actually invest more in incrementality? Because sometimes, like
18:06
based on that example I gave with YouTube, sometimes Increasing incremental ROAS
18:11
is opposite of increasing platform ROAS. Like if I were to invest more in
18:16
YouTube, platform ROAS is going to drop, but potentially incremental ROAS will
18:20
increase. So as a media buyer, and then, you know, it's also a lot of like
18:25
changing how the entire organization thinks, because if the CFO is not on board
18:30
and they're still holding you to a ROAS target, you know, You're gonna have some
18:35
friction there where you're like, ROAS is flawed. I'm trying to track iROAS, but
18:39
then the CFO doesn't understand it. Anyways, the reason we're building it at
18:44
Stella, and we've already built it, it's just, and it works. It's just, I would
18:49
say the dashboards aren't fully there yet, but the functionality works
18:54
perfectly.
18:59
Yeah.
18:59
So the idea is we look at every touchpoint and we're running holdout studies,
19:04
we're running media mix models, we're running this automated causal analysis on
19:09
top of all of that. Um, to track how incremental each touchpoint is. For
19:13
example, if you came in from a Meta ad, a TikTok ad, and then converted from
19:18
branded search, if branded search is not— and spent, let's say you spent like
19:23
$100 or 100 quid, as you would say.
19:28
Very good.
19:29
Yeah, yeah, yeah. I'm learning. I'm bilingual.
19:31
Yeah, absolutely. This is really going to help our marketing for the air
19:34
conditioning company.
19:36
Yeah, yeah, yeah, yeah. We can say, we can, we can say we're bilingual. We speak
19:40
American English and original English.
19:43
Yeah, yeah, yeah. The OG.
19:45
So, so sorry, let me, let me follow this thought. I'm sorry. Let's say there's 3
19:48
touchpoints, right? Meta, TikTok, and then you convert from branded search and
19:51
spend $100.
19:55
Yep.
19:56
First touch attribution would say Meta drove that $100. Last touch would say
20:00
Google drove that $100. Every platform is going to say they drove that $100.
20:05
You, if you total up all of them, you'll see 3 conversions. You'll see 1
20:09
conversion from Meta, 1 from TikTok, 1 from Google within the ad platforms, but
20:14
really you only got 1 sale. So that's the difference. But the issue is maybe
20:18
branded search was just there to capture because someone was familiarized from
20:22
TikTok and Meta, then searched for the brand and was going to convert anyways.
20:27
So what our model does is starts assigning, like if we do a branded holdout, we
20:31
might find branded is not very incremental that if we we turn it off, we don't
20:35
lose as many sales to where we can pass back a portion of that $100 revenue
20:40
based on how incremental each touchpoint was from all of our historical analysis
20:44
that we've done for your brand.
20:49
Right.
20:49
So that's the take. So then what happens there is we can then pass back that
20:53
signal. So instead of optimizing your ad campaigns for purchase, which is
20:58
looking at gross sales, we can pass back incremental purchase That has full
21:02
context of your business: Shopify, Amazon, retail. It's looking at Google,
21:07
TikTok, Meta, and we can even go a step further and send back incremental net
21:11
profit or incremental contribution margin, so you can optimize your accounts
21:15
towards actually what's causing sales to happen. So that's that's what we built,
21:20
and I know that was a little complex, but we're trying to solve a complex
21:24
problem. We're trying to simplify it. enough to where marketers can plug in
21:28
their platforms, start measuring, start testing, and feeding back the right
21:33
signal to ad platforms.
21:37
Love that. It reminds me of a chap who came on the podcast a couple of years
21:42
ago, Neil Hoyne, who was a data guy at Google, right? He wrote the book
21:47
Converted. And he was talking about an example. He was like, how do you do
21:52
attribution? Because they tracked one lady who bought a single pair of shoes off
21:57
a website and there were 236 touchpoints with the company before she purchased.
22:02
Some were paid media, some were organic search, some were social media,
22:07
everything, right? And so Neil was like, at what point do you attribute that
22:12
sale? Because like you say, everybody claims it, which is why we have this
22:18
problem, isn't it, with ROAS? Because everybody's claiming sales to 100%, where
22:23
it could be, you know, split across all your channels. And so it sounds like
22:28
what you're doing, if I can put it this way, is you're solving as best you can a
22:33
complex attribution problem by assigning values to these multi-touchpoints,
22:38
right? To try and create a system which gives you a number at the end, which
22:43
makes sense to the net profit in your bank account. Versus what the platforms
22:48
are telling you versus what you understand about the business. And it sounds a
22:53
bit like hearing you talk about this, Brenden, that what it's not one size fits
22:58
all and each business might well be unique in how it attributes across the
23:03
multi-touchpoints.
23:08
Exactly. There's a lot of, we even released them at Stella, but like we released
23:13
like a bucketed report where We should be releasing one soon again of just all
23:17
the holdouts we've done. And then so advertisers can kind of read those reports
23:22
and see like what's the average IROAS of whatever platform.
23:26
Yeah.
23:26
But it really is different. So for example, the obvious, or the obvious example
23:31
that a lot of people use when they talk about incrementality is branded search,
23:35
right? Like if I'm typing in Matt's AC e-com brand, which I hope that will be a
23:39
domain you buy. I'm likely going to end up on that site regardless of if I click
23:43
a paid ad or if there's a competitor ad showing. If I have high intent to go to
23:47
your website, I'm probably going to make it to your website somehow.
23:51
Yeah.
23:52
So a lot of people point to like branded search is non-incremental. However,
23:57
I've worked with a few brands, even like at Stella, that they're, they're like
24:02
the big category leader in the space. So their name brand is the Non-branded,
24:06
like, I'm trying to think of what I can say. Basically, like, their name of the
24:11
company was also the name of the product they sell. Yeah. So for their case,
24:16
branded search was incredibly incremental because if, even if people didn't know
24:21
about that brand, if they're typing in that product and then there's a brand
24:26
that is like the product.
24:30
Yeah.
24:31
You know, they're going to convert a lot of people, especially the people that
24:35
are like, they want the quality product of what they're searching for. I know
24:38
I'm being vague. I'm just trying not to give away client details, but—
24:41
No, no, no. I totally get the point. I sell omega-3 and it's like if I called my
24:44
company Omega-3.
24:47
Exactly. Yeah, yeah, yeah. Like you are the omega-3. Yeah, it's pronounced
24:50
omega, Matt, by the way.
24:54
Sorry, I'll try my best. I'm very sorry.
24:58
That's okay. I'm sorry. But exactly. But then there's also the case of like
25:02
Maybe you're in an industry where a lot of competitors are being very aggressive
25:06
towards your brand terms and you having branded search is helping you drive
25:11
increment. So it really depends. And what I always say is, I met a guy who said,
25:15
I don't need to test. I have all these benchmarks from everyone putting out
25:19
data. And I was like, it's really super different for every brand. You should be
25:24
testing yourself.
25:28
Yeah.
25:29
As well as you can.
25:32
Well, let's, if we can, let's get into testing. And I want you to first, if I'm,
25:36
if I can ask you to just define for the sake of clarity, a few terms which
25:41
you've used a couple times for those that might not know, iROAS and holdouts.
25:46
Sure. iROAS stands for incremental return on ad spend, which is ultimately what
25:51
most, most brands that are optimizing for a ROAS target, what they really want
25:55
and they don't know is the incremental ROAS target, which means every dollar we
26:00
put into an ad platform, we get back money that would not have happened without
26:05
that ad platform, without that investment. That's what we like to think or trust
26:10
is what ROAS is telling us. But like I said earlier, based on how ROAS is made
26:14
as a metric, It's correlation-based. It does not actually tell— it tells you who
26:19
saw or clicked an ad and then later converted, but it doesn't tell you what
26:24
caused that revenue to come back. So that's what I ROAS is, the incremental
26:29
return on ad spend.
26:33
Fantastic.
26:34
Incrementality or holdout studies are ultimately like the easiest way to explain
26:40
it is just simply turning off ads in certain regions and monitoring your source
26:45
of truth revenue. So for a lot of e-com brands, that's Shopify. in those regions
26:50
compared to the regions that stayed live. And there's a lot of statistical
26:55
nuance that goes into it, but that's ultimately the core of what holdout studies
27:00
are. You're holding out a region and you're trying to see, does revenue drop in
27:06
that region at all? And if so, how much? And can we be confident that that drop
27:11
is because we turned off the that media channel.
27:18
So how do we, how do we do these holdout tests then? Because I can hear however
27:21
many thousand people listening to the show right now going, hang on a minute,
27:24
you want me to turn off ads? You want me to do what?
27:28
Yeah, yeah. And we get that a lot. We actually signed a client that for Stella,
27:32
that was not aware that they were going to have to turn off ads, even though we
27:37
like explained it to them. They're like, we're gonna have to turn off ads. And
27:41
I'm like, yes. So yes, so there is a risk, right? When you turn off ads, if the
27:46
ads are incremental, there's a risk of losing revenue that would've been caused
27:50
by those ads being live. However, the whole point is to measure, to see what is
27:55
incremental and how incremental it is.
27:59
Mm-hmm.
28:00
Because it's not about, you know, it's not about is this platform incremental,
28:05
yes or no? It's about how incremental is it right now at our current spend point
28:10
and how incremental could it be potentially? We might be underinvested in
28:15
multiple channels and we might be at a point or past a point of saturation for
28:21
other channels, right? Like we might be with the exact same total budget. We
28:26
might be able to drive more revenue if we just change the allocation. So that's
28:31
the part where Like, it is a risk, but the idea is the value that comes from,
28:36
um, the insight can actually propel the business much further.
28:41
So how do—
28:43
where do I—
28:44
I guess my question is, let's say I'm running, I don't know, 100 different ads,
28:49
um, across Meta, um, I'm doing Google Ads, I've got my Amazon paid things going
28:53
on because I'm trying to sell on Amazon as well. I've got, I mean, I've got
28:58
multiple different places. Am I doing like one at a time? Am I doing, what's
29:02
the, I guess, what's the strategy of attack if I'm going to start to do this
29:07
that I need to think about?
29:11
Yeah. I mean, that's the fun part. That's like the part we get to figure out.
29:15
Um, so what we typically recommend at Stella is to start at a channel level
29:19
holdout. So let's turn off all of Meta in certain regions. And there's different
29:22
types of holdouts.
29:26
So regions, you mean geographical regions?
29:29
Geographical regions, yeah. So we want to make sure no one in that area is
29:33
getting any, getting served any impressions from whatever channel we're testing,
29:37
right? Because then we want to see like, you know, the drop in revenue that
29:41
starts to come and then how, how long, how long does it take for us to see that
29:45
drop so we can start looking at the lagged effects of Meta as well. Usually
29:49
holdouts run for 20 to 30 days depending on the channel.
29:53
Oh, wow. Okay.
29:55
CTV or like more top of funnel channels might run for 45 days. But usually
30:00
that's what we recommend is between like 20 to 45 days at maximum for a holdout
30:04
study. You can also, the other side of this is actually turning on ads in
30:09
certain regions. So like if you're not activated on like Twitter ads or I don't
30:14
know, like Tatari or CTV, You can activate in certain regions and hold out other
30:18
regions too to see an incremental lift in those regions, which is a little less
30:23
scary for brands, right? But yeah, so we're looking at geographical areas to
30:27
turn off because we don't want to serve any impressions. And then we want to see
30:32
how does that affect our actual source of truth revenue. Oh, so yeah, yeah, I'm
30:37
sorry. I was Yeah. So we start with the channel level. So all of Meta, all of,
30:41
and then once that 20-day increment is up, you go to Google, you go to TikTok,
30:46
every other channel you're on. The idea here is we're trying to establish
30:51
incremental base, baselines. We wanna see how incremental is Google as a whole.
30:55
And we all know Google has tons of different types of campaigns. We have PMax,
31:00
we have search, we have shopping, we have YouTube, we have demand gen, we have
31:04
display.
31:09
Yeah.
31:10
So we can look at, okay, we turn off Google and we lose X amount of our revenue,
31:14
which gives us a 3x incremental ROAS of Google. Now we can start testing again
31:19
with a, with a new process where we go into each tactic. So maybe we hold out
31:24
all non-branded campaigns and we keep live all branded campaigns. So we can
31:28
start to see like what's weighing that ROAS from Google down and what's pulling
31:33
it up, right? So Where should we be investing? And then you can do the same on
31:38
Meta. Like depending on how your Meta accounts are set up, you could do a
31:42
holdout on just acquisition campaigns, like just prospecting campaigns compared
31:47
to retargeting, or depending on your spend level, you could just do a holdout on
31:52
one specific campaign at a time. So you can start to assign the incremental
31:57
value of each individual campaign. So it's kind of like this cyclical pattern
32:01
where you do channel level. tactic level, campaign level.
32:06
Yeah.
32:06
And once you finish that sequence, you should actually restart and go back to
32:11
the channel level. The idea being incrementality and holdout studies are a
32:16
snapshot in time. It all depends on macroeconomic factors, the ads you're
32:21
running, your bid strategy, the algorithm that Meta has right now or Google,
32:26
right? Next month, things might change. How incremental your ads are on Meta, On
32:30
Meta in February is going to be so different than your ads in November when like
32:35
Black Friday, Cyber Monday come up. So it shifts over time. So it's, it's not
32:40
enough to do one holdout and like point to that number. We should be doing
32:45
constant holdouts. And of course, I'm the guy selling you the holdouts. So like
32:50
I'm biased, but the idea is you can't just reference one holdout for like the
32:55
next 2 years. It had—
32:59
Yeah.
32:59
It will shift. And that number becomes outdated, something we call causal decay.
33:03
Like the moment you get that IROAS report back, the next day it's going to start
33:07
to shift.
33:11
This is really fascinating. I guess, how do you deal with, like I say, the 1,000
33:16
people listening to the show going, hang on, I'm going to lose sales here. What,
33:21
what, what have you seen customers gain? Because I think that There's going to
33:25
be a large amount of inertia to this idea, which I understand. But obviously you
33:30
do this successfully. And so you see customers gain, obviously, insight, you can
33:35
reallocate budgets better and more efficiently. But how long is it before, I
33:39
guess, people start to see the payback? How deep do their pockets need to be to
33:44
ride this out? I'm, I'm, I'm, you must get asked these questions all the time,
33:49
right? And I'm really curious.
33:53
Yeah. So I think the funny part is, like I said, like there's a lot of marketing
33:58
leaders that are diving into incrementality or causal measurement because
34:02
they're hearing it a lot. They see a lot of blog articles. They're gonna watch
34:06
this podcast with me talking about incrementality. They see people like they, I
34:11
think there's a shift in the market right now where people think that's where I
34:15
need to be. I need to be around the incrementality, but there's like a lack of
34:19
understanding. Right? Like there's a lack of understanding how to use it. So I
34:24
even see people at Stella, like we try to work with our clients directly. Stella
34:28
is a self-serve platform, but we do have some clients that are like on a
34:32
consultancy type of retainer where we are like making changes for them. But the
34:37
idea is incrementality studies are just one part of it. It's like a snapshot in
34:41
time of how incremental something was at that point in time, at that spend
34:45
level, with that bid strategy, with your ads, with however many competitors are
34:50
in your market. It's a snapshot in time. What we think is the most powerful way
34:54
of doing causality is then moving into another form of statistical measurement,
34:58
which I'm— I know, I know it's hard to follow. I'm hardly— I'm hardly following
35:02
what I'm saying right now. Also, you booked this at 4 AM Eastern time. So like,
35:07
I am like, it is very early right now. So it's not though. It's like, it's 9:42.
35:11
We're good.
35:15
Yeah. And you booked it. I just want to be clear. Anyway, carry on.
35:18
Whoa, whoa, whoa.
35:19
Don't point fingers, Matt.
35:19
When you Point a finger at me, you point 3 back at you. Yeah, yeah, yeah, yeah,
35:24
yeah, yeah, yeah, yeah. It's the heat in your room. It's getting to your head.
35:29
It is. It must be getting to me. Yeah, yeah, yeah, yeah, absolutely.
35:32
Yeah. So, um, so holdout studies are one form of— now I have to be serious
35:35
again.
35:38
Hold on, let me go serious face.
35:41
Okay, so holdout studies are one form of statistical methodology, but it's a
35:45
snapshot in time. There's another form of statistical methodology that we also
35:49
support at Stella called Media mix models. Media mix models look at like a
35:54
2-year time span of your data and it starts looking at other things like what's
35:58
something called Fourier contribution, which is seasonality. So it looks like
36:02
how do ads perform in March compared to April compared to May? It starts to look
36:06
at things you can add in control variables of like, this is when I have sales
36:10
throughout the year. These are the emails I send. This is the non-branded Uh,
36:14
term on Google Trends, like you can see kind of like, okay, the popularity of
36:18
the industry is increasing or maybe it's decreasing. Maybe a new competitor
36:22
launched and you see their search queries start to increase because they're
36:27
taking up more market share. Yeah. Um, media mix models are more holistic and
36:31
you can calibrate them with holdout studies. And it's actually better if you do
36:35
calibrate with a holdout study. So the idea then becomes media mix models can
36:39
start looking at forecasting. So a media mix model can understand the variance
36:43
in your data, like what causes revenue to go up or down. Again, calibrating with
36:47
a holdout weights it in a different form of causal analysis too. So now we have
36:51
2 different forms of causal analysis working together. But then once it
36:56
understands like, okay, if we spend an extra $100,000 on Google, we know how
37:00
that should affect revenue.
37:04
Yeah.
37:04
We know how ads should perform in April or May or like whatever the next quarter
37:09
is. Like we understand that seasonality. Media mix models are great at then
37:14
using what's called like a budget optimizer or a scenario planner. Most media
37:19
mix modeling tools have something along those lines. Obviously we have one at
37:24
Stella. So what we do is we build a media mix model that is very accurate based
37:29
on metrics like MAPE, R-squared, in-sample, out-of-sample. There's a lot of ways
37:34
to verify how accurate a media mix model is. And then you go into the budget
37:39
planning or a scenario planning and you forecast out the next like 3 months. So
37:44
like, this is our budget for the next quarter. This is, uh, you know, how, what,
37:49
how can we use this budget to maximize incremental revenue? And that's what
37:54
media mix models will tell you. It'll tell you, you know, this channel's at
37:59
saturation, we need to pull back. This other channel, we need to increase
38:04
revenue. And this is With your given budget, this is how you maximize
38:09
incremental revenue. So that's the power of causal measurements, this umbrella
38:14
that has holdouts and media mix models. But our opinion, yeah, our opinion is
38:18
you should use both. You shouldn't just do holdouts or just media mix models.
38:23
You should be doing them both together because media mix models are more
38:28
forward-looking to understand How much can I be spending on these platforms
38:33
before my next dollar is not returning as much as it could be somewhere else?
38:38
Yeah, which is incredibly powerful information.
38:41
Exactly. But like, I— and I understand it because I live and breathe it now. I
38:45
didn't understand it as much as I would have liked to when I was at the agency,
38:49
right? Like, because obviously I'm doing this every day, but I'm still seeing
38:54
that pattern emerge where Maybe some marketing leaders understand that it's
38:58
powerful, but it gets over— it gets overwhelming looking at all this data
39:02
science because they're marketers. They're not data scientists, right? They're
39:06
not statisticians. So that translation into what I'm showing you and then what
39:10
it's telling you, like what to do next, that's what we're trying to bridge at
39:14
Stella. So not just measuring causal growth, but knowing what to do next, where
39:18
to invest.
39:22
Yeah.
39:23
increase causal growth.
39:25
That's the, the trick, isn't it? Because, um, I can imagine actually a lot of
39:29
people listening to the show are going to go, well, this sounds all great and,
39:33
um, I like the idea of it, but how I get started and how I maintain this is a
39:37
very different thing. Because it's— it doesn't sound like it's something that I
39:41
can just go to Claude and go, Claude, could you do that for me? Um, I, I—
39:46
it's—
39:46
it sounds like it's a lot more involved, which I then puts a high barrier to
39:49
entry on it for, I would have thought, a lot of small business owners.
39:53
Yeah, exactly. Like I said, like, I think it's just important. You don't, you
39:57
don't necessarily have to do holdout studies, right? Like, that is the— people
40:01
call it the gold standard of measurement. I don't know who came up with that,
40:06
but I love it because it's good for business. But yeah, but you don't have to do
40:10
a proper holdout study to understand the concept of incrementality and know even
40:15
if you're making $1 million or million quid just for your audience a year with
40:19
your e-com business, like you're still dealing with these same issues. They're
40:23
just at a much smaller scale. And as you scale, it's going to get harder and
40:28
harder to know what's causing growth to happen.
40:32
Yeah.
40:33
So it's important to not like, I guess the main message for like the smaller
40:37
advertisers, smaller e-com brands is Understand what ROAS or CPA is actually
40:41
tracking. Make sure you have multiple forms of measurement. What I like to do is
40:46
multi-touch attribution, post-purchase surveys, and then when you're big enough,
40:50
causal analysis too. I think that really closes the loop on like what's actually
40:55
causing growth for your company.
40:59
Yeah, yeah.
41:00
But a lot of people are stuck on just multi-touch attribution, or even worse,
41:04
just looking at in-platform metrics, which eventually will fail you the way they
41:09
failed me. Yeah. So I think that's the biggest thing is like, you don't have to
41:13
do holdouts to understand the concept of what's causing sales to happen might be
41:17
different than what you're seeing in ad platforms.
41:21
Yeah. Fantastic. I— one question, I suppose I'm thinking about this. If you use
41:26
different agencies across the ad platforms, like you've got a Facebook agency
41:31
over here or Meta agency, you've got a Google guy over here. Um, with this kind
41:36
of system, are you better off having, I guess, one agency across all platforms
41:41
that understands what's going across everything, or does it still work
41:46
effectively with multi-agency?
41:51
So I, I've just always been a thing, regardless of causal measurement, I think
41:55
it's really powerful to have one agent agency managing your entire ecosystem
41:59
because there's better collaboration.
42:03
Yeah.
42:03
I think my clients, so there's a few consulting clients I work with like
42:07
directly and like I control the whole strategy of what's happening with their,
42:12
you know, with their paid media. And I think that is so effective rather than
42:16
having so many cooks in the kitchen and not understanding like, yeah, you know,
42:20
we pivoted Google to this strategy because we wanna see this, but then on the
42:24
other side, your Amazon agency sees an effect happening and they need to spend
42:28
or else they don't get their percentage of spend. And like all the incentives
42:32
are like misaligned. So I'm always a big advocate of just having like one agency
42:36
or one main brain for the strategy. Obviously there's like the VPs of marketing
42:40
at the company that could kind of control that. But I think without the direct
42:44
communication, I think also sometimes agencies feel competitive that like maybe
42:48
you have an agency running your Meta and a different one running Google, but
42:52
maybe the Google agency also has a service line.
42:56
Yeah.
42:57
Where they run Meta, so they might be trying to poke holes in the Meta strategy
42:59
to get that business.
43:01
Mm-hmm.
43:02
I'd rather just have it all to one. When it comes to causal measurement, it is
43:07
hard to coordinate all this. It just makes it harder, especially when there's
43:12
more cooks in the kitchen. What I would say is it can come down to the main
43:16
decision maker, the marketing leader that understands where we need to be
43:21
investing in the ad platforms. Stella, we have a tool we call the Always-On
43:25
Incrementality Tool. It does an automated causal analysis. We're very candid
43:30
with people that like the more you automate, the less accurate it becomes.
43:35
Yeah.
43:35
So we always recommend do a proper holdout, do a proper MMM, and then calibrate
43:40
the Always-On tool. The Always-On tool will show you at a campaign level where
43:44
to be spending that week to maximize incremental returns on whatever sales
43:49
channels you've connected. So it can be Shopify or it could be Shopify and
43:54
Amazon combined if the company wants that. So if all your agencies are plugged
43:58
into Stella and can see like what's causing growth to happen, the idea is Stella
44:03
becomes the central brain of the operation that all your ad buyers can start to
44:08
see if everyone's aligned on the same goal of driving incremental growth rather
44:12
than just like platform reported ROAS.
44:17
Mm-hmm.
44:18
that's a great place to start. I think, yeah, I don't know. I know I was
44:22
rambling. But basically, like, I think it's always a net positive to have like
44:25
less people controlling the strategy.
44:29
Yeah. Very good. And the— just quickly, well, actually, let's do this. Because
44:34
I'm aware of time. So first things first, question for Matt. This is where I ask
44:38
you for a question for me. And I will go away and answer on social media. What
44:42
is your question for me, Brendan?
44:47
Yeah, I, I want to know your craziest, like, client story or your craziest,
44:51
like, account story. Like, if you've worked with other marketers, um, I have a
44:55
bunch in my head that I don't think I can ever say, but I want to hear from you.
44:59
I want to hear what your story is.
45:04
Crazy, crazy account. Okay, I'll answer that on social media. If you want to
45:07
know my answer to that question, come follow me at Matt Edmundson. But Brendan,
45:10
listen, I— if people want to reach you, if they want to connect with you, they
45:13
want to find out more about Stella, what's the best way to do that? Where do we
45:16
go?
45:21
Yes, if you're a serious marketer and you want me to be serious and give you
45:26
real advice, you can follow me on LinkedIn at Brendan de la Rúa. That's where
45:30
I'm a little bit more buttoned up, as buttoned up as I can be. And if you want
45:35
more jokey, haha content, that's still educational and talking about marketing
45:39
and causality, you can follow me on my TikTok @brendanbuilds. I try to play a
45:44
good balance on TikTok and LinkedIn and Twitter where I'm posting different
45:49
things on all the platforms. On LinkedIn, if you wanna really learn about
45:53
incrementality, that's where I would follow me. If you want a goof and a gaffe
45:58
and some sarcastic content, I'd follow me on TikTok.
46:02
Always like sarcastic content. Uh, we will of course put those links in the show
46:06
notes. So if you're listening on a podcast player, just scroll down, uh, you'll
46:11
see the show notes, the links to Brendan. Uh, if you're watching on YouTube,
46:15
just click in the description, the links will be there. Of course, if you're
46:19
subscribed to the newsletter, they'll be in your inbox. Um, and they'll be on
46:23
the website, like I say, with the blog post, they're going to be everywhere. And
46:27
if you're a Slingshot AI user, just ask Sam to put you in touch with Brendan and
46:32
he'll be It will do it. So Brendan, listen, firstly, thank you. Great
46:36
conversation and eye-opening in many ways. And I think you've answered a lot of
46:40
questions people have about ROAS and why it's not working well for them. But
46:44
before we go, we like to do this thing called saving the best till last. And
46:48
this is where I would love to hear your top tips on the post-purchase survey. So
46:53
you've mentioned that a couple times as one of the key things to do. And you
46:57
mentioned one of your favourite questions was what almost stopped you from
47:01
buying, which I think is a great question, or words to that effect at least. But
47:05
if I'm going to start to think about these post-purchase surveys, again, people
47:10
get very nervous because they think it's friction. What's your top advice for
47:14
anyone running an ecommerce business out there? The microphone is yours for the
47:18
next few minutes. Over to you, my friend.
47:22
Yeah. One shout out, Jarrell Blades, the VP of growth at Tushy. He's the one who
47:26
really explained that to me about asking what almost stopped you from
47:29
converting.
47:33
Right.
47:34
I think, you know, there's a book, The 7 Habits of Highly Effective People, that
47:37
talks about begin with the end in mind. So when you're building out a
47:41
post-purchase survey flow, you kind of have to wonder like, what are the
47:44
questions that are gonna help me actually optimize the funnel? Because
47:48
post-purchase surveys are happening after someone's already made the purchase.
47:52
You've already converted them, but now the idea is take as much information as
47:55
you can that's going to help you convert more people better, right? Like, what's
47:59
the reason they're buying it? I work with a lot of supplement brands that, you
48:03
know, have people have different ailments or injuries that they might be buying
48:06
a certain product, and there might be new ones that start popping up, uh, and
48:10
where we can start making ads specifically on whatever they're talking about,
48:13
right? Yeah. So appeal to a wider audience people, which allows us to scale
48:17
meta, right? Post-purchase surveys are just, they also tell you like more word
48:21
of mouth where on marketing, everything's going to be click-based, like on
48:24
multi-touch attribution, you're going to see someone clicked an ad or saw an ad,
48:28
but really what caused them to convert was their friend told them about it, or
48:31
they went over to their friend's house and used their shower and they had this
48:35
like shampoo they've never had before. And it, you know, made them go bald or
48:39
maybe opposite.
48:43
Yeah.
48:43
Grew their hair. It was finasteride the whole time. You know what I mean? And
48:47
then they go and buy it and that's why, and they ended up clicking an ad because
48:51
they saw it. Right. But I really love post-purchase surveys. Like any company
48:55
can have them, lead gen, e-com, B2B. Like why did you convert? What ad do you
48:59
see? There's some post-purchase surveys, like if you use NoCommerce on their
49:03
premium plan, they have conditional logic. So if you say, I saw an ad from Meta,
49:07
that might serve you. The next question is, which ad did you see? So now we can
49:11
start asking like, which ad? Was it a video? Was it a static image? And a lot of
49:15
times when you're going through the customer journey in general, you're seeing
49:19
multiple ads. So the ad that they're gonna tell you they saw, if they're being
49:23
truthful, there's also that caveat with post-purchase surveys. Yeah. Some people
49:27
just click whatever.
49:31
Yeah.
49:31
Is the ad that was most memorable, the ad that they remember the most, right? It
49:35
might not be the one that they just clicked. Clicked to confirm. 'Cause we can
49:38
QA that with the UTMs, with the post—
49:41
Yeah.
49:41
With the multi-touch attribution data, we can see which ad they clicked on, but
49:44
maybe they're saying they remember an ad that they did not click on.
49:47
Right.
49:48
That we didn't see. And we're like, wow, so this ad must be very impactful here.
49:52
So that's what I love about post-purchase surveys. We're also building one at
49:56
Stella. It already works. We're just waiting for approval from Shopify. So we'll
50:01
have the full brain within Stella, where you'll have your causal measurement,
50:05
you'll have your multi-touch attribution and your post-purchase survey all
50:09
within Stella. So we can start tying back the entire customer journey and
50:13
sending back the right signal to all of your ad channels, which I think is going
50:18
to be really cool. But it just comes from the love of post-purchase surveys and
50:22
the value that you get from that qualitative data.
50:26
Fantastic. Fantastic. Brandon, listen, uh, thank you so much for coming on the
50:30
show, man. Really enjoyed the conversation. Love what you guys are doing at
50:34
Stella and, um, hope the launch goes well of all the different myriad tools
50:39
you've got coming up, uh, anytime soon. Um, love the sarcasm, uh, and appreciate
50:43
the, uh, the tip on air conditioning.
50:47
Matt, I will, I will close Stella. I will end Stella. You know, I don't even
50:51
care. You and me, if you're down, I want to start the AC company. There's so
50:56
much more opportunity in that than what I'm doing.
51:00
There is. There is. Yeah. And the other one I thought about, uh, whenever I come
51:03
to the States, I'm like, we should do solar power here in America because it
51:07
just seems crazy expensive to me. And in America, for some reason, I'm not quite
51:10
sure why. Um, whereas in the UK, it seems to be much more sensible and we have
51:14
less sun.
51:17
Our constitution says, uh, we don't like solar panels. Solar power, uh, the
51:22
founding fathers said do not. Get solar panels. We actually get a lot of people
51:27
door-to-door selling solar all the time, and we're like, you ain't from here.
51:33
You ain't from around here.
51:38
Yeah, we like fossil fuels. We like pollution. Yeah, that is American. That is
51:41
uniquely American.
51:45
Yeah, definitely. I get it. I do. I do. Brilliant. Yeah, what a legend. Really
51:48
enjoyed this conversation, Brendan. Thank you so much, man. Really appreciate
51:52
it.
51:55
Yeah, thank you for having me.
51:57
Well, there you go. Another fantastic conversation lined up on the eCommerce
52:00
Podcast. Wasn't that great? Loved it. Make sure you do connect with Brendan
52:04
across his various channels, whether it's LinkedIn, whether it's TikTok,
52:08
whatever it is. I'm sure he would love to hear from you. But yeah, everything,
52:12
like I said, about eCommerce Podcast on the website ecommerce podcast.net. You
52:16
can find everything out about us. Uh, but that's it from me this week. Have a
52:20
phenomenal week wherever you are in the world. Uh, but from Brendan and from
52:24
myself, bye for now.