Latest / The Yield Doctor with James Deaker / Alex Roucourt on Adverting Uplift

Transcript

Alex Roucourt on Adverting Uplift · The Yield Doctor with James Deaker

0:00
Hi, I'm James Deika, the yield doctor,
0:04
and today I'm going to be talking to
0:05
Alex Rukort about challenges in
0:08
assessing advertising uplift in modern
0:11
advertising and some techniques to
0:14
address them. Alex has spent many years
0:17
in digital advertising. He spent time at
0:20
companies like Meta, Marin Software, and
0:23
Apps Flyier. Most recently, he and I met
0:27
when he was working with Dennis Buckheim
0:30
and the team over at Think Medium. And
0:31
that's actually where I met him.
0:34
Currently, he is the VP of partnerships
0:37
at Workm Magic, a firm who is developing
0:40
turnkey solutions for marketing,
0:42
experimentation, and measurement. Alex,
0:45
like me, is from Pacific Island, having
0:47
spent much of his time in New Calonia.
0:50
Alex, I am so pleased that we were able
0:52
to connect today and discuss how to
0:55
assess advertising uplift.
0:59
Excellent to uh to join you and uh talk
1:02
some measurement today. So, I'm excited
1:04
for the chat. Let's do it. Brilliant.
1:06
Now, Alex, uh, I want to get into the
1:09
complexity of the current environment,
1:12
but I think, uh, many of our listeners
1:14
like me have a basic understanding of
1:16
measurement, but don't understand some
1:19
of the current challenges. So, let's
1:21
talk about some of the things that have
1:23
changed. And so, let's start simple. If
1:26
you and I were trying to run a campaign
1:28
15 years ago on the open web when
1:32
cookies was still the primary currency,
1:35
how would you manage campaign
1:37
effectiveness then?
1:39
Yeah, that's a great question. So to
1:41
keep it high level to start, basically
1:43
back then you would use third party
1:45
cookies to track all of your media
1:47
exposures, right? You would tag clicks,
1:49
views, connect them to your conversions
1:52
with I want to say, you know,
1:55
intentionally simpl oversimplifying
1:57
things here. You have you had little
2:00
impediment to doing this, right? And
2:02
then the common approach back then was
2:04
also to give le last touch credit across
2:06
your media channels. So basically, you
2:08
know, whatever drove the conversion was
2:10
the closest to the conversion. And so I
2:13
would say overall it would have been a
2:16
fairly rudimentary world when it comes
2:18
to uh to measurements. There was little
2:20
focus on
2:21
incrementality much less mobile you know
2:24
conversation if any conversation about
2:25
mobile cross device tracking. So it
2:28
would have been a much more simple word
2:29
and I think from an experimentation
2:31
standpoint uh AB testing existed but it
2:34
was much more manual than today. So you
2:37
know back then if you think about
2:38
platforms and publishers and publish
2:40
some publishers maybe supported
2:41
randomized user level control groups um
2:44
but the one thing that's maybe the
2:46
biggest difference with today is that it
2:49
was let's call it easier to tag
2:51
treatment and control groups with a
2:52
tracking pixel and then look at the
2:54
outcomes like at a user level. Uh and
2:56
obviously today it's a it's a much
2:58
different world. So, so let's talk about
3:01
that because I've got in my mind three
3:04
things that have changed significantly
3:05
and you may have others. So, let let me
3:07
propose a list of three things and you
3:09
can react to it. If I compare the
3:12
current environment to the environment
3:14
15 years ago and one of the biggest
3:16
changes obviously being the rise of
3:18
mobile and that meant that as you just
3:20
actually mentioned you have to measure
3:22
cross ID across different devices. Um
3:26
the the second thing in addition to the
3:27
rise of mobile is the increased
3:30
importance of running on closed gardens
3:33
like Meta and YouTube where you
3:36
potentially have less signal and less
3:39
control. And then the the third piece,
3:41
again we just touched on it, is the
3:43
reduction in overall cookie volumes as
3:46
the major web browsers have implemented
3:50
some form of privacy controls. And
3:52
obviously Google has probably
3:53
implemented the least with Chrome. But
3:57
but those would be my three. And you
4:00
know, would you agree with that list?
4:02
Are there other things that you would
4:04
add?
4:05
No, I definitely agree with that list. I
4:08
think you know putting it another way in
4:10
that time span so we look back 15 years
4:12
right fast forward to now it's like
4:15
experimentation solutions vastly
4:17
improved right so if you look at the
4:19
wall gardens that you mentioned they all
4:21
have their own sort of like you know
4:22
experimentation approaches and tools if
4:25
you think about meta and so on um so
4:27
there's there's much more many more
4:30
options to run an experiment in the
4:32
market today but then on the flip side
4:34
one you're probably not going to get
4:36
full access to the data that you would
4:38
need to run these experiments across the
4:41
platforms that might make up your media
4:43
channels. And then the other aspect is
4:45
that even within the platform, so even
4:46
let's let's stick with meta as an
4:48
example. Uh due to signal loss, so both
4:52
on the cookie side, privacy controls,
4:54
but also losing visibility of device IDs
4:57
on iOS, you might not even be able to
5:00
get a clean, you know, uh test and hold
5:03
out group at a user level. meaning
5:06
having reasonable confidence that your
5:08
hold out group doesn't include or is
5:11
clean and doesn't include people that
5:12
shouldn't be in there to keep it clean.
5:14
Um so basically now you know you have
5:19
um you you you have solutions that work
5:22
well at showing you what work best
5:24
within the walls of of your of their
5:26
solutions which is helpful but if you're
5:28
running 10 different media channels you
5:30
need to have another approach that's
5:32
maybe less reliant on signals. the user
5:35
level signals and then you need to have
5:37
a solution that can work potentially
5:38
across channels. And so I would say with
5:40
these three points that you made, this
5:42
is essentially what you have to think
5:44
about now if you really want to have a
5:46
solution to get a sense of like how your
5:48
media channels are performing. Okay. So
5:50
given those changing dynamics, how
5:54
either have marketers responded or how
5:56
are they responding now to try to
5:59
address those challenges?
6:01
Yeah. So you know basically I would say
6:04
marketers have been looking for
6:06
alternative approaches because they
6:07
realize all of these gaps both you know
6:09
like the platform silos and then the
6:11
signal loss aspect and so basically you
6:14
know it's like a game of figuring out
6:17
what's the best compromise between a
6:19
high degree of accuracy if you're trying
6:21
to test for lift and then not relying on
6:25
at least so many user level signals. Uh
6:28
so you know one consequence of that is
6:31
that there's been sort of like a
6:32
resurgence of older marketing techniques
6:35
like media mix modeling is definitely
6:37
one that's come back to the forefront.
6:40
Um and then I want to call out also a
6:43
renewed focus on statistics and science
6:46
to make up for gaps in signals. Put in
6:49
another way if you can't have access to
6:51
all of the deterministic signals to
6:53
figure out exactly where your
6:54
conversions your sales are coming from.
6:56
you want to use a little bit more
6:57
science to run an experiment uh that can
7:00
still get you a fairly accurate re view
7:03
of what's actually driving your
7:05
business. And so to get a little bit
7:07
more concrete here, um one of the most
7:10
robust methodologies that's emerged uh
7:12
in a world of reduced signals is geolift
7:14
testing. And it it essentially aims to
7:16
compare treated versus control
7:18
geographies. So it's at the at the geo
7:20
level. So you don't need the user level
7:22
aspect. And that's how ultimately you
7:25
estimate incremental impact from any
7:27
given channel. Essentially you turn ads
7:30
or you keep ads running in a portion a
7:32
group of regions and then you turn it
7:34
off in an other comparable group of
7:37
regions and then you get a statistically
7:40
backed reading on the incremental impact
7:43
of that channel without having to use
7:45
you know user user level signals. And so
7:48
let's go a little bit deeper on that
7:50
because I mean I know that your company
7:52
work magic's investigating ways of
7:54
running the campaigns in the similar
7:56
geographies and looking at at the
7:57
outlift. Can you go one level deeper and
8:00
talk me through how that actually works?
8:02
Yeah, absolutely. So a few benefits
8:04
before I get there. One, it's privacy
8:06
friendly, right? Two, it's grounded in
8:09
causal inference. So it's basically
8:10
showing real cause and effect. And then
8:13
it's scalable across platforms. For
8:14
instance, you know, if you want to run
8:16
across all of your media channels, you
8:17
can have that same approach, that same
8:19
design replicated across your Meta,
8:21
Snap, Tik Tok, Google, whatever
8:23
platforms and in programmatic other
8:25
channels, CTV that works to answer your
8:27
question. Um, so essentially what you
8:30
want to do is specifically you need to
8:34
have a read uh on your sales channels,
8:36
right? So let's say in our example, I
8:38
I'll take a work magic specific example
8:40
from our client base. We work with a lot
8:43
of D2C direct to consumer e-commerce
8:45
brands. And so these brands, they often
8:48
sells, they have multiple sales channels
8:50
such as, for example, Shopify or Amazon.
8:53
Maybe they're selling in retail. Long
8:55
story short, we want to get access to
8:57
that order data. And that's essentially
9:00
what we we're going to pair with the
9:02
platform side metrics whether that's
9:04
through API integration with a Tik Tok
9:06
whether it's programmatic ingesting all
9:08
of these ad exposures and then we
9:10
connect the dots essentially and so the
9:12
way you go about this about this is that
9:15
we have a team that looks at historical
9:17
order data regardless of platforms
9:19
regardless of channels right and then
9:22
basically able to being able to create
9:25
comparable groups looking at the the
9:27
historical orders
9:28
in different geo regions and then
9:31
running the media like I described
9:33
earlier which for example for a platform
9:35
like Meta would be running your usual
9:39
ads without making any changes in a
9:41
specific group of DMAs and then pausing
9:44
that media in the other group and then
9:46
looking at how many incremental orders
9:48
occurred in the group that was actually
9:50
exposed to ads.
9:53
you you and I met for for coffee a
9:56
couple of weeks ago an example and and I
9:59
want to play it back now to make sure
10:00
that I'm actually understanding how this
10:03
works. So if I'm running a campaign and
10:06
half the spend is going on meta and half
10:08
the spend is going on snap and we run in
10:12
two similar geographies and we keep
10:14
using the term geographies. What are we
10:15
talking here? We talking DMAs or states?
10:18
What's what's the level that we're
10:19
talking about? It has to be ideally at a
10:22
DMA level. So exactly to your point in
10:24
the US which is a US construct right. So
10:26
there's equivalent in other regions. If
10:28
you would want to run in a European
10:29
countries they all have sort of like sub
10:32
levels of geo right or regions but if
10:35
not DMA zip code also work at zip code
10:38
level. So for example, just to give you
10:40
a concrete example, if you're looking at
10:42
your offline sales, retail sales on
10:45
target for example, if the data can be
10:47
seen at a store level, right? Store
10:50
address, that can also
10:51
work. Okay? And so so now to com to
10:54
complete the example, I'm running two
10:56
campaigns. Let's say I'm splitting the
10:58
spend between meta and snap. I'm running
11:01
in two geographies.
11:04
Um my question was around well what
11:06
happens if you're running about the same
11:08
ad spend in both both geographies and
11:11
now having reflected on I think I
11:12
understand the answer which is you try
11:15
not to have even spend running in both
11:17
geographies you want to vary it and
11:20
potentially change it over time is that
11:22
the right concept it's exactly the right
11:25
concept in this example that you gave
11:27
with meta and snap equivalent spend
11:30
level I think what would make sense you
11:32
know as a brand if I put myself in their
11:33
in their shoes is trying to figure out
11:35
which of the two platforms if not the
11:37
two plat the two platforms drive
11:39
incremental returns and then I can make
11:41
you know budget decisions investment
11:43
decisions accordingly and so the way you
11:46
would actually do it is you would look
11:48
at all of your total orders you would
11:50
come up with these groupings that I
11:52
mentioned before and then you would
11:53
basically pick a platform but you don't
11:55
necessarily have to so what I'm really
11:57
saying here is you could run a metadata
11:59
lift test you could run a snap lift test
12:01
but you could also even run a study that
12:03
has three cells and that's testing for
12:06
meta snap and then a control group and
12:08
then essentially what you would do is
12:10
run this lift test for a given period of
12:12
time usually would be three to four
12:14
month uh three to four weeks rather
12:16
sorry and then on the back of that you
12:18
would be able to figure out whether by
12:21
pausing the media in some of these
12:22
regions for let's say you're doing a
12:24
meta test how many incremental orders
12:26
you got and then from there you can make
12:28
corresponding budget decisions which
12:29
might look like okay after running the
12:32
test ideally on the two platforms. So I
12:34
would do maybe meta first, snap 10 then
12:36
or the reverse. Then you have a sense of
12:38
which platforms drive incremental
12:40
returns and to what extent. And then
12:42
you're able to say potentially well I
12:44
need to increase my snap budget because
12:46
it's actually more incremental than I
12:48
thought and then maybe keep meta
12:50
business as usual. That's a metaup
12:51
example but essentially you want to test
12:54
both platforms so that you're able to
12:56
make more informed budget decisions and
12:57
potentially increase the budget or
12:59
decrease the budget on any of the two
13:01
platforms.
13:02
So the example we just talked through, I
13:04
think I get how that works. Um, as you
13:08
mentioned, there's a lot more complexity
13:10
now and to do what you're talking about
13:14
requires that the company doing the
13:16
analysis in this case you guys uh you
13:19
have a lot of data scientists,
13:21
potentially AI, there's a lot of sort of
13:24
um complexity in the analytics that has
13:26
to go on. So I guess one question is um
13:30
what do you hear from the advertiser
13:34
side from the marketer because if we go
13:37
back to the example we started this
13:38
conversation with 15 years ago um you
13:42
know it was open web but you could run
13:44
an AB test and you'd have a sense pretty
13:47
quickly of of how things were shaking
13:49
out. the the model we just talked
13:51
through which I I think I now understand
13:54
uh is a is a multivariant model uh
13:57
there's a little bit of trust me there's
13:59
a little bit of you need to look at time
14:01
series analysis do you run into
14:03
challenges where marketers are
14:06
struggling to interpret the results or
14:10
believe the results h how's this landing
14:12
with marketers
14:14
so I think for geolith specifically
14:16
which is only a portion of what we do by
14:18
the way but to your to your
14:20
There's
14:21
definitely put it this way from a
14:24
service perspective other than the tools
14:26
which you know makes it turnkey you can
14:28
connect your platforms and so on which
14:29
is already a big help and then
14:31
facilitating the implementation of the
14:33
test and the hold out groups data
14:35
science is the other big benefit that
14:37
you would get because to your point of
14:38
where your question is going a lot of
14:40
marketers understand these approaches
14:43
right they understand that there's like
14:44
ways to better get a sense of how their
14:47
media is driving performance and
14:49
incremental performance specifically,
14:51
but most of them don't have the
14:53
resources inhouse, especially if you're
14:55
looking at like the mid-market D2C
14:58
brands, right? They don't necessarily
15:00
have the resources to uh to do all of
15:02
this, right? And specifically the the
15:04
data science analysis aspect, both the
15:06
test implementation, which requires a
15:08
power analysis like the design and then
15:10
also interpreting the results. So that's
15:12
a a big part of what we do. I think some
15:15
of the concerns that are all fair that
15:16
we hear from brands when we discuss test
15:19
implementation I mean the most common
15:21
one would be around having a hold out of
15:23
any kind which is basically along the
15:25
lines of hey you're saying all of this
15:28
this is awesome I do want get to get to
15:30
these results but also I'm running a
15:32
fairly tight program and then I'm
15:34
concerned about pausing ads let's say my
15:36
snap ads in any given group of regions
15:39
now the way we usually tackle this is
15:41
like two-way it's like on, you know,
15:44
it's for the greater good. There's going
15:45
to be maybe a small, you know, minimal
15:47
impact in the short term, but for you to
15:49
get a much better read of what this
15:52
particular channel is actually doing for
15:54
your business. And then the other aspect
15:56
is because we are grouping, you know,
16:00
two equivalent groups or two equivalent
16:03
sort of like regions or groups of DMAs.
16:06
um we have the ability to play a little
16:08
bit with like the size of the howout uh
16:11
to make it sort of like to reduce impact
16:14
and then there's also the fact that
16:16
ultimately if you're looking at total
16:18
incremental orders um it's often kind of
16:21
like a we're affecting only a minimal
16:23
number of incremental orders by pausing
16:25
in some regions. And then the last piece
16:28
that we say in these situations is also
16:30
that since we heavy up the span usually
16:32
uh in other regions right or increase in
16:35
the test cell uh that should make up
16:38
most of the time for the lost orders
16:40
from the hold out cells where the ads
16:41
are not running. So I get I got into the
16:44
weeds a little bit here but that's a
16:45
common objection and to your point about
16:47
like trusting the model making sure
16:49
there's like data science representation
16:51
to walk clients through the methodology
16:53
that's always very important. Alex, this
16:56
is being so useful. I love these types
16:58
of conversations and there there's a
17:00
part of me that's very selfish. I I do
17:02
these because I always feel like I learn
17:05
so much when I have these types of
17:07
conversations. Uh thank you for taking
17:09
the time. Is there any last closing
17:12
thoughts that you have and and what is
17:13
the best way for for people to find you?
17:17
Uh so we have a workmagic uh website as
17:19
you can imagine which is which is at
17:21
workmagic.io.
17:24
We're very happy to have conversations
17:25
with even just marketers that are
17:27
interested in the methodology, you know,
17:29
if not using our product, talking about
17:31
our product, sure, but we love to talk
17:33
measurement, we love to talk data
17:35
science, we love to get into the details
17:37
like we did today. So, we're always
17:39
happy to have a great conversation. You
17:41
can also find me on LinkedIn. Uh, and
17:43
you know, I'm always happy to pick up
17:45
the conversation in the DMs. Uh, and
17:48
then maybe move it live. Brilliant.
17:50
Again, thank you so much for taking the
17:52
time today. Cheers, James. Thank you.