Latest / The Yield Doctor with James Deaker / Digital Advertising needs AI Guardrails

Transcript

Digital Advertising needs AI Guardrails · The Yield Doctor with James Deaker

0:00
The biggest risk in digital advertising
0:03
from AI is probably not the one that
0:06
most people are talking about. It's not
0:08
job replacement. It's not creative
0:11
generation.
0:12
It's what happens when autonomous
0:15
systems start making commercial
0:17
decisions without clear ownership,
0:20
governance, or accountability.
0:23
The following is my talk from
0:25
programmatic AI in Las Vegas where I was
0:28
one of the only speakers to tackle the
0:30
subject headon because frankly in our
0:34
industry people like chasing opportunity
0:37
much more than they like talking about
0:39
how to manage risk. And frankly we
0:42
struggle with everyday business risks,
0:45
pricing risk, operational risk,
0:48
governance risk and incentive risk. But
0:51
AI dramatically changes the stakes. And
0:55
if we don't start building the right
0:58
control layers around these systems
1:00
right now, we're only going to recognize
1:03
the consequences once they become very
1:06
real. And one of the reasons that this
1:08
is always on my mind is due to a class I
1:13
took that was literally a life-changing
1:15
class. When I first came to America, I
1:18
had the opportunity to do a class with
1:20
Ronald Howard at Stanford, who is known
1:23
as the father of decision analysis.
1:27
Ron analyzed everything from whether he
1:30
should ride his motorcycle and incur a
1:34
one ina million chance of death. And I
1:37
used the same framework later to work
1:40
out whether I should run with the bulls
1:41
and pamplona
1:43
to make sure that we all understood and
1:46
internalized some of the concepts that
1:48
he taught in class. He even changed the
1:51
way that we went about taking the tests.
1:54
Everybody knows how to take a
1:55
multi-choice test abd.
1:58
But Ron made us assign probabilities to
2:02
those tests so that if you didn't know
2:05
what the answer was going to be, you
2:07
would assign a 25% chance to A, to B, to
2:11
C, to D, and you would effectively get a
2:14
zero on that uh question. If you were
2:19
100% sure that the answer was A and that
2:23
was in fact the correct answer, you
2:25
would score one on the test. But here's
2:28
the where things got interesting. There
2:30
were real asymmetric downside
2:33
consequences.
2:34
If you assigned a probability of 100% to
2:37
A and it was not A, then you would score
2:42
negative infinity on that question. And
2:45
by implication, you would score negative
2:48
infinity on that test and for the class
2:52
overall. So in other words, really
2:55
messing up one question could mess up
2:58
your whole entire quarter in that class.
3:01
And this was how he demonstrated the
3:04
idea of asymmetric risk.
3:07
All of this has been on my mind because
3:09
as a follow-up to a talk that I gave at
3:11
AD monsters six months ago, one of the
3:14
things I talked about was the idea of
3:17
where is AI going to have an evolution
3:20
in adtech? where is it going to have a
3:24
revolution and where is it too early to
3:27
tell? And the moniker that I used and
3:31
the graphic that I used is showing on
3:32
screen, but it was this idea of is there
3:35
going to be potholes in the car? Cuz
3:37
maybe the cars are self-driving, but if
3:40
there's potholes in the road that we
3:42
don't know about, they're going to
3:44
somebody's going to trip over them or
3:46
the cars are going to drive into them.
3:48
So all of this was on my mind as we
3:51
talked about how AI is going to change
3:55
the risk profile for our industry.
3:59
AI done well should eliminate a lot of
4:03
dayto-day normal risks. And here I'm
4:07
talking about the things where there's a
4:09
typo in an operational line or a pixel
4:13
gets mismatched and so a campaign is not
4:16
running as effectively as it should.
4:19
Those are the kinds of things that with
4:21
the right AI checks in place, we should
4:25
see dramatically reduce. But what I'm
4:28
asserting is that AI significantly
4:32
increases the existential risk. Meaning
4:35
the idea that without the proper
4:38
oversight, a a company could go
4:41
potentially quickly bankrupt and even
4:44
drag down others within the industry. So
4:48
AI is going to reduce the idea of small
4:52
operational risks but potentially
4:54
increase some of the more significant
4:57
ones. So one of the examples of this is
5:02
media math. When Media Math and this is
5:05
by the way not an AI example, but it
5:08
media math filed for bankruptcy, the
5:11
industry faced immediate unresolved
5:15
questions about counterparty risk
5:17
exposure and companies had to quickly
5:20
scramble to work out where they stood.
5:24
But this is what I would call an
5:26
everyday normal business risk. This is
5:29
the kinds of things that mature
5:31
industries have processes around
5:33
counterparty risk and there are
5:35
well-defined procedures.
5:37
In our industry in digital media,
5:40
digital advertising, adte, we don't have
5:43
good procedures around those kinds of
5:45
things. So, we have to learn from other
5:48
industries and other industries are
5:50
already seeing some of the early signs
5:53
of problems with AI. A few examples. One
5:58
is that Air Canada in 2024 had a chatbot
6:02
on its site that was offering a uh
6:07
bereavement to one of its customers who
6:09
was asking about could he fly and it had
6:12
details on the bereavement policy.
6:15
Unfortunately for Air Canada, those
6:17
details were inconsistent with the
6:20
bereavement policy on the rest of its
6:22
website. Air Canada tried to defend
6:25
itself by saying that the correct policy
6:28
was listed on the website, but a judge
6:31
found that a reasonable customer would
6:35
trust the AI just as much as the text on
6:39
some other part of the website. So the
6:41
implication was that the AI output can
6:45
incur contractual liability.
6:48
A second example of AI incurring
6:51
significant downside risk is an
6:54
algorithmic algorithmic example where
6:57
Zillow launched a whole i buying
7:00
division uh associated with buying and
7:03
selling houses. Unfortunately for them,
7:06
there were some assumptions built into
7:08
the algorithm that were completely
7:11
incorrect. It meant that that business
7:13
went upside down very very quickly and
7:16
incurred a lot of losses and that
7:18
division of Zillow ultimately had to get
7:20
shut down. And the lesson there was that
7:24
a model error at scale can actually
7:27
amplify the losses that you can incur
7:29
within your business. And then the third
7:32
example was Microsoft Tay, the chatbot
7:35
that Microsoft launched and then within
7:38
a few hours had to take down because the
7:42
chatbot was learning from the people
7:43
that it was interacting with and
7:46
creating offensive and toxic content.
7:49
And the lesson here is that feedback
7:52
loops can degenerate very quickly in
7:54
some of these systems.
7:57
What's an example that can occur within
7:59
our industry?
8:01
There are a number. One of the examples
8:03
is that in our industry historically,
8:06
there's been a very strong link between
8:08
human decisions where the intent was
8:10
clear and accountability was clear. So
8:14
pricing, targeting, deal approvals. Some
8:17
of these might be machineassisted, but
8:19
at the end of the day, it's been very
8:20
clear that there's a person who's
8:22
responsible for this. As we move into a
8:25
world of AI, we're going to have AI
8:29
agents coordinating with each other
8:31
without potentially a person in the
8:33
loop. They are potentially going to be
8:35
optimizing without understanding and
8:39
they are potentially going to be acting
8:40
without a clear owner. And so let's
8:43
think about some of the ways that the
8:45
agents can be interacting. One is the
8:48
deployment of illegal creatives at
8:51
scale. Right now, most creatives are
8:53
going to go through some kind of human
8:56
review where a person is going to look
8:58
at them. As we move into a world where
9:02
millions of creatives are going to be
9:04
generated by AI, it's not going to be
9:07
feasible for a person to review every
9:10
one of those. If there is offensive or
9:13
illegal content that is generated and
9:16
deployed at scale, that creates the
9:18
opportunity for massive downside risk.
9:21
Another
9:23
is the idea that these AI systems are
9:26
currently not set up to deal with many
9:29
of the regulations, in particular
9:31
privacy settings, GDPR in Europe, and
9:34
other things that exist that we're aware
9:36
of, but the AI systems are not well
9:40
configured for. So, the potential exists
9:43
for a violation of those terms of
9:45
service.
9:47
A third might be the signing of
9:50
contracts with unlimited liability. And
9:53
the fourth is unknowingly targeting of
9:56
protective user classes. Again, these
9:59
are all things that can occur in the
10:01
industry right now, but with the right
10:03
controls in place, we steer away from.
10:06
And if you talk to people behind closed
10:08
doors, we're already seeing the early
10:10
signs of these types of problems
10:12
occurring.
10:14
So let me speak to an example I know all
10:17
too well which is the idea of where you
10:20
need to stand on pricing and pricing
10:22
collusion. Uh it is why I am the yield
10:26
doctor. So, historically, if there were
10:30
companies that were accused of pricing
10:32
collusion, you could usually point to an
10:34
in-person meeting that occurred where
10:37
one company either met with another one,
10:39
exchanged some kind of communication,
10:41
and pricing started to move within the
10:44
industry.
10:45
What many of us are concerned about is
10:49
that in an AI first world, we start to
10:51
see the AI agents start to make pricing
10:56
decisions that look like pricing
10:59
collusion. And so obviously there could
11:01
be a couple of different flavors of this
11:03
and one that I call implicit and the
11:04
other explicit. The implicit one is
11:07
simply that the independent models learn
11:11
to avoid pricing competition. And so
11:15
both on the
11:17
publisher side and the demand side, but
11:19
particularly on the publisher side, they
11:21
may start raising or protecting prices
11:24
in a co what appears to be a coordinated
11:27
manner even though no actual signals may
11:31
be exchanged.
11:33
The more explicit is if the agents start
11:36
to talk to each other without human
11:39
oversight and coordinate to raise the
11:42
prices. Again, that would be an example
11:45
that by today's definitions would
11:48
constitute pricing collusion. So the
11:51
idea that regulators could look at this
11:54
behavior, decide that it looks like
11:56
coordinated price manipulation and say
11:59
that the owners of those systems allowed
12:02
pricing uh coordination behavior to
12:04
occur.
12:07
All right. So what I've talked hopefully
12:09
by this stage in the talk I have scared
12:12
you all silly about some of the downside
12:14
risks that can occur. What are the
12:17
potential controls that we as leaders
12:20
and organizations can start to put in
12:22
place and what I would advocate for is a
12:25
multi-layer in this case three decision
12:28
layers that need to be put in place. One
12:30
is a configuration layer where the
12:33
decisions are worked out as to who owns
12:35
each process. A second layer is around a
12:39
uh the acceptable risk thresholds so
12:42
that we set limits, set constraints and
12:45
work out what would trigger an
12:47
exception. And then the third is
12:49
ensuring that all the processes have
12:51
reversibility and and audit trails. And
12:54
so let's go through those one by one.
12:58
The configuration layer first is a
13:00
pretty simple idea. It's the idea that
13:03
for every process you have, there is an
13:06
owner. And that owner either needs to be
13:08
a person with their name against it or
13:10
maybe it's a functional role within your
13:12
company. But it's the idea that if
13:16
you've got a process, whether it's a
13:17
bidding logic, budget allocation,
13:20
optimization, objectives, it's not
13:23
sufficient to say it's a software
13:26
process, it was just running. You need
13:29
to have someone who understands both how
13:32
it's running and how the inputs and
13:34
outputs work and when things start to
13:37
deviate from accepted acceptable
13:39
parameters.
13:41
The second layer is the idea of
13:44
acceptable risk thresholds.
13:46
Uh you cannot define or you cannot
13:49
deploy AI faster than you can contain
13:52
it. And so you need to work out what the
13:55
failure boundaries are. So you don't
13:58
pilot an AI system and then sit back and
14:02
say, "Oh, things don't look very good. I
14:05
wonder what's going wrong." In advance
14:08
of deploying your system, you establish
14:11
what the thresholds are going to be
14:13
where you would throw a red flag and
14:15
say, "This is outside of our common
14:18
operating procedures. We need to pause
14:21
the AI. We need to reconfigure. We need
14:24
to shut it down." And so this whole
14:27
layer is about defining the exact
14:29
triggers for the automated systems to
14:31
stop and working out when humans need to
14:34
step in. And then the third layer is
14:37
this idea of reversibility and audit
14:40
trails. If a system is in place and it
14:46
goes rogue, you need to be able to work
14:48
out how to stop it and reverse it. And
14:52
one of the things at the programmatic AI
14:55
conference that was talked about is
14:57
common practice now in a lot of agencies
15:00
is as they bring a campaign live,
15:03
they're taking screenshots all along the
15:06
way of here's what the campaign setup
15:09
looked like. Here's the creative live.
15:11
They're proving and demonstrating that
15:13
everything is correctly configured. We
15:16
don't necessarily always do that on the
15:18
publisher side, but this idea that
15:21
either you need uh audit logs sitting
15:24
behind the scenes or as simple as taking
15:26
a series of screenshots to ensure that
15:29
every process is working. You need to
15:32
work out how you're auditing what's
15:34
going on and just as importantly, how to
15:37
reverse it. And the reverse,
15:39
unfortunately, is not going to be as
15:41
easy as a software roll back. For those
15:43
of us who have been around through a lot
15:46
of enterprise software, you know, if
15:48
there's a problem with the current
15:49
release, usually you're going to be able
15:51
to roll back if it's a very major
15:53
change. It's not clear that we are going
15:56
to have the same processes in place to
15:59
be able to roll back to previous
16:01
versions. If not, we need to have a
16:03
process in place for that too.
16:06
So the control layer requires that there
16:09
is a human failure mode expertise and
16:13
one of the things that I think is a
16:15
common misconception in the industry is
16:18
that the control layer is purely
16:20
software. In fact, we need human in the
16:24
loop operators to supervise the software
16:27
and we need to be able to detect when
16:30
the AI confidently outputs something
16:33
that is wrong.
16:35
One of the challenges we have is that a
16:38
number of companies right now are trying
16:41
to cut their way to efficiency and
16:43
saying that it's all about AI
16:46
efficiency. In practice, what we are
16:49
going to see is that the people who can
16:52
catch when the AI is doing things it
16:54
shouldn't do are some of the people that
16:57
are currently being undervalued. We need
17:00
experienced operators and there is a
17:03
real danger that we are cutting the
17:06
ladder out so that we don't have those
17:09
operators around in our companies. Uh
17:12
experienced operators are going to
17:14
become more valuable not less valuable
17:18
in the medium term because we need
17:20
people to oversee these processes. And
17:23
I'm not seeing a lot of companies
17:25
stepping up to put those people in the
17:28
right roles. In fact, it's usually
17:30
looked at that there is just
17:31
efficiencies to be had. Maybe there are
17:34
down the road, but we need to look at
17:37
how do you actually get to a more
17:39
efficient place before you start to
17:42
think about reducing the type of
17:43
oversight and control.
17:46
the idea that you can cut uh your way to
17:50
AI oversight. Again, we you're not going
17:53
to get there through a depleted talent
17:55
pool.
17:56
So, one of the ways that you want to
18:00
deploy your AI software is by allowing
18:03
some experimentation and then thinking
18:06
about your production systems. And so,
18:09
there are two tracks here. what I've
18:11
just been calling pilots, you could call
18:13
the frontier track, where what you're
18:15
trying to do is learn quickly, discover
18:19
what's next, work out the most efficient
18:22
ways of deploying AI within your
18:24
company. This is an experimentation
18:26
path. It is not a production path. You
18:30
want to fast follow with something that
18:32
is safe, reliable, that has controlled
18:34
impact. That's where you deploy what
18:37
you've already experimented with. So the
18:40
key takeaway here is AI should not roll
18:44
out everywhere throughout your
18:45
organization all at once. You need to
18:48
have a phased approach to it and you're
18:51
going to deploy different pieces of your
18:53
AI architecture at different times. The
18:56
frontier track again built for discovery
18:59
and experimentation.
19:01
You're going to have strictly bounded
19:03
pilots. You're going to have tight
19:05
operational guard rails and you're going
19:07
to have humans in the loop to look at
19:09
the output. So the fact that it's
19:11
experimentation and pilot doesn't mean
19:13
you're running with no controls and no
19:15
oversight. It just means that you are
19:17
limiting the impact. You are very
19:20
strategic about what you're doing in
19:22
each of these situations. Once you have
19:25
confidence in that subprocess, you can
19:28
follow up with a fast follow track. And
19:31
so if you look at the graphic, those
19:33
learnings from your pilot get
19:35
incorporated, used in conjunction with
19:38
the right decision control layer that
19:40
talk uses all the things that I've been
19:42
talking about and you start to get to a
19:45
scale deployment for things like
19:47
pricing, negotiation, bidding, etc. As
19:51
we come to the end of this talk, uh it's
19:54
been a little bit depressing thinking
19:55
about all the things that can go wrong.
19:57
And the thing to take away is what can
20:00
you do right now? And there's two main
20:02
boxes here. One is implement the control
20:05
layer and the other is implement two
20:06
different strategies. The control layer
20:09
is about configuration authority
20:12
defining acceptable risk thresholds and
20:15
ensuring that you have reversibility and
20:18
audit trails. Then the strategy to
20:21
implement is around a frontier model
20:25
where you do your pilots and your
20:27
experimentation not in production but
20:30
then you fast follow into production
20:32
ready systems.
20:34
The companies that are going to win in
20:37
the AI era are not going to be the ones
20:40
that deploy AI the fastest. They're
20:43
going to be the ones that deploy at
20:45
machine speed while maintaining control
20:48
of those processes.
20:51
I'm James Deika. I am the yield doctor.
20:54
Thanks for watching. Please join me next
20:56
time.