Latest / The Yield Doctor with James Deaker / Nate McCoy Building Ad Tech

Transcript

Nate McCoy Building Ad Tech · The Yield Doctor with James Deaker

0:00
my guest today is Nate McCoy and we are
0:03
going to be talking about how to
0:04
navigate the challenges of software
0:07
development in advertising technology
0:10
nate has been leading engineering teams
0:12
in adtech for over 25 years at Aquantive
0:16
into its acquisition by Microsoft and
0:18
then at AOL into its acquisition by
0:21
Verizon and merger with Yahoo as a
0:24
result Nate and I have worked together
0:26
at many different companies nate it is
0:30
so good to see you thank you for taking
0:31
the time today great to see you too
0:34
James happy to help excellent now I said
0:37
you and I go way back do you remember
0:40
where you and I first met for the first
0:42
time yeah so I I remember an interaction
0:45
at a at a Microsoft All Hands company
0:48
meeting that we used to have at Safe
0:50
Cove Field in Seattle i think they've
0:52
stopped doing them years ago like that
0:54
but uh Yeah and you told me something
0:56
that I hadn't remembered do you remember
0:57
what you told me about that interaction
0:59
what I told you i I do i mean I was so
1:02
distinctive to me because I'd just been
1:04
an acquired employee and you were like
1:06
the old hand you'd been there a year or
1:08
two by this stage and I and I will say
1:10
disclaimer I'm sure Microsoft is a
1:12
different company now this is 15 years
1:15
ago but you were describing to me that
1:18
the seating arrangements made it clear
1:20
where you were in the Microsoft
1:22
hierarchy that the Windows and Office
1:25
developers would always be down the
1:27
front and you said "Look when we were
1:29
first acquired we were right in front of
1:31
the stage and they were pointing at us
1:33
and making a big deal about how great we
1:34
were and then the next year we were
1:37
further back in the first row of seats
1:40
not even on the field." And as you were
1:42
telling this story we started looking
1:44
around for our seats and we were in the
1:47
nosebleleeds and I kept walking further
1:50
and further back and they had wings on
1:53
the stage and I had a seat in the third
1:56
row or third tier and you could not see
1:59
the speaker and given that you just
2:01
described to me that this showed how
2:03
important you were at Microsoft I
2:05
quickly realized where we were in the
2:07
Microsoft hierarchy at the time oh it's
2:09
it was so true and the the display
2:11
advertising world that you know we were
2:13
in largely grew up in Microsoft because
2:15
of of MSN and they had acquired you know
2:18
our company back in 2007 so it was a
2:20
couple years before they acquired your
2:22
company as you said so I had experienced
2:24
the whole you know climbing to the
2:26
nosebleleed section I don't know if that
2:28
was the year but I I'll never forget
2:30
that uh Balmer always used to do this
2:32
thing where he would come out and he
2:34
would run around the floor of the
2:36
stadium to great cheering and you know
2:39
loud music and uh it was it was amusing
2:42
to watch him because he was like you
2:43
know high-fiving and slapping everybody
2:45
running down and he just kept going for
2:47
a long time like way past like the
2:49
cringe moment for me but I'll never
2:51
forget that u there was this year and it
2:53
was at the time where Microsoft was
2:55
really big into the the Windows phone I
2:57
don't know if you remember the Windows
2:58
phone you probably do because they gave
3:00
it all to us they gave us all a version
3:04
there was a newly acquired company I
3:05
don't remember which one it was sitting
3:07
in the front row and as he gets to the
3:08
front row He sees a guy filming him on
3:11
his iPhone and he grabs the iPhone out
3:14
of this guy's hands and throws it on the
3:16
ground and stomps on
3:18
it it was like "Oh man." Well anyway
3:23
corporate shenanigans yes absolutely um
3:27
I want to get into some of the lessons
3:30
learned that because you have so much
3:31
experience but but before we even get
3:33
into that let let's set the stage a
3:35
little bit what's different about adtech
3:39
and and by that I mean either when you
3:41
started 25 years ago what was it like to
3:44
build in that environment compared to
3:46
other environments and if you want to
3:47
get into what's different now we could
3:49
go there as well well let's let's start
3:51
with like a little context so at the
3:54
time and it was late 1999 I followed a
3:56
bunch of friends to go work at a startup
3:59
called then called Avenue A uh and this
4:02
was just before the dot bubble burst so
4:04
we were actually uh we we went public
4:07
and we got lucky because it was just
4:09
months after we went public that the
4:11
bubble burst but we had put $120 million
4:14
in the bank and so we were able to to
4:17
ride that that challenge out we lost
4:20
almost all of our customers because most
4:21
of our customers were you know digital
4:23
and they were um spending money on
4:26
digital advertising but then boom
4:28
they're gone so we split into two
4:30
divisions the digital agency called
4:33
Avenue A uh focusing at that time on
4:36
brickandmortar companies to bring them
4:38
online and our product group which was
4:39
called Atlas and we renamed the parent
4:43
entity Aquantive um and I was introduced
4:46
to this idea that you could pay for a
4:48
name i think we paid like $50,000 in
4:50
consulting for the Aquantum name and so
4:54
and the reason we needed that is that
4:56
Avenue was competing with agencies that
4:59
were all trying to also become digital
5:01
agencies themselves and Atlas was
5:04
selling services to those other agencies
5:07
and so we didn't want to have them have
5:09
to write a check to you know anything
5:11
that had Avenue A anywhere near the
5:14
contract so we were Atlas we were part
5:17
of Aquantive and and uh and and there we
5:20
were there was very little adtech at the
5:23
time doubleclick was out there already
5:25
and they'd been out there for a couple
5:26
of years i think that's one of the
5:27
reasons why our company came into
5:29
existence was to compete with them uh
5:31
there was a publisher ad server from
5:33
Germany uh ironically called
5:36
Adtech and they ended up getting bought
5:39
by um by AOL I think about five years
5:42
after that big publishers like you know
5:45
Yahoo had their own in-house stack which
5:47
might just have been a person who was
5:48
you know changing out the HTML manually
5:50
to put in a new banner ad the Lumiscape
5:53
didn't exist as far as I knew uh for
5:55
adtech let alone like it is today where
5:58
there's like domains of you know
6:00
Lumiscape charts um the so the pinnacle
6:03
of advertising at the time was was
6:05
banner ads running on publisher ad
6:07
servers and every publisher was an
6:09
island uh there wasn't any programmatic
6:11
there you know ad networks were nent um
6:14
Google hadn't been released yet uh so
6:17
more you know specifically like you know
6:19
in in adtech I I hadn't worked in sort
6:22
of large product development groups
6:24
creating something for the internet
6:25
before i had worked solo a lot or you
6:27
know we were work we would work on
6:29
things that would be shipped on CDROMs
6:31
or floppy discs even back in the day so
6:33
this is different it's online it's
6:36
running 247 and we were having to invent
6:39
everything that we had as every company
6:41
at the time was having to invent
6:43
everything that they had to figure out
6:44
how to scale uh reliably so we um went
6:48
through all of these growth pains about
6:50
figuring out a software project like how
6:53
do you plan it how do you code it how do
6:56
you test it because you're not spending
6:58
a year doing that you have to do it
6:59
continuously constantly getting pushed
7:01
you got to release got to release but
7:03
all of us engineers we only knew how to
7:04
do what were called waterfall projects
7:06
that you plan up front and then you work
7:09
on for a long period of time and then
7:10
you release so shifting that mindset
7:13
internally was a lot of what we did we
7:15
didn't use open source and I don't know
7:17
how much of your audience is sort of
7:18
familiar with the open source world but
7:21
it didn't it did exist back then but we
7:24
didn't trust it to run you know our
7:26
hundred million dollar business couple
7:28
hundred million dollar business we we
7:30
even internally were saying like we
7:32
can't really rely on it right um and so
7:35
that was not uh something that we did we
7:37
were very very operationally intense i
7:40
remember talking to one of our founders
7:42
of Aquant a guy named Mike Alggon and in
7:45
a one-on-one and he was pushing me as
7:48
leading the engineering team at the time
7:50
on the operational staff we were
7:53
something like two software engineers
7:55
for every one sort of person running it
7:57
in production two to one the cost of
7:59
that was enormous and it was eating into
8:02
our margin dramatically and he was
8:04
saying like this has to change at that
8:07
time and this is like the early as I
8:09
didn't yet have the vision for how that
8:10
would change but eventually we get there
8:13
when we fast forward so two 2:1 for
8:16
those who aren't sort of familiar with
8:18
usual ratios what would be a a ratio now
8:20
that would be acceptable of sort of
8:22
developers to to operations I think an
8:25
unacceptable ratio now is 10 to1
8:28
But I would say in most of the years
8:30
that I was at Yahoo AOL we were
8:33
operating in roughly 10 to 12 to one um
8:36
I don't accept it i want to I would be
8:37
pushing on that um I think uh there were
8:40
times at Microsoft where we were as
8:43
software engineers basically in some
8:45
sense almost had zero where we really
8:47
pushed all of the operations to the
8:49
developers now we most companies pulled
8:51
back from that there is some
8:53
specialization that is useful but now in
8:56
this day and age if you're operating a
8:58
system that is scaled in production
9:00
you're also a software engineer you're
9:02
writing code to sort of operate it and
9:04
do all the tooling that is necessary for
9:06
that but it's a very much smaller group
9:09
and uh and and then and all the people
9:11
that are writing the software they sort
9:12
of rely on this tooling to exist uh but
9:15
they can change it too so I would say
9:17
you know something you know higher than
9:20
than 10 or 12 to one maybe 20 something
9:23
that could be good 5% of your staff
9:25
focused explicitly
9:28
on production environment talk to me
9:30
about the the fact that when you were at
9:32
a quantive it sounds like you had to be
9:36
more agile even though that may not have
9:38
been the term at the time and not
9:40
focused on these big waterfalls were you
9:42
ahead of the curve or did you have to go
9:44
back to waterfalls so I
9:46
remember 2008 through 2010 Microsoft was
9:50
still doing these massive waterfalls a
9:53
six-monthl long release cycle and even
9:56
when I moved over to to Yahoo uh it
9:59
would have been 5 years later than that
10:02
before Yahoo really embraced agile it
10:04
had been also very much on the sort of
10:07
large water hole as you know there's a
10:09
lot of risks associated with that
10:11
because you've got the whole industry
10:13
could have moved on in that six-month
10:15
window and you can be releasing
10:17
something that is in a sense almost no
10:18
longer relevant so talk to me about that
10:20
transition especially as you came in
10:23
from a company that it sounds like you
10:24
were actually ahead of the curve there
10:26
for a while it in some sense this is the
10:28
story of my career uh and and it would
10:30
it would
10:31
actually shock me if knowing you and how
10:36
your mind works if you and I didn't talk
10:38
about this a lot in the Microsoft days
10:41
because you know you saw what was going
10:42
on and you know we developed you know a
10:45
rapport so um and I had opinions so
10:50
around about 2003
10:52
um like we we created our product Atlas
10:55
right it was great we did this death
10:56
march project we focused we got it out
10:58
the door we call it a death march but
10:59
it's one of the most memorable
11:00
experiences we ever had and then 9/11
11:03
happened just when we were going to
11:04
start to sell it and so we didn't sell
11:06
it to anybody in fact our stock like
11:08
plummeted to like you know 70 you know
11:11
70 cents and uh from like 60 $70 right
11:15
um and we laid off like a lot of our
11:18
engineering staff we were down to
11:19
something like 25 people um but we were
11:22
still sort of operating in this
11:23
waterfall plan you know release and
11:26
deploy over many many months you know
11:29
six months and they would people would
11:31
say well you got to get it down to three
11:32
months instead of six months and we
11:34
would say but the planning phase takes
11:36
two months and the testing phase takes
11:38
two months we can't we can't do it so
11:40
one of the guys that I worked with and I
11:42
I wish it was me he was the visionary I
11:44
was like the second you know it always
11:46
takes that first person who's like you
11:47
know breaking it open but you need that
11:50
second follower to like really like like
11:52
get everyone else to come along and I
11:53
feel like I was kind of like that second
11:55
who was the visionary ken Horn was his
11:57
name and he saw this agile software
12:01
development model of essentially doing
12:03
it in month-long sprints they called it
12:06
and uh which is all this terminology has
12:08
leaked its way into you know a lot of
12:11
different domains now um but I saw it
12:14
and I was like that's our that's the
12:15
solution so he was doing it on one
12:17
little team and I was trying to take it
12:18
to like a much larger team and
12:21
management was resistant actually to
12:23
changing even though they wanted us to
12:25
go faster but we got them to engage and
12:27
said like you know it can't be worse so
12:30
we started doing it we were really by
12:32
the book on agile for a long period of
12:34
time one of the things we quickly
12:37
noticed was that the software testing
12:40
piece of the pro of the problem was was
12:43
taking like past our sprint boundary
12:45
like we would be ready for them to start
12:47
testing at the end of the month they
12:49
didn't have any time to test and it was
12:50
always kind of a crappy
12:53
experience so we just said you know what
12:56
there are models that talk about not
12:58
even having testers that you have
12:59
software developers testing your own
13:01
code and we said why don't we go to that
13:03
so the software testers that could write
13:05
code they became developers the ones
13:07
that couldn't they left the company and
13:10
uh and so we then became just a
13:13
developmentonly shop and we wrote our
13:15
own tests and nothing went wrong
13:17
everything just kind of worked and we
13:19
were like it wasn't any worse for sure
13:21
and it got better we were more quote
13:23
agile by the way just tangent or related
13:25
but I really despise the word agile now
13:27
because it's been used so much that it
13:30
is now burnt for me like I the practices
13:33
of being you know continuous and and uh
13:36
small batch and minimizing risk and all
13:38
of that I'm 100% for but I just don't
13:41
really use the term agile anymore so we
13:43
got to Microsoft and yeah you're right i
13:46
mean they were the company that was most
13:48
well known for shipping on floppy discs
13:50
and CDROM and they just did not have the
13:52
mindset of the
13:54
engineering product development teams at
13:56
Microsoft about a third of the people
13:59
were software developer engineers in the
14:01
discipline of test and that was true
14:04
when we got there 2007 by the time I
14:06
left in 2015 there was no one in the
14:10
discipline of software developer
14:12
engineer and test at Microsoft but it
14:14
took like battle after battle after
14:16
battle after battle to get them there i
14:19
remember the delivery engine for Bing
14:22
had this one terrible release where this
14:26
it was this giant complicated scaled
14:28
system and it took them six months to
14:30
write the code it took them three months
14:33
to just integrate all the various pieces
14:35
that needed to come together for the
14:37
deployment that was nights and weekends
14:39
that was just like intense scrutiny and
14:41
it finally convinced them that we have
14:43
to just do something different in the in
14:45
the Bing and the advertising technology
14:47
team so we were ahead of the curve at
14:48
Microsoft i got pulled in to go talk to
14:51
Bing search team 2,000 people like I got
14:54
in front of uh it made me nervous i got
14:56
up in front of like a whole bunch of
14:58
engineers and said "It's okay you can do
15:00
this you don't have to have testers."
15:02
And eventually I the the company changed
15:05
and and it went its way um I think there
15:08
were there was still Yeah there
15:10
definitely was still testers when I got
15:11
to AOL in 2015 I was able to I had a
15:15
larger role and responsibility i was
15:17
able to help them move away from having
15:18
this tester it's not that testers are
15:20
bad most of them became software
15:22
engineers um on our team it's the it's
15:25
the phase it's the mindset it's like
15:27
this extra communication and you know
15:30
you can write code to test code and it's
15:32
still code and that's fun so anyway yeah
15:36
I've watched this transition to just
15:38
continuous delivery and continuous
15:39
deployment over the last 25 years and
15:41
and why is it a battle cuz you the way
15:45
you describe this is like a war and I
15:49
think it's about odds where I have seen
15:52
good engineers are very pragmatic and if
15:56
they see a better tool they'll jump on
15:59
that tool whether the tool is a software
16:01
tool a better programming language and
16:04
they want to be on the cutting edge so
16:07
given that you had established a model
16:09
that worked presumably at a quantive
16:12
that was more around continuous delivery
16:15
you had something that worked why do you
16:18
feel like it was so many battles to
16:20
break the status quo around waterfall
16:23
development well think of Microsoft in
16:25
2007 i mean under the year the years
16:30
from like 2000 until you know Bomber
16:33
left they they were still writing
16:36
decades of massive success with the
16:39
model of waterfall projects right so why
16:42
would they think any different like we
16:44
we are the biggest company on the planet
16:46
or one of the biggest companies on the
16:48
planet our way works you know we haven't
16:51
really seen demonstrated success they
16:53
hadn't felt the pain of Google eating
16:55
their lunch really really at all of
16:57
these internet companies that even
16:59
though they got quote internet religion
17:00
maybe in the mid 90s uh and when they
17:02
started coming out with the internet
17:03
explorer browser they they just didn't
17:06
really buy it all internally and if you
17:08
have like corporate vice presidents
17:11
managing a team of you know 20,000
17:13
people at Microsoft that are all testers
17:15
actually they didn't have a 20 a single
17:18
test at corporate vice president but you
17:20
might have like a person who has a
17:22
thousand software developer engineers
17:24
and tests working on office you want to
17:26
protect your job you're doing it the
17:28
right way and by the way office was not
17:31
live then it was still shipped and and
17:34
so that is the model in order to safely
17:37
delete deliver that CDROM that somebody
17:39
plugs in and installs office so me
17:42
coming in and just saying "Hey this
17:43
really works for us." They just didn't
17:45
have any reason to believe it or care
17:48
and and and like you said the display
17:50
advertising team had already moved up to
17:52
the third bleacher you know seats they
17:55
they didn't really respect it but when
17:57
problems as an online system Bing B
18:00
being ads our display advertising when
18:02
they ran into like this problem of
18:04
trying to get the delivery engine for
18:05
Bing Ads just already making a billion
18:07
dollars I think at that time to deploy
18:10
after six months with no changes and
18:12
they couldn't they said I guess we do
18:14
need to try something different and so
18:17
people people did but I remember having
18:19
conversations with the head of testing
18:21
for advertising where and he was a
18:23
senior to me um you know he was a peer
18:26
of my boss saying like look you could
18:28
minimize risk if you make tiny little
18:31
changes that you can quickly roll back
18:32
from continuously rather than doing a
18:35
very big change that if you mess it up
18:38
you're going to lose a million dollars
18:39
an hour his argument was exactly the
18:43
opposite the risk is if you continually
18:46
deploy because that's when you're going
18:47
to have a problem you need to really
18:49
test the crap out of it and have it like
18:50
all their ducks in a row and then deploy
18:52
it all at once so just the fact that
18:55
every company on the planet had figured
18:56
out how to do this and Microsoft and
18:58
probably many others um hadn't but they
19:01
were seeing all the success that
19:02
eventually and they were feeling the
19:04
pain of trying to be online they finally
19:07
just you know changed and just said like
19:08
"Look we're going to we're going to
19:10
adopt this." And and then they actually
19:12
made like watching them make the
19:13
transition to eliminate the role I was
19:15
like kind of shocked i didn't know that
19:17
it would happen but they did change and
19:18
they're 100% like you know they're
19:21
cloud-based and all of that now um oh
19:24
that reminds me of things that we didn't
19:25
have back in the beginning we didn't
19:27
have cloud like we just had our own
19:29
machines and we rented space and data
19:31
centers that's that's a huge change as
19:34
well because everything going is if it's
19:36
not already it's going cloud for many
19:39
many companies most adtech
19:42
um let's stay on the components of the
19:45
ad stack and I'm not going to articulate
19:47
all the various components of an ad
19:49
stack but for those who haven't really
19:51
thought it through I want to give a
19:53
sense for how complex it is and you know
19:56
you have all these different things that
19:58
have to work together you have an ad
19:59
delivery engine you have to have creat
20:01
creative review of the advertising
20:03
assets potentially you have to have
20:05
inventory forecasting an analytics layer
20:08
a targeting layer now you would have to
20:10
have in programmatic an auction layer to
20:12
clear all of your inventory um from your
20:15
perspective what is the most difficult
20:19
thing about delivering an advertising
20:21
platform
20:22
difficult thing about delivering an
20:24
advertising platform
20:26
um it is incredibly complex technology
20:29
that's operating in in an ecosystem it
20:32
operates in a much different way than
20:34
say a publisher serveing a a web page
20:37
bunch of content that is you know
20:39
deployed and and images that are
20:40
deployed a request comes in a whole
20:42
bunch of things come out um but they
20:44
they happen at a different scale with
20:46
the with advertising and header bidding
20:48
now and all of those things you know
20:50
we're we're running many many many
20:53
auctions between a supplier of
20:55
advertising like an SSP and a buyer of
20:58
advertising like a DSP for every ad slot
21:01
and it all has to happen in under a
21:04
couple hundred milliseconds two 300
21:06
milliseconds is is max so I mean that is
21:09
just inherently scaled and complex
21:12
running up to you know many many
21:14
hundreds of billions of auctions a day
21:16
at Yahoo and our biggest competitor was
21:19
claiming a trillion auctions although we
21:20
were always kind of skeptical because
21:22
you know we thought we saw all the uh
21:24
inventory on the internet anyway but
21:26
regardless of that very very complicated
21:28
now you're also representing many tens
21:31
of millions uh even billions over the
21:33
whole year of dollars of spend and all
21:36
of that has to be um chosen to be to
21:39
make a bid and prioritized at runtime at
21:42
extremely extremely fast this bidding
21:46
process and the optimization of it and
21:48
the targeting of it that's a PhD
21:50
research scientist level problem that
21:53
you it takes experimentation it takes um
21:56
it takes working with the engineers to
21:59
run those experiments and then interpret
22:01
those experiments and then have many
22:03
experiments running simultaneously so
22:05
every advertising platform is an is a
22:07
research experimentation platform of any
22:09
scale that exists in in in this day and
22:11
age and then even just the complexity of
22:14
of working with research engineers who
22:16
don't necessarily understand um uh
22:19
technology scale like they understand
22:21
the science of it and they understand
22:22
the analytics of it but in order to be
22:24
able to actually code it and run it you
22:26
have to very deep engineers as well who
22:29
can interface with that PhD scientist
22:32
and and write this code but it goes like
22:35
beyond that i mean there's so many
22:36
challenges of serving an advertising
22:38
stack if you uh represent that you have
22:41
um bought and and delivered uh an ad but
22:45
you have discrepancies with a tracking
22:47
system that your customer might be using
22:49
whether you're on the supply side or the
22:50
demand side those investigations into
22:53
why their numbers don't match up with
22:55
your numbers they can sometimes take
22:57
teams of people months at a very large
22:59
organization to figure out the analytics
23:02
to figure out what was changing to
23:03
figure out what's going on i mean
23:06
there's thing after thing after thing
23:08
that makes delivering an advertising
23:10
technology stack very very complicated
23:13
and also for you know the right mindset
23:15
super interesting 100% and as you're
23:18
talking through the discrepancy issue I
23:21
I know how disruptive distracting that
23:24
is to the engineering team because I was
23:26
often the business person caught in
23:28
between the client yelling about the
23:32
discrepancies and and the need for
23:33
actually resolution uh I know you hadn't
23:37
wanted to necessarily get into AI today
23:40
but you've given me the the question
23:42
that I need to follow on
23:44
you reference the large number of data
23:47
scientists and researchers that have
23:50
already been in this industry for
23:53
decades so when people ask me about the
23:57
impact of AI I take the position that
23:59
there are components that are ripe for
24:03
AI to jump in and but when they talk
24:05
about the targeting and segmentation and
24:09
delivery engines I do see incremental
24:12
value from AI but I believe that there's
24:14
been so much machine learning already
24:17
deployed over the past few years
24:20
particularly at the biggest companies at
24:22
the Metas at the Googles
24:24
etc that we won't see real revolution in
24:28
that part of the stack in the near
24:30
future so I' I'd actually love your
24:32
perspective because as you can tell I
24:33
I've taken a position but uh you're
24:36
actually a lot closer to this than I am
24:38
well it's I've been close to it since I
24:40
was a child read reading science fiction
24:42
and walking watching Star Trek i've been
24:44
into the idea of AI uh like forever
24:47
before it even really existed and I've
24:50
I've watched over this time um and it's
24:53
it's grown in fits and starts so prior
24:56
to chat
24:57
GPT you know blowing up the world a
25:00
couple of years ago um artificial
25:03
intelligence really was was kind of
25:05
focused in this area that you mentioned
25:07
machine learning which is really taking
25:10
very large sets of data and using that
25:12
to basically figure out how to predict
25:15
what's going to happen like if this
25:17
thing happens this way and that thing
25:18
happens that way and we have the data
25:20
and we can do the analysis we can create
25:22
a model that we can actually reference
25:25
in a delivery engine to say what is the
25:27
most likely outcome right now and we can
25:29
put a score on it based off of these
25:31
machine learning exercises google and
25:33
Facebook are enormous because their
25:36
machine learning was on point like they
25:39
did a really good job with it um there's
25:41
a lot of other reasons but you know they
25:42
they put a big focus into it and and and
25:45
every stack that I've worked on since
25:47
you know the early as has had a focus on
25:51
machine learning with with researchers
25:52
to get there so before you know the
25:55
current like set of like uh what we call
25:58
AI um that was really the whole world
26:00
you know it might be doing things like
26:02
image recognition there was you know
26:04
beginnings of work with speech
26:05
recognition um and prediction things
26:08
like prediction models not just for
26:10
advertising but people are familiar with
26:12
you know having Netflix give them a
26:13
recommendation of what to watch and that
26:15
was happening that's been happening for
26:16
for years and that is and was what AI
26:20
was called now I think uh what has
26:23
happened to the word AI is kind of like
26:25
to me almost like what has happened to
26:26
the word meme because when meme first
26:29
came out on the internet it really
26:30
referred to a viral concept just some it
26:34
was a conceptual term now meme basically
26:36
means those images with words on them
26:39
people make a meme they're making memes
26:41
all the time it has changed in its
26:43
meaning so the meme of meme is now
26:46
different
26:48
um AI has gone through this transition
26:51
as well because I think if you just poke
26:53
a person on the street you would get if
26:55
you ask them what AI is they would think
26:57
catch GPT right so we can claim victory
27:00
that we've been doing AI for decades but
27:03
in some sense you know that has sailed
27:05
it's going to happen but um but people
27:07
when they say AI I think they're
27:08
thinking about chat GPT and all the
27:10
amazing things that are happening with
27:12
that but a the to to your point about
27:15
like the innovation in machine learning
27:17
that will keep going and is relevant to
27:19
advertising stacks but the current
27:21
version of AI is not relevant to digital
27:24
ads serving yet right although I think
27:27
that there are ways that it's possible
27:29
that you could train a large language
27:31
model on all of the transaction data
27:33
that happens in advertising and maybe
27:35
there's some thing that could be had
27:37
from that that's an area that is
27:38
unexplored but it would be very costly
27:40
to do it could cost you a lot of money
27:42
to do but I don't I don't really right
27:44
now see it happening and we all
27:46
experience working with these things
27:47
like they're all knowledgeable idiots
27:50
like they know everything but they come
27:52
back with things that are just like "Oh
27:54
we got to change that we got to change
27:55
that this is better you're going in the
27:57
right direction you're saving me some
27:58
time but I am spending a lot of time
28:00
guiding you." Right so the
28:02
hallucinations are also there they
28:04
confidently assert something that is
28:06
just not true and so that version of AI
28:11
um not directly applicable to the
28:14
optimization problem of advertising but
28:17
every sort of like thing that surrounds
28:19
it yeah sure like I'm a software
28:21
engineer and I need a test for the code
28:22
that I just wrote to ask the AI to just
28:25
write that test i'll change it a little
28:26
bit but the test will work and if I'm
28:29
writing a document and I you know want
28:31
some feedback or even like an initial
28:33
version and I need to you know create
28:35
like a requirements document my thoughts
28:37
are unordered the AI creates one I say
28:39
"Oh I can change that here's how it
28:41
should be." All of that stuff helps make
28:43
everything surrounding the advertising
28:44
technology go faster um first versions
28:47
of things commentary on what you're
28:48
doing but yeah AI is is not that exact
28:52
it's not directly applicable yet to the
28:55
serving but I do think in the future
28:57
there are some interesting things that
28:58
will happen like for example um and this
29:03
is just me but I kind of got this from a
29:04
science fiction book but it this is well
29:06
before Chat GPT became viable and just
29:10
parathetically I had despared that we
29:12
would ever see something like Chat GPT
29:14
because I've been following AI research
29:16
for years and it just kept making
29:18
incremental progress but sure we can
29:20
recognize cats and photos now but you
29:23
know the idea of something that you
29:25
could talk to that gives an intelligible
29:26
response that's useful i didn't know if
29:28
we would see this for another 10 or 20
29:30
years and so you know the fact that it's
29:32
happened has like opened up my science
29:34
fiction brain again so so I'm imagining
29:37
a future where AI agents are really
29:40
reuseful chat GPT4.5 just came out and
29:44
it has um deep research i am working on
29:47
a software project right now that's kind
29:49
of advanced and I asked it to go figure
29:52
out a problem that I needed to solve and
29:54
you can watch like the text of what it's
29:57
trying to do as it's doing it it was
29:58
going out and looking at websites it was
30:00
saying this looks like it could be
30:01
useful but maybe not it spent five
30:04
minutes on that five whole minutes of
30:07
you know GPU time and the expensive data
30:10
centers and it came back with a document
30:12
that was annotated with with links to
30:14
you know source materials that pretty
30:16
much answered my question and and and I
30:19
I could see is directionally accurate
30:21
and I thought wow that is amazing and I
30:24
might even pay 200 bucks a month for
30:26
that which is what OpenAI is charging
30:28
for it but uh so we'll see i don't know
30:30
how often I have that kind of research
30:32
question but I think in the future this
30:33
will become just like standard it'll be
30:35
part of what we do every day i think in
30:38
the future we'll have AI agents that act
30:40
on our behalf for many things find all
30:43
of the trip options that I want to take
30:44
give those options to me summarize let
30:46
me book that trip go book that trip as
30:49
my agent and it will reliably do it i
30:52
think we'll have an AI agent that
30:53
receives all of the marketing and
30:55
advertising and acts as an in between
30:57
layer i think advertising will be
30:59
advertising to my AI agent i will still
31:01
trade my eyeballs and my attention for
31:03
free content but it's going to let
31:05
through what is more interesting to me
31:07
which also benefits the advertisers in
31:08
the first place but I think there's
31:10
going to be a lot of digital advertising
31:11
that happens to AIs that I never see
31:15
that it will let through when when it's
31:17
useful and I think that is a world that
31:19
will eventually come do uh I actually
31:22
connected with a former wrapped
31:24
colleague yesterday and we as you would
31:26
expect got into AI and he was reminding
31:29
me that back in 2006 2007 we used to
31:34
have a button on some of the the rap
31:36
things that was optimize and it was in
31:39
some sense just conceptual but we got
31:41
feedback that people didn't actually
31:44
want to have a machine optimize their
31:46
campaign they wanted to have the control
31:49
and so we were actually discussing the
31:51
mindset change that has occurred over
31:53
that period of time as we've all got
31:55
increasingly comfortable with the fact
31:57
that we don't have to control every
31:59
aspect of campaign delivery and I think
32:01
we you know that train is going to
32:03
continue it's going to continue yeah hey
32:06
um I I've got to ask you about all the
32:09
time that you spent on
32:12
integrations and uh transition of one
32:15
software version to another particularly
32:18
given all the different companies that
32:20
you worked in i expect you have strong
32:23
positions so I'm going to leave the
32:25
question very open-ended to just
32:28
say what do you have to say about doing
32:31
integrations
32:34
well my strong opinion don't do them
32:38
well that was what my answer was first
32:39
off don't do them that is the number one
32:42
answer when I when when I joined AOL it
32:46
was just before Satcha Nadella became
32:48
CEO at Microsoft and I was basically
32:51
done with Microsoft and it just wasn't
32:53
really the right environment for me i
32:54
think it's probably better now but um
32:56
but at the time I met a bunch of people
32:58
from AOL because we were outsourcing our
33:01
display technology to AOL and they all
33:03
seemed like the people that I used to
33:05
work with in the early days at Aquantive
33:07
you know they had each other's backs
33:08
they weren't sort of like fighting each
33:09
other for territory they were fun to
33:11
have a drink with there was like a lot
33:13
of good things going on so I joined in
33:16
2015 not long after that well right then
33:19
Verizon was buying AOL not long after
33:22
that Verizon bought Yahoo they brought
33:24
us together and I counted 14
33:27
distinct advertising technology stacks
33:30
in play across Yahoo AOL at the time and
33:34
I very much lived out the retiring of
33:38
most of those stacks in my career at
33:40
Yahoo and uh looking back on it I would
33:43
say that there are a couple of things
33:46
first off when I say don't do them I
33:49
mean I think it actually goes back to a
33:50
more fundamental piece if you want to
33:54
have these technologies or these people
33:56
these engineers pick one right don't try
34:00
to merge them together just pick one and
34:01
take all of the engineers and put them
34:03
on the one that you want to go with and
34:05
then just have them change it to make it
34:06
like work the way that you want because
34:08
bringing them together sacrifices so
34:11
much opportunity cost time and and when
34:15
you pick one that you want to like make
34:17
better you're going to lose something
34:20
and you're going to have some revenue
34:21
breakage probably lose some customers
34:23
from whichever system that you're
34:24
shutting down accept it because that
34:26
mean you're going to have that time back
34:28
to now build the features that are going
34:30
to get you the the revenue that you want
34:32
we did it worst of all ways on some of
34:34
the big migrations that we did like we
34:36
migrated uh the you know SSPs that we
34:39
had onto to one stack and moved about a
34:41
you know a billion dollars of business
34:43
from one system to another and we really
34:46
killed ourselves doing it nights and
34:48
weekends i was up on a Saturday night
34:50
asking you know trying to troubleshoot
34:52
issues with the with the migration
34:54
because the the directive from the
34:58
leadership needing to you know they
35:00
needed to solve their own business
35:01
problems i get it but they said "You
35:03
cannot lose any money there can be no
35:05
revenue breakage." And with that the
35:08
project took much longer than it needed
35:10
to we probably cut corners that we you
35:12
know shouldn't have um and and we made a
35:16
Frankenstein's monster that kind of put
35:18
us behind in terms of competing with
35:21
other companies for years and I don't
35:23
think Yahoo has fully ever even
35:24
recovered from all the migrations that
35:26
they've done so with and you know this
35:29
this existed at Microsoft as well just
35:31
integrating into your stack i don't know
35:34
to me it's a fool's game i would say
35:36
pick your thing invest in your thing if
35:39
you have those engineers or or that
35:41
product that comes in if it's a better
35:43
product switch to it and abandon yours
35:44
don't you know it's the sunk cost
35:46
fallacy don't believe you just have to
35:47
keep it because you've already spent
35:48
money on it right determine your future
35:51
and go after it and it actually like
35:54
feeds into like you know like to me what
35:56
the most difficult thing really for
35:59
advertising technology development which
36:01
I think is something you wanted to talk
36:02
about is totally related which is focus
36:06
right my and I don't think this is
36:08
limited to just digital advertising
36:10
technology I just have witnessed it in
36:12
you know the 25 years I spent in it
36:15
right in some sense you know this is
36:17
cynical or I don't know funny or amusing
36:20
list have fewer product managers that
36:23
might be the solution because every like
36:25
senior product manager that you have
36:27
they need to advance their career they
36:29
need to have a win right so if you gota
36:31
you know if they all need to be doing
36:33
something well you're going to try to do
36:34
all those things you're going to say yes
36:35
each of them has their own thing that is
36:37
more important than everyone else's but
36:38
they have just one engineering team that
36:40
they got to come to to solve that their
36:42
problems or you can allocate engineering
36:44
teams to all of them but they're all
36:46
going to be you know time sliced they're
36:48
all going to be working on their own
36:49
small bad you know like minuscule thing
36:52
pick a big thing do it really well no
36:55
one really cares if you have a bunch of
36:57
features if the most important feature
36:59
doesn't work as great as it should if
37:01
it's not easy to use and doesn't make
37:02
you money right then you're failing so
37:05
pick a few small thing or big things
37:08
focus on getting them done and reap the
37:10
rewards of that focus focus focus number
37:13
one problem to delivering an advertising
37:14
platform probably the number one problem
37:16
in engineering you know everywhere is
37:19
being able to focus on what you need to
37:20
get done i I didn't think we'd end up as
37:23
the solution being less product managers
37:26
but I love the cander
37:30
by the way just no shade on product
37:32
managers like I love product managers
37:34
they're great to work with but I just it
37:35
seems like every place that I've worked
37:37
I've just seen like we have so many
37:40
constituencies with so many giant needs
37:42
right like and the greatest successes in
37:44
my career came from when we said we're
37:46
doing one thing even though I didn't
37:48
like doing that migration the
37:50
billion-dollar migration I talked about
37:52
we really focused on doing that and we
37:55
were successful we didn't have revenue
37:57
breakage right you know I got a
37:58
promotion out of it um but you know so
38:01
good for me
38:04
um thank you for sharing the stories and
38:06
as as we have just a little bit of time
38:07
left are there any other uh stories that
38:10
spring to mind and then the amount of
38:13
time you've spent in this industry
38:14
everybody seems to have one you wouldn't
38:17
believe this kind of story have you any
38:19
spring to mind for you i don't think
38:21
mine is so much you wouldn't believe it
38:23
i I feel like I have a lot of stories
38:24
but they they often come down to sort of
38:26
like you know uh like the revenue
38:30
breakage i feel like that's more like a
38:31
negative story like you know management
38:34
made a bad call we should have allowed
38:36
revenue breakage and you know that's not
38:37
that's not so amusing but I do remember
38:40
being really struck by something when I
38:41
first got into the industry uh Avenue A
38:44
at the time they had hired a CEO the
38:46
founders realized that being CEO wasn't
38:48
their skill so they hired one from ABC
38:50
Sports a guy named Brian McAndrews and
38:52
he was great really good operational guy
38:55
he didn't need to be in the limelight he
38:57
was really good at letting people
38:58
excellent people do excellent work he
39:01
just was good on making sure
39:02
accountability was lined up like
39:04
everyone knew who was responsible for
39:06
what and I thought that that that was
39:08
great one of the very first meetings
39:10
that pulled in directors and I was a
39:12
director at the time directors and above
39:14
uh that we had when he joined the
39:15
company about a hundred of us sitting
39:17
around was uh was spent we spent like
39:21
hours on debating whether advertising on
39:24
the internet was digital or online the
39:28
great online versus digital debate and
39:31
it just amused me more than anything
39:33
else is that we were spending all of
39:34
this time as an engineer i was clear
39:37
back in 1982 when I bought a dot matrix
39:40
matrix printer i had to call my
39:42
electrical engineering friend to figure
39:44
out how it worked and he said "Is there
39:46
a button on that that uh says online?" I
39:49
said "Yeah." He said "Well press that
39:50
button." I pressed it and all of a
39:52
sudden it started printing i was like
39:54
"Online i don't know what online means."
39:57
All right but now I knew what it meant
39:59
and then America Online was big in the
40:01
90s right and so we're you know now
40:03
January 2000 what are we we online or
40:06
digital i'm clearly
40:08
digital because online is for old people
40:12
so anyway that was funny to me how
40:15
expensive that meeting was to figure out
40:16
if we were digital or online uh I've
40:19
been in similar meetings that have been
40:22
a lot of naval
40:23
gazing nate this has been so much fun as
40:26
we come to the end of our time have you
40:28
got any closing thoughts and what's the
40:29
best way for people to to find you if
40:31
they're interested in tracking your work
40:35
well uh I gotta update it because I
40:38
don't work at Yahoo anymore i'm
40:40
semi-retired um so uh LinkedIn is
40:43
probably the best i I've boycotted
40:46
Facebook for years because of all the
40:48
death and political morality issues but
40:53
uh I am definitely on LinkedIn and I
40:55
link to things that I do there you know
40:57
check out my YouTube channel it's called
40:58
Nate Piano and I have published some
41:00
classical music there that I that I've
41:02
played a lot of it during the COVID so
41:04
it's been a little while i just recently
41:05
Are you a subscriber to to your uh Yes
41:08
subscribe and I just wrote a song
41:10
actually so I'm going to record it and
41:12
and put it out there um soon uh I've
41:15
started working on a on a project that
41:17
is completely different than anything
41:19
I've ever worked on but it kind of fits
41:21
with my um I don't know science fiction
41:24
fantasy you know visualization digital
41:27
art uh mindset um as a musician as a
41:31
piano player I just I love like digital
41:33
technology and the intersection of audio
41:36
and visual and creativity so I'm
41:39
starting a
41:40
a business that I'm calling Hana which
41:43
is the Japanese word for flower it's
41:45
great because it's four letters and easy
41:47
to say and remember but the idea is that
41:51
almost all of the creative tools that
41:52
exist um whether they're uh video
41:55
editing software or audio editing
41:57
software or CAD tools or 3D animation
42:00
they all require subject matter experts
42:02
and they're difficult to learn you got
42:04
to you know read manuals and you got to
42:06
watch YouTube videos to even touch the
42:08
surface but then I look at something
42:10
like Minecraft and kids are making these
42:12
amazing like architectural wonders and
42:15
even you know adults are making them
42:17
collaborating working in real time and
42:19
everything is is straightforward like
42:21
you could tell what you need to do and
42:22
I've had the thought that what if you
42:24
made creative software that was like a
42:25
video game so all the choices are easy
42:29
to to do uh they generally have good
42:31
results um it's fun you could work on it
42:34
with your friends and I'm thinking that
42:36
it would what I'm envisioning would be
42:37
something that could scale for from
42:39
something into your living room to
42:40
something that you could be running on
42:42
stage with like a concert to something
42:44
that would be you know useful for all
42:45
the visuals at a club or a rave or in a
42:49
lobby in Las Vegas or Dubai or running
42:51
the sphere in Las Vegas but all tools
42:54
that you would make um you know in a
42:57
game-like fun environment so Hana watch
43:00
for that i I wonder how many people are
43:02
going to reach out to you to collaborate
43:03
on that now and that sounds very
43:05
exciting relative to the bowels of
43:08
adtech that you and I have spent time in
43:12
hey Nate thank you so much uh hope you
43:14
have a good rest of your day and really
43:15
appreciate the time absolutely james
43:18
take care