Latest / Agentic Conversations (formally mlops.community) / MLOps for Ad Platforms // Andrew Yates // MLOps Coffee Sessions #130

Transcript

MLOps for Ad Platforms // Andrew Yates // MLOps Coffee Sessions #130 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Andrea Yates. I'm the CEO and founder of
0:03
promoted IAI. I like my coffee, black with one
0:07
or two ice cubes, so it doesn't sculpt me.
0:11
Abby, welcome back. It's awesome to have you as a
0:14
co-host. I really like this.
0:17
I love talking to you like today's interview is wonderful.
0:22
Yeah yeah so we were talking to Andrea.
0:25
It says you all just heard and he is I love the experience that
0:31
he has. I love how he saw a need for
0:34
this ads platform and he went out and he created it himself.
0:39
He said in the interview which I thought was fascinating at
0:44
Pinterest, there was nothing like what he Wanted when he went
0:48
out looking at the market to see for the Pinterest ads platform.
0:53
And so he said, why not? Just create it myself and he's
0:57
trying to go and serve these very, very Advanced tech
1:02
companies and their ads Marketplace.
1:04
And so we chatted with him about the Spotify is in the Facebook's
1:08
and the Pinterest of the world and the ads platform that they
1:11
have, let me just explain it real quick and layman's terms
1:15
and see if you understood it the same way.
1:17
Way. I think it's basically like when
1:21
you use Facebook or Twitter or Tik-Tok for example, and you
1:27
want to put ads on Facebook, you have the option to go to the
1:34
Facebook ads platform. And there is where you say what
1:38
kind of creative you want? You say how much you're willing
1:40
to spend. He is basically creating that
1:46
platform. Where people can go and they can
1:48
engage with the Facebook's of the world, but as an outsourced
1:53
service and one of the reading fascinating things about their
1:58
product. At least for me, is their focus
2:01
on revenue and which is not something I see.
2:04
And especially when it comes to out, it's such an important
2:07
factor. Hey yes, I'm putting out ads but
2:10
how much revenue will be? Actually making us this
2:12
particular ad. Yeah and that he brought up so
2:16
many good points with Real time. And how you need to be on top of
2:20
it, because you can go from profitable to unprofitable real
2:24
quick with ads and that is you're playing with money and
2:28
sometimes you're playing with a lot of it so you better, make
2:31
sure that it's correct. Yeah, I love this the tiny bit.
2:35
He talked about the streaming deer as well, and the challenges
2:39
that they are in. It take would streaming vs.
2:41
Bosch ticked as well. Yeah, interesting things for me
2:46
in the same kind of conversation.
2:47
A, he has this very clear thought process about how he
2:52
sees things. So, for example, there was this
2:54
one discussion where we asked, what readings are you to go in
2:57
at? And he said, well, all these big
3:00
companies are essentially a business isn't, when I
3:03
personally look at Google, I'm like, a, like a Cespedes, my
3:06
never thought. Oh, my God, but acts are the
3:09
only thing that are actually making money for that business.
3:12
So, you know, they might have hundreds of products, but I
3:15
think of the day, this isn't it. Is so important, so important.
3:20
So let's get into it with Andrew Yates and for anyone that wants
3:25
to check out his company, it is again, it's called promoted AI.
3:29
You can find it online on the internet and for anybody else,
3:33
that is out there and you're listening, you find Value in
3:36
this, it would mean the world to us.
3:37
If you give us a like, or a follow or you, maybe leave a
3:42
review that would be you more incredible.
3:45
We would be so thankful because that Triggers the algorithm and
3:48
lets the world know that this is worth listening to.
3:53
All right, let's jump into it. Dude, I'm super excited to talk
4:02
to you. I really want to hear all about
4:05
the this ad platform what you've been doing.
4:08
I don't know if you know this but we have done some system
4:12
design like deep Dives on ad platform, specifically the Uber
4:17
ads platform and we made like animated video out of this.
4:21
So ad platforms is something that is is close to my heart.
4:24
We haven't done anything with the Facebook or Pinterest ad
4:28
platforms, which I You were closer to than the Uber one, but
4:33
before we get into any of that, we should probably just start
4:36
with your background. I know that you went to college
4:41
for one thing and then you came out, you went to the valley and
4:44
your life spun around. Can you give us a little bit of
4:47
the download on on what happened there?
4:51
Yeah. Well, hi everybody.
4:53
I left school and came to San Francisco and thought hey, at
4:57
Tech is what makes the valley run?
5:00
Let's learn antec, but it's not so easy to get into ad tech
5:05
there at that time that was like, 2012 or 13 or so.
5:11
A lot of people are starting at Tech startups but they weren't
5:16
successful in comparison to Facebook or Google.
5:19
So I went to go work at Facebook ads.
5:21
The industry join Pinterest, learn industry there as well,
5:25
and then started promoted all the way to Winter 21, ask you
5:29
something here. So you kinda interesting that I
5:33
could work with the one very specific thing because a
5:36
particular conversation but somebody some sort of article
5:39
that sort of said hey let's get into a semantic.
5:42
This is an up-and-coming space. Oh well it was already the
5:47
space. I think people try not to look
5:50
directly at the Sun One and want to pretend other Industries
5:54
compare. But Google is an ads business,
5:57
Facebook isn't ads business, Amazon has a huge ads business,
6:02
Microsoft has a huge as business.
6:04
Apple has a huge as business. Those are the biggest most
6:07
important technology companies. Their businesses are somehow
6:11
commercial media and it touches on every other business.
6:14
It's challenging to think about in a couple of ways.
6:17
One is because there doesn't seem to be much opportunity for
6:20
promising start. In this space, and that was
6:23
actually, one of the things that inspired us to get started.
6:26
We're seemed like there had been such a long number of years
6:30
where people were not ambitious in edtech, because you could go
6:35
to Facebook or Google and they were so dominant for so long
6:39
that there was an opening. And in fact, I got that feedback
6:42
at Pinterest. When when I was having lunch
6:44
with someone from Business Development, she says, hey,
6:47
there's no good startups and adtech.
6:50
And and she's right, I mean, according to Pinterest standards
6:53
there be nothing there, that Pinterest would have any
6:56
interest in in requirement regarding using their product
6:58
or, or any kind of acquisition or that Engineers working on.
7:02
It wouldn't have been any of any interest to a company like
7:04
Pinterest. And in fact, I think Uber would
7:06
have found the same thing. I know that they Outsource their
7:09
ads for some time. This is something that we
7:13
explicitly go after and it's not so much on the technology side.
7:16
It's more of like honestly building your ad.
7:19
But not goofing around with the measurement.
7:22
Attribution model to pull, forward revenues to try and make
7:26
your numbers every quarter or keep repeating that process over
7:28
and over. There's a technology to it as
7:30
well. But there's also like an
7:31
organizational practice about something about ads, makes it
7:35
misses, getting back to load it to the motivation.
7:38
And and I think the opportunity, there's a tremendous amount of
7:41
technology and thinking behind ads.
7:44
There's a phrase, about top minds of a generation have been
7:48
wasted on Getting people to click on ads, people really hate
7:53
to talk about ads. They really don't want to
7:55
identify. So, rad tech people.
7:57
There's a famous quote from Twitter about ads or Like
8:00
Oxygen, you have to have them, but it's not something you, you
8:04
really celebrate or want to work on, we come in from come at it
8:08
from the opposite side and especially this idea of an
8:11
honest ad system which is own it like you're here to make profits
8:17
for everybody. And designed the system in such
8:21
a way that you make it as Fair as possible.
8:24
And there's a lot of technical logical implications to how you
8:26
would build such a system, and we'll get into that, especially
8:28
from the ml offside. But a lot of it comes down to
8:31
measurement and philosophy in the organization as opposed to
8:36
technology that the idea of predicting clicks is not new
8:41
people done that before. But the idea of having your
8:44
entire organization maximize for one goal versus always Having
8:49
this extra magical lever, the Poland, a little bit extra
8:52
Revenue, at the end of the quarter or hey, the core
8:54
business isn't working. Let's just goof around with a
8:57
couple of things and try to get those budgets in and later it'll
9:01
get fixed. Do not do that and do it.
9:04
The right way is, is incredibly powerful.
9:06
So that's what we're solving your a promoted as in honest ad
9:10
system, that is so great that we can run all commercial media,
9:13
not just adds so maybe you can take us.
9:19
Through the evolution of what has happened with adtech,
9:24
because I imagine 10 years ago, that was where people were
9:28
tinkering with machine learning and they were trying to get some
9:31
machine learning into ad Tech. And now it's basically a, it's
9:36
like table Stakes. Yes, there was an interesting
9:40
book called chaos monkey, so that Antonio, Jesus last name is
9:45
escaping me at the moment, but he wrote an interesting book
9:48
about Out ads at Facebook back in 2010 or so.
9:53
And he had this philosophy around Facebook ads would
9:57
totally fail because they didn't participate in this real-time
10:01
bidding ecosystems open ecosystem.
10:03
And instead they built this closed ecosystem, and you would
10:06
have to send all of your data to Facebook and only Facebook would
10:09
be able to use its own data. Most people expected Facebook,
10:13
did not be a trillion dollar company.
10:15
They had a lot of struggles. I worked on the That came later.
10:19
So I worked on the team that worked on the machine learning
10:22
application of all of that data and turned out that back around
10:27
2010 or so, both Facebook and Google.
10:30
In particular, we're able to come up with techniques to start
10:34
monetizing the data that they had directly through machine
10:38
learning and click and conversion prediction models.
10:40
And it worked really, really well.
10:42
Exceeded, everyone's expectations, one of the first
10:44
examples of it was the mobile app install ads on.
10:49
Facebook, which was just like this breakout market and then
10:51
this idea of monetizing mobile ads was just it before.
10:55
It was a banner ads on web and mobile ads didn't work at all.
10:59
So there's this interesting Dynamic where it's not like just
11:03
only maximizing like if you have a recommendation system making
11:07
it a little bit better, get you a little bit better.
11:09
But for ads there's this threshold of profitability where
11:14
if it's unprofitable, you should do it, never.
11:17
But as soon as it crosses over, To enough efficiency where stars
11:20
become unit profitable, you should buy infinite under of
11:24
these and that's what was starting to happen in 2010 or so
11:27
is using machine learning successful applications of
11:30
machine learning ads were becoming efficient enough.
11:33
That this ecosystem was built up to build.
11:37
A Facebook, for example, is was the biggest success story of
11:40
this. Going a little bit away from
11:45
let's say, Facebook and thinking a little bit more towards the
11:48
bigger Market. Places, where the big challenges
11:51
they're facing today? I know there are laws.
11:53
Okay for user privacy, like gdpr in Europe and there's a
11:58
California protection act, I believe in California.
12:01
Well, first of all, if it's on your own website, those are not
12:04
relevant to you. And second that's relevant for
12:08
cross-site tracking. So, if you're trying to
12:10
aggregate data across, Us many different sites, then it's a
12:15
challenge. If your Google somehow they have
12:18
come up with the you just Google has been affected for sure.
12:21
For sure, certainly. Facebook is in fact, affected
12:24
more. However, they have come up with
12:27
strategies to continue figuring out what you're doing around the
12:29
web everyone because that's her core expertise in there.
12:31
A trillion dollar company, there are other companies that do this
12:35
less successfully. So if you are a sa unicorn
12:40
Marketplace, you are. Not going to be able to come up
12:43
with the entire Playbook of Google, to figure out how to
12:45
track people around the web, and on different mobile devices.
12:49
And so, you're going to have to have a different Playbook.
12:51
But bringing this back down to earth regarding ads on Market
12:55
places, it shouldn't be that relevant because the
13:00
transactions are happening on your own Marketplace.
13:03
Why do you need to go and track people around the web?
13:07
You can track them on your own app that has never changed.
13:10
That's the same, you know what they're doing?
13:11
And Yeah. So as far as when it comes to
13:18
your time at Facebook and Pinterest, one thing that I'm
13:22
very curious about is building out these platforms.
13:26
You spoke a little bit about the vision there, which is a whole
13:29
nother Beast that I want to get into in a minute.
13:31
But maybe just like for people who have never worked at either
13:35
company, can you paint the picture of what it looks like to
13:41
work on? A data team in each one of these
13:44
companies like what's the structure of the team?
13:48
How does the, how do the goals? What you're working with?
13:50
Okay ours I imagined. But what are the things that
13:53
you're looking at like in the detail to be able to just like
13:57
show us what are the differences there?
13:59
Yeah there's not a single data team at these bigger companies.
14:04
One, one thing that I think is there are two aspects of how I
14:08
think it can be done. Better versus how it's done in
14:10
practice one, Batch versus streaming and the second is
14:15
Unified across your whole app versus ads having their own
14:19
system. I'll start with the letter, it's
14:20
a little easier most companies when they do ads as there's a
14:25
there's a very clear cultural break.
14:27
Our Core Business is peeked out, we need to make more profits and
14:32
the existing search and Discovery team will go build a
14:35
team, both in social media and in marketplaces it, maybe even
14:40
food delivery apps ride. Perhaps, they will do their own
14:43
measurement, they will do their own inventing, they will do
14:46
their own machine learning, they will do their own delivery
14:49
system. And then, there's some sort of
14:50
orchestrating system that combines at results with Organic
14:54
results at the end. And then someone says, well, if
14:57
only you got us this cost-per-click, or this positive
15:01
return on that investment, we give you this huge budget but
15:04
we're only spending the small budget and then they have like
15:07
an team that's coming outside. Well, we have all these ad
15:11
Impressions, but when we originally started, our atom
15:17
attribution model says that, they're not generating value.
15:20
But we look at all these ad Impressions were serving the
15:22
data side of it is it's predicting clicks, and
15:26
conversions, and tracking Impressions and insertions and
15:28
things. These are the sort of things
15:30
that you would expect to be table Stakes for everything on
15:34
your app. Why is it so substantially
15:36
different for clicks and conversions?
15:39
And for Impressions? Why is it for an ad?
15:43
It's a Wholly different thing than when it's your ordinary
15:47
organic product. Like ordinary things that are
15:48
showing up and say who bereet sore or your Marketplace or
15:52
Facebook news feed or Pinterest or for whatever.
15:54
Nevertheless they are usually constructed as entirely two
15:57
separate systems. When if you think of it as one
16:00
thing you can get the benefits of both on the ad side, the ad
16:03
side has a strict definition of visibility around.
16:08
An impression has a uniform definition that you're not
16:11
supposed to mess around with and I Maybe impression, like you
16:15
have to have seen it for a certain amount of time with a
16:16
certain number of pixels. And then for a click, you're
16:19
supposed to Define to your advertisers.
16:21
This is this is what a click means and then stick to that
16:25
definition and not, there's another sort of measurements you
16:28
think not, not change the definition of clicks to make
16:30
them more common, so that you can build more of your
16:32
advertisers you're supposed to stick to it.
16:34
A common definition, you can benefit tremendously from this
16:37
sort of rigor of measurement, definitions because the better
16:42
measurement allows you to train more.
16:44
Models on the other side of it, the ads.
16:48
Well, you have one Marketplace. Could you use the data of all of
16:53
what people are doing across your entire app to better
16:56
optimize your ads? Well, well, yes, you could.
16:59
But you have to cooperate with your search and Discovery team
17:03
to be able to have the data pipeline in place and and or
17:06
things like. And I think you've probably
17:08
experienced this before you see an ad and write a book.
17:14
Right below. It is the organic listing and
17:16
you're thinking, wow, like could these two systems talk to each
17:19
other and they could right there are that's like from a pricing
17:24
aspect is like would you Bill would you pay the same amount
17:27
the win first? If you are already going to win
17:30
first if you didn't pay for an ad and the answer is almost
17:33
always know you pay less or nothing at all, you can set
17:38
these sort of systems up from the measurement side, but you
17:40
need the unification between the ad side and and there.
17:44
Next slide. So, that's the the part around.
17:47
There's a lot to be learned by combining together.
17:50
What's Best in edtech with what's best in the organic
17:53
overall product into an overall measurement solution.
17:56
There's also this idea of batch versus streaming, I'll talk a
18:00
little bit about that most systems Sterling match
18:04
especially if they were built five years ago because streaming
18:07
it's really hard. It's just it's just a whole
18:09
separate other discipline for batch you can throw everything
18:13
The Hive and you can have a different.
18:16
You can have data scientist frequently, write SQL queries
18:20
and generate what or models in a notebook and everyday generated,
18:26
which works pretty well to be clear.
18:28
And when I worked at Facebook, by the way, we were doing that
18:31
was back in twenty thirteen or fourteen or something like that.
18:34
Like daily batch is okay. Go get it.
18:38
Go get some of the job done. The biggest difference in ads is
18:41
your spending other people's money in real time?
18:44
So of model, delay of like two or three days what's been
18:48
happening for those two or three days like have you been spending
18:51
someone's money and way that they would like to change very
18:54
quickly when I said that Facebook was doing Daily Bat
18:57
training they were but they also have these other real-time
19:00
systems to make sure that if something went awry that you
19:04
could respond to it immediately. These were calibration models of
19:07
these were also Pacer models regarding hey I have a budget.
19:12
I need to try and paste this. Only throughout throughout the
19:14
day. So in ads, use case the
19:17
streaming case becomes much more important because you can't have
19:23
your model be wrong, in an individual case for too long and
19:29
you have to intervene. And so the real-time streaming
19:31
becomes much more serious on the outside and it again excite can
19:34
benefit from it too. And not to mention the Price
19:39
just that it takes to stand this up and to get streaming up and
19:45
running from a resources perspective, right?
19:47
From all the engineers that need to know about this and right
19:51
now, I think the streaming ecosystem is so complicated and
19:55
it's just so confusing. Okay.
20:00
How are searching Discovery piece but the bigger tech
20:04
companies. Oh there are many different
20:07
teams doing search and Discovery frequently.
20:09
They'll have an R&D team they'll have like a practice team that
20:12
will Implement what the research team has done two different
20:16
models and then frequently. They separate the organic team
20:20
from the as team that typically there will be searched and
20:23
discovered. They won't call it search and
20:25
Discovery for ads. They'll call it as ranking and
20:27
ads targeting maybe it's easier to talk.
20:29
Worried about a smaller company, let's say, like a doordash door,
20:33
or like an Airbnb. So, frequently they'll have
20:36
platforms team that are building tools will be a data team.
20:40
That's providing all of them Eventing and providing the raw
20:45
material for producing both creatures and training examples.
20:49
They'll be usually in ml infrastructure team for training
20:51
models, hosting the models, there's frequently a research
20:54
and development team. That's coming tomorrow
20:56
architectures. Then there's frequently a Data
21:00
science team, that is analyzing models and frequently provides
21:05
an interface to maybe more business-oriented product
21:08
management, type of teams, then there are usually one or several
21:13
types of call them, like, middleware kind of engineering
21:17
teams that are taking the raw resources for the machine
21:22
learning systems, and then applying them to specific
21:26
application. And those would be typically
21:28
named things like, And for such and such surface, were certian
21:33
such-and-such search and retrieval or maybe like such and
21:36
such product. Those teams would be maybe
21:39
working in python or Java and applying models or training
21:44
models using the infrastructure provided by other teams and
21:47
using the data provided by other teams in your experience.
21:52
How was it like, would you all have like, a week or a month to
21:58
try and fix? Technical debt.
22:00
And how would you go about that? And then I'm just thinking about
22:05
all these teams potentially playing together and interwoven
22:09
and the one thing that's common with all of them is the data.
22:13
I imagine we're what was the strategy around it?
22:18
Oh, interesting. I think you're getting at one is
22:22
who who owns signals and then who can apply them.
22:27
So, once you're consuming a signal, what if that signal
22:31
changes and who owns that life cycle?
22:33
So frequently there needs to be versioning.
22:36
I've seen attempts to manage this.
22:40
It's hard. I think, I think one part of it
22:42
is it just recognizing that at that?
22:44
It is harder than that. It's complicated.
22:46
And it frequently requires a little bit more relaxation on
22:52
always measuring every single, a be impact and more thinking of
22:57
things as as services that you consume as a black box, which is
23:02
very difficult for teams to do, sometimes if they're not quite
23:06
meeting kpis, and someone was to change the signal, but that's
23:10
going to mean, is going to be regressed for a few weeks or
23:13
even months while they come up with the next version?
23:16
Like, Maybe changing model architecture.
23:18
That can be very challenging to manage internally.
23:21
And so, I've seen people try to build feature stores, and
23:25
machine learning platforms to try and make this all automated
23:29
and run by the machine with varying levels of success.
23:33
My personal opinion on this is it takes strong technical
23:36
leadership from the top and manage these kind of horse
23:39
trading issues and I'm not just allow internal teams do forever
23:44
argue with each other about horses.
23:46
Trading in a be experimentation and launch reviews and keep a
23:50
more unified vision. Of this is the direction.
23:52
We ultimately want to go so long as this is using more data.
23:57
There's some reason mechanistic reason for this to be useful
24:01
that even if the metrics are ambiguous or mixed that they
24:06
have the confidence to continue to move forward versus forcing
24:10
everything to go through. Such a super clean, a be
24:13
experimentation and launch review for every service that
24:15
everyone. Depends on that.
24:17
Could be a tremendously slow and frequently gets corrupted
24:21
because eventually someone needs to get something launched and
24:23
they'll just bypass that progress.
24:25
And there's then once you have that happening it gets pretty
24:29
ugly but I think it takes a certain level of very strong
24:33
technical leadership from the top to help guide people to take
24:39
some risks and okay, is it okay if this is launched, even if
24:42
it's not quite sure. Well, we have faith that this
24:46
This is the right sort of architecture that we want and
24:48
not Thai people so desperately to kpis and a be
24:53
experimentation. That's my personal opinion.
24:56
I don't think this can be purely solved from a technical and
24:59
infrastructure standpoint. In terms of feature stores and
25:02
Signal Management, and versioning that said there are
25:05
some best practices, like when you launch a new version of your
25:08
signal, you should make a new version of it and then people
25:11
adapt into the new version and then you have some sort of
25:13
system for. Hey, we're going to deprecate
25:15
this other After six months or something like this, and how the
25:19
resource and budget to be able to have parallel versions of
25:22
your system, so that people can migrate over, like there's some,
25:25
I'd say table Stakes for how team should work together when
25:30
they're consuming other signals and working together.
25:33
But that said, I don't want there to be any belief from me
25:38
that this can be solved by technology.
25:41
It is an organization leadership challenge that can be
25:46
facilitated. Facilitated by technology.
25:48
I and a lot of people that come on here and most of the
25:51
co-hosts, we all know that. It's not always about the tech
25:56
and we try to keep it as much as possible about this
26:00
organizational piece to because everyone who knows about devops
26:04
that is very much a organizational piece as much as
26:08
it is a tech piece. And there is one thing that I
26:14
was thinking about when it comes to Promoted the tool that you've
26:18
built is were you looking to build something that was going
26:23
to plug into one of these bigger companies?
26:26
Or was it something for for someone who's going to zero to
26:31
one and they can just have it right off the bat before they
26:35
need to clean everything up former?
26:40
It's for big marketplaces is for social media, it's for this
26:43
could run Pinterest. Dan and I were interest
26:47
Engineers. We're trying to solve the things
26:50
that we wish interest had done the problems that were solving
26:54
our. What, if you are a scaled
26:57
Marketplace and the difference in, let's say, let's say you can
27:01
make five percent more Revenue per user.
27:04
That's the difference between profitable and unprofitable at
27:08
scale. It's a difference between you
27:11
should continue to be in business versus eventually you
27:14
will go out of business. It A tremendous step change and
27:19
it's like every day you get better every day you worse and
27:22
these kind of efficiencies 510 percent is their huge at scale.
27:26
When you are a multi-billion dollar market place, the
27:29
efficiency matters tremendously and that's what we're going
27:33
after knowing what, you know about the organization and all
27:38
of these different teams that you mapped out, just a minute
27:41
ago. How do you fit into that like,
27:45
dude? Just plug into all the different
27:47
teams. Do you say, we're going to
27:50
specifically talk to the ml platform team?
27:53
For example, what does that look like?
27:55
Yes, we is kind of a naive idea, but it turned out to work
28:01
accidentally. We decided to build our own data
28:05
and measurement infrastructure. Because how hard could it be?
28:09
Well, Dan Hill, my co-founder is just a genius at building
28:14
infrastructure and Set up our own data infrastructure.
28:17
So what we do is we provide our own streaming data
28:21
infrastructure and we provide it at no additional cost and we
28:24
measure everything in your Marketplace, everything in your
28:28
app, we do all of our own Eventing.
28:30
We do all of our own measurement join but we can plug into other
28:34
sources, like from segment or amplitude or whatnot.
28:36
But when you are generating, the kike types of volume and quality
28:40
of data that are necessary for doing very high level types of
28:43
machine learning, you're talking terabytes of data, And this is
28:46
something we've also seen is people think they're logging a
28:49
lot of data and then they need to start logging annotated
28:52
Impressions. Like, you know, mobile device
28:56
you trying to send like I don't know, 20 attributes per
28:59
impression and someone just Scrolls or if you can go and you
29:02
only have like 20 Impressions. Oh no that's not going to work.
29:06
So we build all of this infrastructure for measurement
29:10
and joining and we provided at no cost and that gives us a
29:14
extremely found. Foundation to build amazing,
29:18
machine learning models that have a picture of, here's
29:20
everything that you're doing. And then regarding working with
29:22
other teams or other machine learning teams, the style of
29:26
machine learning that we have. You can stack models, you can
29:29
send to us. If it's a, if it's a user model
29:31
you can just send it in our user CMS system.
29:34
If it's an item model you can like sent you can stick a
29:36
clustering or whatever it is collaborative filtering or
29:39
embedding to upload it to our item CMS and if it's a item
29:44
query or user across It's like a relevance item relevance or user
29:48
relevance model you can attach it on the insertions that you
29:51
send to us. And their way, our system is
29:53
built, is we're super wasteful in some sense on data
29:57
infrastructure, but we've made it very efficient and we can
29:59
model on top of all the other models that you currently have.
30:02
And we can also report back to you every day.
30:05
Here's where your models missed out.
30:07
Like, here are the additional features and information that we
30:09
had that were able to get you more performance and we used
30:13
your other models as signals and can combine them together.
30:15
And then From a machine learning perspective, it.
30:17
So if you can model on top of another model, the other
30:20
features that are important are explaining the residual error of
30:24
off of what these models were. So we can interface with your
30:27
existing in-house data science team where we can ingest their
30:30
models, and then build on top of it versus entirely replacing it.
30:34
And same thing for other data signals, you can just send it
30:36
all to promoted. And at the end of the day, our
30:39
data infrastructure is this enormous.
30:42
D normalizer Joyner, in real time, collecting all of these
30:45
signals. And then, Training the models in
30:47
real time to be able to find. Here's what the signals are.
30:51
Interesting that brings me to the next question.
30:55
You're still a very small start-up I believe eight people.
30:58
I'm not entirely sure if the team size has changed recently
31:02
but where are the big engineering challenges that you
31:05
faced with such a small team? That's a great question,
31:08
especially because you're catering to a very big space and
31:11
you're working with companies that have important their
31:15
challenges as well. It's not like let's just test
31:17
it. You have to sort of prove it
31:19
which is it's very Roi based business, the proving part.
31:25
So here are some challenges. One is we had to strip out
31:27
tremendous number of features. We focused exclusively on
31:33
proving that our models, produce Revenue at the expense of
31:37
virtually every other feature. And by the way, one of the
31:41
things that got that D.D prioritized was metrics and a
31:46
Maybe debt testing you'll tear from a variety of customers that
31:52
will say, oh, hey, or of a variety of vendors and say, hey,
31:55
we'll do an a/b test animal will outperform the such-and-such,
31:58
vendor your existing system at the level that we sell.
32:01
Nobody will take our metrics, they will run their own, a be
32:05
experiment. So although we have we said
32:08
fully support a be experimentation, we have a
32:11
dashboard, we have metrics and whatnot, what we found was No
32:16
matter what you produce, if you, if their team has to be able to
32:20
measure that a be experiment, they have to prove to themselves
32:24
that you are outperforming their other models or any other
32:27
vendors. Oh, we've been benchmarking.
32:28
And don't go Lea. We've been benchmarked against
32:31
variety different versions of combinations of In-House models
32:34
and a to be personalized and whatnot.
32:37
So one of our challenges is you're fighting against reality.
32:41
It's just hard to build a, such a complicated system that Can
32:46
reliably produce an increase in revenue and keeping super
32:51
focused on that one. It allows us to improve the
32:56
revenue of our customers over very strong, bass lines, and
33:00
also, according to their in-house team who has a mandate
33:03
to show that one we don't work and therefore, they get our
33:07
budget to go hire people, right? And and so they are very
33:11
motivated to has to do really, really work, and we have to
33:13
explain how it works. So that, Trust us, and I feel so
33:18
great when we win the trust of our customers and not just like
33:22
the business team, but the individual data science.
33:26
And and machine learning experts that were working with and the
33:31
challenge of this is that it's hard to build such a system.
33:34
It's just a lot of work. The other challenge is we have
33:37
to strip out a tremendous number of features.
33:40
Another challenges which we were.
33:41
We're going back now. Another challenge is we can only
33:44
work with a few customers because it's challenging to
33:49
provide that level of service for a few Enterprise customers.
33:52
And so, for example, we can't provide today, we can't afford
33:57
to provide a freemium version of our product.
34:00
We won't do free trials because we don't have the staff to deal
34:04
with it. We don't have a Solutions
34:05
engineering team today to be able to do with that.
34:09
We will wonder well how it will be going.
34:12
Going any free trials but today we have to pay up front.
34:18
Yeah. Like how much white gloving
34:20
needs to actually happen because of all this complexity well in
34:24
theory not in theory. Here's all the open source sdks
34:28
install it. The way that we ask and it just
34:31
runs in theory in practice. These are large engineering
34:35
teams that are prioritizing for a variety of different reasons.
34:39
And And if something is not clear or if something comes up,
34:43
you always kind of integration with some hypothetical other
34:46
platform, that might be better. So, in practice having some
34:49
service level to help get integrated, it's not because
34:52
people can't do it or they don't want to.
34:56
It's more like from a practical standpoint.
34:59
That's how real engineering organizations are actually run.
35:02
Yeah, makes sense. The other side of it is that
35:06
helps us build a fantastic product like we have to eat our
35:08
own dog food. It better be amazing.
35:11
You, we have a much tighter feedback loop if we're there in
35:14
the trenches helping making sure everything works.
35:17
And it turns out, there's all sorts of, oh, you know, the
35:20
mobile devices this way. But actually, the web is this
35:23
other way, they're not compatible.
35:25
But you know, like do you find out all of these weird things
35:28
and you have to help them work through it.
35:29
Whereas, if it's as medium priority task it will people.
35:33
Oh, I don't know what to do. I'll just wait until someone
35:36
tells me what to do and then it did right.
35:37
Never gets ya can help. That for yeah, well that kind of
35:42
rolls into the next question which is how I think.
35:46
I read your hiring for a staff machine learning engineer.
35:50
And I would imagine that is for this type of thing.
35:56
I feel like if I understood correctly, you are allowing the
36:00
company's the Enterprise companies to also bring their
36:03
models and plug it in to promote it.
36:05
Is that correct? Yes.
36:07
From Big Tech perspective. That's how Moles are frequently
36:10
built. As you frequently, we were
36:11
talking about Signal Management. These are frequently out
36:14
literally other models owned by other teams and there exists
36:18
Technologies to aggregate them all together and have like a
36:21
final model. That does the ranking.
36:23
We're using that type of Technology.
36:24
Yes, that's right. Yeah.
36:26
So it's like the when you plug into Pinterest or whoever
36:33
Pinterest can say, okay? Now we have these models from
36:37
this side of the company. We want to start Running them on
36:40
promoted that's what it is. No, they would sell us the model
36:44
execution result. So you would send us the score
36:46
for users or four items or computed on the insertion and
36:50
send them to us on our metric system.
36:51
We won't actually host a model and executed on our side.
36:55
You just send us. Okay.
36:56
Here's our here's our model outputs are so that interface
37:00
you can okay and only like a binary, we would execute it and
37:06
what you can do is if they have a Jupiter know Look, and they
37:10
can execute it and then send us the evaluation results and load
37:13
it to our item content management system API or user
37:18
content management API or if this model is appropriately
37:21
designed they can execute it in real time on a request with low
37:25
enough Layton sees so like a relevance modeling, does this
37:28
item match this match this query and they have some sort of score
37:32
or frequently this will be in an existing search retrieval
37:36
system. With a head like a search
37:37
quality score. You can Send that to us and then
37:40
we can model on top of it. And the way our system works is
37:44
we're constantly re selecting features and modeling on top of
37:47
all of the signals we have available and will tell you hey
37:50
here's how important the signal you sent to us was in the final
37:53
result and we can also like do a hold out if you want to to see
37:57
here so much improved by including it or not, for
37:59
example. Now we're getting to the end of
38:03
time and I've got one last question for you which I think
38:05
everyone is really interested in because you Change real-time and
38:11
batch. And that is a Hot Topic that
38:13
comes up quite a bit in the ml Ops community and I'm wondering
38:17
in your eyes. What are the biggest gotchas
38:21
when you're doing streaming and you're doing real time?
38:25
Oh, a lack of commitment. And the second is having or
38:28
streaming system so slow that it may as well be batch.
38:31
The commitment part is, it's really difficult to build
38:34
streaming. It's like you usually have a
38:36
batch system first and it works pretty well and When you try
38:40
doing the streaming system and not only, is it a huge load and
38:44
all this infrastructure and now you have like ml Ops Team have
38:47
to watch it or data Ops team has to watch it, but maybe you run
38:51
an a/b test. And it wasn't that impactful or
38:54
something like this. And this comes into the lack of
38:56
commitment, which is, oh well, we'll try it, but you didn't
39:00
design your product or your strategy around having real-time
39:03
availability it was, hey, you have an existing product and
39:07
system designed around. Ouch like this sort of hey it
39:11
takes a while for signals to become available and then you
39:14
sort of you just replace that directly with streaming and and
39:17
then it's like oh well do an a/b test tube.
39:19
Will be data-driven about this, this is not commitment.
39:22
This requires no management really except for resource
39:25
procurement and project management leadership is like
39:30
Tick Tock. Like oh it is going to be
39:32
reactive to what you were doing right now and this product
39:35
wouldn't work at all. Fundamentally it would be a
39:37
fundamentally different product like the difference between
39:40
Facebook Newsfeed from 10 years ago and Tick-Tock today if
39:44
reacting in real time to what you're doing or or like from an
39:48
ad system it's hey we have we started with this really ad hoc
39:54
you know bid your click and you get a report every day and now
39:57
we're going to put under this real-time system and somehow
39:59
that's going to make it better. Well, maybe a little bit but
40:02
that's different from 0. We have this real time pacing
40:05
system that is streaming results and it's it's going Dynamically
40:10
adjust your internal systems to get an inefficiency like how
40:14
Facebook is built their discount Pacer system, for example.
40:17
So, the commitment piece is, did you design your company and
40:21
product strategy around streaming?
40:23
Or is it just like an upgrade of a subcomponent that you could
40:28
take or leave, depending on an AP experimentation and the
40:33
latter, there's nothing wrong with it.
40:35
It's just don't expect to become Tick-Tock because you replaced a
40:39
hive job with the Flink job and you know, some back-end system
40:42
or letting it run so slowly may as well be bad.
40:45
The second one is, it's not streaming if it's not real time,
40:47
let's say you're joining events. Okay.
40:50
How long are you waiting for your event three hours?
40:53
If it was like some sort of user recommendation system is someone
40:56
still on your side anymore after three hours?
40:58
No it or like, hey, it's just takes a really long time for
41:03
this to go through your whole big long pipeline to be able to
41:06
get the response through whatever is applying that
41:08
signal. To our to whatever is consuming
41:11
that signal. Well you need to figure out what
41:13
the latency these are for your product and you have to meet
41:18
those Layton sees if you don't have any idea and it's just well
41:21
let's just make it faster and hope it's better somehow.
41:24
Then you're more of the former case but it's a tremendous
41:27
investment in complexity and infrastructure costs and it
41:31
might be a very modest Improvement of kpi metrics.
41:35
If any versus hey I want a real time recommender system.
41:39
Oh, that means it needs to react in like three seconds.
41:42
Okay. Do you have the full signal
41:44
pipe? All the way through the system
41:45
to whatever is consuming and three seconds?
41:47
If not. Then why are you doing screaming
41:49
at all? Why don't you just do batch?
41:50
It's obvious that you've seen some things.
41:52
We're also hiring our biggest need, by the way.
41:55
Yes, we have a job wreck for, for machine learning.
41:59
We have three more around data infrastructure and streaming.
42:03
So if you want to build real data streaming services that get
42:06
used for real value, come work at promoted and we go Come work
42:10
for us, the Flink Engineers. I want to bring this back home
42:13
to guess we're coming at that at over time.
42:15
But at the end of the day and this is what promoted is about.
42:18
You have one Marketplace profits, you pull from your
42:22
Marketplace, have to come from somewhere.
42:25
Have to just there's one Marketplace you have one mobile
42:28
screen. You need to optimize the entire
42:31
Marketplace altogether. And the only way you can do that
42:34
is be super honest about where your value is coming from so
42:38
that you can train on. Models that can deliver that
42:41
value. If you say, hey, maximize.
42:43
This ridiculous nonsense story. I've created for pulling forward
42:46
budgets. This quarter from my brand,
42:48
advertisers, your models will happily go.
42:50
Do that. And that's sort of tech debt is
42:52
different than the sort of tech debt of what I think people
42:55
frequently think of is hey where my signals and data
42:58
infrastructure is more like an organizational, trust debt,
43:02
those kind of compromises compounding over time.
43:05
End up with a lousy system, it will still deliver search and
43:10
feed results in ads and they will charge some amount of money
43:13
and people will use it, but it just won't ever work really
43:16
well. Andrew, we I don't even know
43:21
what to say. I think this has been awesome.
43:25
You blew my mind a little bit. It took me a minute to really
43:28
wrap my head around. What exactly you were doing and
43:32
why? And now that I see the
43:37
motivation behind it, and what you are are dealing with and how
43:42
you are impacting. It's enticing.
43:46
Congratulations on on what you're building and I To see
43:52
this popping up all over the place soon enough.
43:54
If anybody out there is interested in doing stuff with
43:58
streaming, hit up and Drew. I, uh, I'm guessing on LinkedIn
44:03
is the best place to talk with you.
44:04
Otherwise LinkedIn in the description below and yeah,
44:10
yeah. Linked In A dataset promoted II.
44:14
Well, thank you write on. There we go.
44:17
Hi, this is Michael Eric. I'm the founder of pun intended,
44:20
Innovation and confetti. I and this is a quick Public
44:23
Service Announcement. If you're tired of having
44:26
nightmares about the machine learning models, you have in
44:28
production, then I encourage you to subscribe to the ml Ops
44:31
Community podcast, it's where all the cool ml Ops kids hang
44:35
out and learn how to tame their ml models.