Latest / Agentic Conversations (formally mlops.community) / Ads Ranking Evolution at Pinterest // Aayush Mudgal // #211

Transcript

Ads Ranking Evolution at Pinterest // Aayush Mudgal // #211 · Agentic Conversations (formally mlops.community)

0:00
Hold up. Before we get into this next
0:02
episode, I want to tell you about our virtual conference
0:06
that's coming up on February 15th and February 22nd.
0:10
We did it two Thursdays in a row this year because we wanted to
0:15
make sure that the maximum amount of people could come for
0:19
each day since the lineup is just looking absolutely
0:24
incredible. As you know, we do.
0:27
Let me name a few of the guests that we've got coming because it
0:31
is worth talking about. If you got Jason Louie, we've
0:34
got Shreya Shankar, we've got Drove who is product Applied AI
0:39
at Uber. We've got Cameron Rook Wolf
0:42
who's got an incredible podcast and he's Director of AI at Rebuy
0:45
Engine. We've got Lauren Lockridge who
0:49
is working at Google also doing some product stuff.
0:53
Ohh, why is there so many product people here?
0:55
Funny you should ask that because we've got a whole AI
0:59
product owners track along with and engineering track and then
1:04
as we like to, we've got some hands on workshops too.
1:07
Let me just tell you some of these other names just for a
1:10
moment, you know, because we've got them coming and it is really
1:14
cool. I haven't named any of the
1:15
keynotes yet either, by the way. Go and check them out on your
1:20
own if you want. Just go to home dot ML OPS dot
1:22
community and you'll see what we've got.
1:24
Tungi, who's the lead researcher on the deep speed project at
1:28
Microsoft. We've got Holden who is the open
1:31
source engineer at Netflix. We've got Chai, who's leading
1:36
the AI platform at Uber. You may have heard of it.
1:40
It's called Michelangelo. Ohh my gosh.
1:42
We've got Fazan, whose product manager at LinkedIn, Jerry
1:46
Lewis, who created good old Lama Index.
1:50
He's coming. We've got Matt Sharp, friend of
1:52
the pod, Shreya Rajpal, the creator and CEO of Guardrails.
1:58
Ohh my gosh, the list goes on. They're 70 plus people that will
2:03
be with us at this conference, so I hope to see you there.
2:09
And now, let's get into this podcast.
2:12
Hi, I'm Ayush. I'm a senior machine learning
2:14
engineer at Interest and I like to have my coffee, but I think
2:19
for Indians, like I've been not used to black coffee and stuff.
2:22
I think Indians used to have some kind of, like I would say
2:24
fake coffees, like ready to make coffee.
2:27
You add milk to it and that's it.
2:29
And I think that's the coffee I still like.
2:32
Come back to the one and only ML OPS Community podcast.
2:36
I am your host Dimitrios, and today we're talking with Ayush.
2:40
This is what I would call a candid conversation.
2:44
I appreciate everything about how honest and upfront Ayush was
2:50
with his whole experience from 2018 till now, working on the
2:57
ADS team at Pinterest and bringing ML into their
3:02
capabilities. He walked us through everything
3:05
from the nascent ML project to the advancements that they've
3:11
had. And I think one huge takeaway
3:15
from my side was along the lines of how evolution happens and how
3:20
evolution happens when you're at a big company like this.
3:24
Because in 2018 they had to make decisions on what tech stack
3:29
they were going to be using. And he talked about how the
3:33
pipelining tool that they were using, they ended up having to
3:36
get off of because it was something that was out of
3:38
Twitter and Twitter ended up not supporting it anymore.
3:41
There was no company behind it and they realised that this
3:46
might not be the safest bet. So they got off of it and they
3:50
had to evolve and they did this many times throughout their
3:54
journey, as he goes on to explain.
3:56
And so I like looking at this evolution and how he breaks down
4:03
the evolution in terms of what the ROI is every time they
4:09
upgrade or they make the choice to quote unquote lift and shift.
4:13
The thing that kept coming back to me was, is the juice worth
4:18
the squeeze? That's one of the sayings that
4:20
my dad used to always tell me, you know you you're doing this,
4:23
but I don't know if the juice is worth the squeeze on that.
4:27
And he thinks about it in that way.
4:30
He really thinks, OK, what's the ROI?
4:32
If we're going to rip out some technology and make sure that we
4:37
have a better technology, is it going to be worth it?
4:41
And are we going to be regretful 2-3 months down the line or a
4:48
year down the line because a potentially better technology
4:51
has come out? So he talks through the decision
4:54
making on that and what they did specifically for their Ads
4:58
optimization platform at Pinterest.
5:02
Hope you enjoyed this conversation.
5:04
As always, if you do share it with one friend, it will mean
5:08
the world to me and I will see you on the other side.
5:18
What are you doing in India? You've been travelling the
5:20
world. What is going on?
5:22
Yeah, I think, I think Pinterest is very flexible ourselves.
5:26
Like I think they allow us to walk like 90 days outside the
5:29
country and I thought it's a better time to enjoy that like
5:34
I've been. I think out says number mate
5:37
somewhere. I went to Europe for like 2
5:40
weeks. Then being in India, I think
5:42
this is a time where it's really auspicious to get married in
5:46
India and they're like a couple of my friends who are getting
5:48
married. So I thought, let's maybe just
5:50
pay back. Ohh.
5:52
I thought you were gonna tell me you were having a wedding.
5:55
Ohh, my wedding happened like two years ago.
5:58
And so, well, congratulations, simulated congratulations.
6:01
Yeah, thanks so much already. I think printers allows that.
6:05
They're not many companies right now, like there are many
6:08
companies are calling people back.
6:10
But I think Pinterest has been in a good spot there, still
6:13
allowing people to work. Remotely so when it comes to the
6:17
Indian weddings, be honest with me.
6:19
Have you been to one with an elephant yet?
6:22
Yes, just recently last week, yeah.
6:27
See So Why? Fun.
6:29
Yeah, it's it's fun. It's different than we're used
6:33
to in America. But yeah, it's Yeah, When do we
6:35
have? Yeah, whenever my Indian friends
6:38
are getting married in the US, I ask if they're going to have an
6:41
elephant and they're like, dude, you know how expensive that
6:44
would be? That's just not in the cards
6:46
right now. So either I have the wrong
6:48
friends or if they need to go to India.
6:51
Yeah, I think, yeah it's India is a good, it's a good time for
6:55
India right now it's like so the normal other it's pretty hard if
6:59
you go in the summers. Yeah.
7:01
So let's give people a little bit of background on what you're
7:04
up to, what you've been doing, because you've been working with
7:07
interest, as you mentioned, you've been there for a while
7:09
and you've been loving it. You haven't even changed ever.
7:12
You haven't changed the team. So maybe we should start with
7:15
that. Like, what's the deal with this
7:16
team you're falling in love with?
7:18
Yeah, I think that's a good question.
7:20
Like I joined Pinterest in March of 2018, so I'm nearing 6 years
7:25
I've been on the same team. Like initially I never wanted to
7:29
be on that team. I would say like, that's a funny
7:31
part of it. I was like, what?
7:33
Like if I want to be like ads, Like I want ads ranking.
7:37
Like our job is to like rank ads and make sure the best set of
7:40
ads correcting the advertisers to users.
7:43
But I was like this was the last thing that probably I wanted to
7:47
do in 2018, but then luckily I ended up on this team and I
7:51
haven't changed the team ever. Like it has so much to offer.
7:54
There's so many things that you can do both technically and also
7:58
like as a product, like both from a technical and product
8:01
standpoint, it has satisfied, I would say like what I was
8:05
looking for in a job and that's exciting.
8:08
Well, it seems like you've gotten to create a lot of
8:12
different projects right over your eight years or since 2018.
8:17
I guess my math is not so. Strong for years 6.
8:21
Years. I knew it was wrong, but I just
8:23
want to tell myself how wrong I was.
8:26
I wouldn't admit it to myself, but I guess it was It was pretty
8:28
wrong. So the thing about it is that
8:32
you've created a whole slew of projects when it comes to
8:37
working with this team. And if it's alright with you, I
8:40
would love to just breakdown these different projects and how
8:44
you created them, how you went about designing them and what
8:47
you would do different. Now that we are in 2024 and
8:51
obviously technology has advanced quite a bit.
8:53
So ML in 2018 was a whole different beast than ML in 2020
8:58
and now we even call it AI. So it is, in 2024, a whole
9:03
different beast. Yeah, I think, yeah, I think
9:06
looking back, I think it has been a long journey.
9:08
I would say in machine learning, I think in 2018 like I think you
9:14
you're not doing like we were doing like product building,
9:16
like winters was growing at a faster pace and like advertising
9:20
was something that printers had started.
9:23
I think we were around like pretty much like just building
9:26
products, not caring that much about how to make the products
9:29
better in terms of machine learning, but like just getting
9:32
from zero to one. And one of the major projects
9:35
that I worked on was like conversion optimization.
9:38
And one thing about ads business is like you have like good
9:42
leaders at Google and Facebook where you know, OK, these are
9:46
the things that work for them. And it came up to the point that
9:50
how do you make it work for Pinterest?
9:51
Like you have some direction, you have some direction.
9:54
This is what you want to build. But how do you build is totally
9:57
different at different companies depending on your users and your
10:01
advertisers. So I think we knew that
10:03
conversion business is big. It's going to work because it
10:07
works for our competitors and that's what the advertisers also
10:10
care about. So what it is mostly about is
10:13
like they get when you start building products, the
10:16
simplistic product you would build is to like just can you
10:19
deliver more of these ads to users than the next product
10:22
would be. Can you drive more clicks on
10:24
these products or that like advertisers care more about
10:26
clicks, not just people viewing it.
10:29
And then the next step would be like, it's not only about
10:32
clicks, but can you get users to convert more on those products
10:36
like buy something or take to like an active card on their
10:39
website and all of those like complex things which advertisers
10:42
care about. So conversion product refers to
10:45
the later product. So that's where when I joined
10:48
the other two products were already built up in general like
10:51
click product was there, we had we had that also the impression
10:55
product but conversion product was not there.
10:58
The one thing in 2018 I would say it was not much machine
11:02
learning at that point. It's about how do you connect
11:04
the pipelines together, how do you get through that and I think
11:07
one of the. Major pipelines look like, What
11:10
was it? What?
11:10
Yeah, and that's totally different what that is today.
11:14
But like in terms of at that time you were using for basic
11:20
Etls you are using they got gasketing like Skylar, Gasketing
11:25
and Scalding that came from from Twitter and until maybe last 1
11:31
1/2 years we were still relying on that kind of a legacy stuff.
11:34
And I think over time you started removing like less of
11:37
those like legacy technologies because I think both from the
11:42
technology was not getting updated but also finding
11:45
engineers who are familiar with it kind of toolkit is also gets
11:48
harder because not many people are using.
11:50
So one thing at least over time like Pinterest starts started
11:54
moving that those colour not Scala but it was in Scala but
11:58
that cascading and building jobs two more like Sparks Parkways
12:02
jobs and I think that has something which bag in 2018 was
12:06
not that popular as I saw what we had in 2018 was probably what
12:10
we started with and then. This is fascinating that you
12:13
mentioned that just to drop in a line real fast is that it wasn't
12:18
necessarily because of the pain that the engineers were feeling
12:22
on the pipelines when they were using Scala jobs in whatever
12:28
like 2022 or 2023. When you said you replaced it,
12:31
you ripped it out or you slowly migrated over to Spark jobs.
12:35
It was more because of the ability to find engineers that
12:40
understood how this worked and were proficient in it.
12:44
And then also, you had a little bit of fear.
12:47
I can imagine if Twitter wasn't keeping tabs on this roject
12:53
anymore and there wasn't really any support behind it.
12:55
Yeah, that's scary. Yeah.
12:58
Yeah, I think that's a good point.
13:00
And I think both of those reasons and also I think Spark
13:03
provided a lot more flexibility, scalability and scaling was not
13:08
providing that over time. So I think like things start
13:12
working in 2018 might not work in 2024 depending on progress
13:16
that happens in the open source community specifically.
13:19
And I think that fuels and then migrations are hard.
13:23
So you need to make a choice at some point, like whether it's
13:26
worth to migrate or not. Yeah, but.
13:29
Yeah, the ROI of the migration too, that's huge piece.
13:32
So don't be more from like Tensorflow 1 to Pytorch.
13:35
Like we just changed the complete language in general
13:38
because I think we thought and I think it's paying off pretty
13:41
well today that we made that decision.
13:44
But it's always hard to make those kind of decisions as such.
13:48
Yeah. In the moment you're not quite
13:50
sure. You're like, I think this is the
13:51
right move, but we'll find out in five years.
13:54
Yeah. But I think I would say the key
13:56
is to keep innovating and keep checking like what's working
14:00
where like in the in the industry or like what's working
14:02
well in the research community. And I think having systems that
14:06
can migrate faster to new technologies as needed, I think
14:11
that's the key to keep keep iterating faster.
14:15
At some point like in 2018 like we used to have a totally like
14:20
an hybrid stack of machine learning.
14:22
Like we used to train models like deep learning was speaking
14:25
up for sure in recommendation systems.
14:28
But at that scale like the way we were doing was we used to
14:32
have, we don't used to do deep learning like deep learning
14:34
started in 2018 like moving towards deep learning.
14:37
But we used to have like an example CBTS totally different
14:41
language trying to embed it into like a tensor flow based
14:45
logistics and totally different and then embedding that again
14:49
into a C library that we maintained for serving which is
14:52
kind of like critical systems. But that was what we had, I
14:56
would say at least in 20/21/20 somewhere in 2020 to 2 like like
15:01
slowly systems started moving more towards more principle
15:05
approaches of doing machine learning.
15:07
Yeah. Well, sorry, I totally took you
15:09
on a little bit of a sidebar on these pipelines.
15:12
But get back to this conversion optimization.
15:14
You were going for step three, right, to actually get people to
15:17
buy things when they clicked through an ad.
15:20
So you did step one, which was just showing people ads.
15:23
You you had Step 2 also in there by the time you showed up in
15:26
2018, which was getting the right ads in front of people.
15:30
And then it was your job to figure out how to get people to
15:35
actually buy things once they clicked on the ads.
15:39
Or get the ads to people who were in a buying mood, I could
15:43
say. Yeah, I would say the letters
15:46
like how to identify people who are buying state and also which
15:51
buying state like what particular ad would they buy in
15:53
general like we have 10 thousands of ads.
15:56
You can show any what like which one would be the best candidate
16:00
so as to like and that improves like learning that best
16:03
candidate is important because it helps Pinterest to not show
16:07
irrelevant ads. It helps advertisers to get
16:10
value by showing the right set of ads to the right set.
16:13
People like I think all three needs to be balanced in some
16:17
sense. So how did you go about doing
16:19
this? What was your idea when you came
16:22
through? You were you.
16:23
You must have done something. So I think if you look back, I
16:27
think all of these problems like driving clicks, like driving
16:31
convergence, like like are very correlated to each other.
16:34
In general, like it's just about, if you just look from a
16:37
machine learning perspective, it's just about OK, this is your
16:40
training data set. If we show X to the user, do
16:44
they buy or not? It's like a very simple like a
16:46
binary classification problem. And just in terms of a machine
16:50
learning model, it's not very complicated.
16:52
But then if you look at, you need to change your pipelines to
16:56
get this data like before when we are getting clicked datasets,
16:58
everything is happening on site at Pinterest and all of those
17:02
kind of dogs are owned by like are governed by Pinterest
17:05
architecture. But for convergence this is not
17:08
directly you need to have like a integration with your third
17:11
party providers. How you get that data, how do
17:14
you transform that data, identify and then pass it for
17:18
training. So you had like more complex
17:20
setups in your pipeline, simple very similar to like Etls that
17:23
you have, but you would have an additional pipeline than making
17:27
sure that this pipeline can connect to your model training.
17:29
But model training is just one piece of it.
17:32
But how do you use that predictions of convergence in
17:35
your life when you're deciding on ads?
17:38
Then the other components that you need to build, then train
17:40
your sales team to sell this product.
17:43
Then there would be a lot of bugs.
17:44
So you need a lot of visibility over it, like what's going wrong
17:47
so that you can figure it out and then it needs to be
17:50
performed like you cannot have like because you are now
17:54
comparing this product to a product which is driving plaques
17:57
and you need to be better than that for sure.
18:00
So at this point it needs to come with some level of
18:03
personalization like it can't just come with something random
18:06
like previous priors or something because that product
18:09
would not work. So it has some baselines that
18:11
you need to you need to at least be.
18:15
Well, it's in the name, even performance marketing, right?
18:19
It has to be performant. And the other piece that I'm
18:23
thinking about is you had to do this real time right?
18:27
There wasn't like, yeah, yeah, break that down.
18:31
So even in 18 like when I joined in 18 like I think adds like one
18:36
thing that gives adds different than like non ads is it needs to
18:40
be real time. But also some of those signals
18:43
need to be like computed like very real time something like
18:46
because ads govern how much you are spending, right.
18:49
And then based on your predictions you decide how much
18:52
you want to charge your advertisers and you don't want
18:54
to overshoot their budgets as such since their budgets are
18:57
faked like they would say only $100, they won't give you more
19:00
than $100. You need to make sure this
19:02
calculation is type that we're spending or underspending for
19:06
sure, like if they have $100 you will want to revise for $100
19:11
that you want to spend. And then given the scale
19:14
printers were still pretty big like all of these predictions,
19:17
all of these models were making predictions in real time.
19:21
And having a lower latency tool like having I think something
19:25
around like 300 to 400 milliseconds like in that range
19:28
you need to maintain that latency.
19:30
So I think the modelling architectures were not that
19:33
complex at that time. Coming to 2024 I think we
19:37
started to hit the latency budgets in general like what
19:40
else have becoming complex and I think now and also like we have
19:44
this two tiered kind of structures you don't score all
19:47
the ads like it goes through like targeting and then writing
19:51
so that controls the latency to. Today it's 2024, right?
19:57
You set this up in 2018. What has changed?
20:02
What now are you looking at? And you mentioned how you've
20:05
migrated the pipelines, but what other things have evolved since?
20:11
Yeah. I think one thing in 2018 like I
20:14
think machine learning was brought that much like you would
20:17
probably plug in like there's a lot of plugging that was
20:19
happening in the system, many different like heterogeneous
20:23
systems coming together in 2024. I think that homogeneous Ness is
20:28
coming in like entire pipeline is being trying to return in the
20:31
same framework so that you can control more things.
20:34
And one thing was in 2018 that is much different than what
20:39
we're doing today is like data condition, like there are like
20:43
too many steps into preparing your data before it can be
20:46
trained on a model. So these steps are getting, we
20:49
want to reduce these steps like intermediate steps and make sure
20:52
that it's like 1 big data set, one model training, it's not
20:56
like 5-15 different intermediary tables and monitoring them
21:00
becomes harder. So that's a picture.
21:03
The other shift is I would say like in 2018 makes sense.
21:07
We are building U systems like traditional systems like
21:11
monitoring and visibility was missing in most of those systems
21:15
like it was working and in most of the time it would work pretty
21:18
well. But like developing that kind of
21:21
a monitoring to make sure that everything is right from your
21:24
data appropriation into your model health and like to your
21:27
end to end pipeline health. I think those are things that
21:30
are mostly taken secondly as you're building new products.
21:34
Thirdly, I would say features like features like earlier I
21:38
used to be very like nested objects, like everyone would
21:41
have flexibility to write something, they would write
21:43
their own logic and all teams would like do it in silos.
21:47
They were right, they're all logics to do whatever they
21:49
wanted to do and then translating and we had like a
21:52
lot of different components. So you need to like translate
21:54
from one language to another and all of those kind of things.
21:58
So that is becoming standardised and I think printers has a bug.
22:01
People want to read like Milan, but I think standardising those
22:05
processes, making feature engineering more would say more
22:10
standardised and shareable across different use cases and I
22:14
think that is fueling more faster iterations and making
22:18
sure that you can iterate but also share knowledge across all
22:22
the use cases that Pinterest. And when you talk about
22:25
monitoring, are you talking about monitoring on basically
22:29
every level, like from the data flow level to the systems level
22:33
to the model output level? Yet today we are monitoring
22:38
mostly everything in general like saw 40 offline pipelines
22:42
where our data is being prepared to our online pipelines the way
22:45
we are serving. And for every model, we monitor
22:49
their predictions like whether they fall within a specific
22:52
range over like a time like weekend, like a time frame in
22:55
general like last week or like like a day over day kind of
22:59
emissions. If like these are a lot of
23:01
metrics you need to make sure then how you set up alerts.
23:05
But like monitoring all of those things, monitoring your features
23:09
for how they are logged and used in your training pipelines
23:11
versus also how they are being served And all of those things
23:15
is being continuously monitored and also alerted to make sure we
23:20
can catch things earlier on. And we have more automated
23:24
pipelines. If something is off, I think
23:27
things would not get promoted and they would be stopped from
23:29
being used. I think all of those monitoring
23:32
is in place today to prevent many incidents that we used to
23:36
have and have back-to-back earlier in general.
23:39
So that has like reduced in that sense.
23:41
But also it means now incidents are more complex than what it
23:44
used to be like. There could still be incidents,
23:47
but I think they have become more complex.
23:51
Yeah. So we were talking about this
23:54
before we hit record. And it's interesting when it
23:56
comes to Pinterest because even in 2018 when you joined, you had
24:00
to be thinking about scale from day one, right?
24:03
There was already a whole lot of users.
24:06
And so how did that factor into the decisions that you were
24:09
making? Yeah, I think scale like I think
24:13
today Pinterest has about 489,000,000 active users.
24:17
So I think scale is definitely important and what it means is
24:21
you cannot push anything to production without testing it.
24:24
So definitely that is something that needs to take care, but
24:27
also we need to make sure our systems are healthy to handle,
24:31
handle that scale in general like if you look at like also
24:35
like in let's say in holiday periods like scale increases in
24:38
general like in the other part of the years like there might be
24:41
like lower loads on your systems like how do you manage this
24:44
node. So I think Pinterest on the
24:46
server side, we do have like auto scaling and all of those
24:48
impacts so that we can reduce the usage of our serving systems
24:52
when the scale is low. We can increase it when the
24:55
scale is high. So those are handling some of
24:57
those scales, but they're also you need to make sure that
25:00
modelling like whatever the models we have, like earlier on
25:04
when the number of users were lowered, we can probably have
25:08
simple models. We cannot afford to have more
25:10
complex models either, because the powers that you might get
25:14
from it might not be to add into the revenue that the model
25:17
generates. So you need to make sure that
25:20
your complexity is maintained with your business health.
25:24
It goes with your business or your goals in general and
25:28
currently I would say like we need to make sure our pipelines
25:32
like overtime right back in there in 2018 or EDL pipelines
25:36
could finish early because the scale was lower but overtime
25:40
your scale has increased but you want to have the same like Slas
25:44
for your model trainings and they're training every hour.
25:46
So now you need to invest in optimising your pipe moves much
25:49
better or much more than you could do anything.
25:53
Like maybe make mistakes or like maybe do slower joys.
25:56
Not your joints are not optimised, but now I think that
25:59
becomes critical given the scale that White Plains also need to
26:03
be smarter than what it used to be before.
26:07
So you also went on to do a whole slew of other things, one
26:11
of which is the video ads ranking, and talk to me how that
26:16
was different than the conversion optimization, because
26:20
it feels like there's some similarities there, but there's
26:23
potentially a lot that was different.
26:25
So in this, I think with conversion ads, I think it was
26:29
like many of the components were there because you are dealing
26:33
with the same assets like images in terms of like your content
26:37
understanding. But when you move to videos, the
26:41
way you understand content changes in general, it's no
26:44
longer like a simple image, it's more about like series of
26:47
images. So you need to invest at that
26:49
like in terms of the model it probably remains the same which
26:52
now trying to predict instead of a converger like whether the
26:55
user is going to view this product for let's say more than
26:58
two seconds or five seconds. So the problem from a moral
27:01
standpoint remains the same. But how do you integrate this
27:03
into your systems? Like do you have a good
27:06
understanding of the video, what this video is about?
27:08
Like same features that you would do on images. 1 simplistic
27:12
thing to start this product is you can just say that, OK, for
27:16
this video you have a representative image that
27:20
represent this video and that is what you're going to use.
27:23
So those those kind of that by just doing that you can just
27:27
treat this video as an image and then everything can just flow
27:30
through as what you used to have before.
27:33
But to make it performance and then slowly need to keep adding
27:37
more kind of features on the video understanding and can you
27:40
make those features better which is different than images, then
27:44
you need to be aware that this only impacts the video
27:47
optimization. You cannot reuse these these
27:49
kind of like new features across the stack.
27:51
So that is something then you need to balance between like how
27:55
much is the effort to do that versus like what's the potential
27:58
gain that you would get from that?
28:02
I like how you come back to that idea.
28:05
You said it before in the last question and now you're saying
28:09
it again and I want to highlight it because I feel like it's very
28:12
important that trade off of the effort to basically, is the
28:19
juice worth the squeeze? Is it worth it for me to put all
28:23
this time, energy and effort into this in order to get how
28:28
much of A payoff and how can you?
28:31
I guess my big question is how can you quantify that payoff
28:35
before you've actually done the effort and know about it?
28:39
So I think that that I think there are two things I would
28:42
say. One thing is, is the systems
28:45
ready to do that? Do you have systems in place
28:47
where you can do that easily versus its style that you need
28:51
to optimise your systems to do that?
28:53
For example, if I want to add video features, is the stack is
28:57
too complicated like let's say like back in 2018.
29:01
The way we used to do that was we used to have a separate
29:04
pipeline like for creating a training datasets and that was
29:08
based on efficient and add to the user will log all the
29:11
features that we used to have at that particular instance.
29:14
Because everything was real time and we didn't have a notion of
29:17
it's a feature backfilling or we don't have a notion to add
29:20
features historically back into your data set.
29:23
Because of that, if you need to train models like waters were
29:27
training on, let's say, six months of data or like 3 months
29:29
of data, you need to wait for three months before you add it a
29:32
feature into the system. Now that delays you by three
29:35
months. So at that point, I think we
29:37
started investing in improving our Paxil capacities in general,
29:40
like, we can do feature iterations faster, like because
29:44
if you thought this feature is going to work and it doesn't
29:46
work, you wasted three months thinking that it would work and
29:49
then it just goes down everything.
29:51
So if systems are not ready, I think that's one factor.
29:54
Like how much does it take? Like this you can think about
29:58
new things, but if the systems are not ready, you cannot go
30:01
about doing that and that reduces the return that you
30:04
would get if it's delayed. Other than that, I think it's
30:08
mostly intuitions at time like if you do things simple there
30:12
would be lower like lower risk in general they should start
30:15
moving with like because I think there are tonne of research that
30:17
might go around and you might think about good Powerflex ideas
30:21
that way. OK, this sounds fancy but then I
30:24
think the payoffs if you haven't done the simple things is like
30:27
lower in that sense because it's more complex to build those
30:31
complex fancy things out in general because there are more
30:34
chances of having bugs in the system.
30:36
So having great moving from like a simplistic thing to more
30:39
complex, I think that reduces a lot of the like risk and also
30:43
the product can keep growing iteratively over that.
30:49
OK, so I if I'm understanding it correctly, it is figuring out
30:53
what the easiest and most simple way to do something is until you
30:57
bump U against some kind of bottleneck or some pain that you
31:03
realise, oh this is really causing us problems.
31:05
Let's now figure out how we can alleviate that pain, that
31:10
specific pain, whether it is the feature engineering or waiting
31:14
for a model to train three months later and and that must
31:19
have been just painful. Trying to wait three months to
31:22
know is this feature actually useful.
31:25
And then you don't want to do anything else special because
31:28
you want to have like, OK, I I updated this feature and so
31:31
let's see if that actually works.
31:32
And so then potentially you're constrained for like 3 months
31:36
while it's training and so you're working with that, but
31:40
also you're waiting. It almost feels like you're
31:43
knowing that if we wait and see technology in a way is going to
31:49
kind of catch up with us. So the longer that we can
31:53
abstain from getting things too complex, we can leverage time to
32:00
be on our side so that when we do go complex, we have better
32:05
tools available. Yeah, yeah.
32:07
I think that's a better way to put it.
32:09
Thanks for doing that. But I think having places where
32:14
you can evaluate moving through like the latter infrastructure
32:18
or like better tooling. I think the tooling is the key
32:21
in general, like building that tooling like because when you
32:24
move fast you might not have the tooling and that bites you back
32:27
later in general. Yeah, yeah.
32:30
Because you've made those decisions and they're
32:33
potentially, I'm not gonna say irreversible, but it's a lot
32:37
harder to rip out the technology than it is to just wait and
32:42
choose a different one in a month or two.
32:45
Yeah. And what example I can share
32:47
like you and I like this hybrid system where you are serving in
32:51
C because that system touches users directly like it's in your
32:56
critical path of serving. Moving that system was pretty
33:00
hard in general like many teams like they had to spend multiple
33:04
multiple quarters to just because you need to move and
33:07
then you need to move by making sure you're still being
33:10
improving your business metrics. It's not that you stop all the
33:13
work, it needs to move in parallel like it's something
33:16
like you want to change like wings on your plane, but the
33:20
plane needs to keep flying like you need to change that in in
33:23
the slides. I think that is something having
33:26
the right tooling and having like right prioritisation calls
33:29
at that point. But eventually you need to like.
33:32
The thing is, it took I would say at least two years somewhere
33:35
there to go from that kind of hybrid system.
33:39
But it's paying off now today. So having that future mindset to
33:42
know, OK, like this definitely is a bottleneck and you can't go
33:46
further in this direction without changing it so that you
33:50
can be better in the future. I think that kind of calculation
33:53
is important at some point. And I know like still companies
33:57
who have such hybrid system, they're like, OK, we want to
34:01
move, but it's harder to move and they don't make the switch.
34:04
But I think it just slows you down in general for the future.
34:08
And do you have any advice for those migrations and how to do
34:12
them gracefully? So I think while it's commitment
34:16
for sure like need to be committed like things might not
34:19
look like very, very beautiful in the beginning.
34:22
So having that commitment from top down that we're committed to
34:25
do this. The other thing is building the
34:27
right tooling. It's not about like OK just
34:30
changing the frameworks but if you have the right tooling to
34:33
make sure that between A&B, the new systems, the everything is
34:38
probably similar. I think that is something
34:41
investing in the tooling is important to make sure that you
34:44
can cover bugs earlier on than later.
34:46
Because when you're doing this kind of system migrations like
34:50
specifically for these large scale models like even if one
34:53
feature is not translated in the you would have like systems
34:57
would just break and you would not know what's going on.
35:00
But can you go back in your systems and replay what happened
35:04
in two of these systems and see like we're privileged and I
35:07
think having that kind of understanding helps to migrate
35:10
poster. And also sometimes getting all
35:14
teams aligned is better, like getting one on like major teams
35:17
aligned is better because you don't want to have this
35:21
migration as a choosing target always.
35:23
Because if you keep iterating and improving the older system,
35:25
you need to make sure that at least there's some consensus,
35:28
like when there's like priority shift that needs to happen just
35:32
to the migration at that point. Because otherwise the post will
35:35
keep shifting and that makes it really hard for everyone to move
35:39
out. But you want that kind of period
35:41
to be very small. In general, you don't want it to
35:43
be like formalise, that nothing goes in, you're just migrate,
35:46
but you want to reduce the time to that kind of kind of like
35:50
shift system migration. That's right.
35:53
Yeah, because it feels like the longer that you don't finish,
35:56
the more chances that you have for second guessing yourself or
36:01
trying to implement something else new and saying ohh well you
36:05
know, actually this might be better, but if you have it as a
36:08
short window of time, then that shorter window of time is not
36:11
going to allow you to Add all these curveballs into the
36:15
project. Toward leaning, because I think
36:18
the migration is always on the infrastructure side.
36:21
You don't see the benefits when you're doing that, but it comes
36:24
out eventually. So I think it needs a bit of
36:27
support to make it happen generally.
36:30
The other thing that's coming through my mind really is going
36:34
back to that idea that you were talking about in the beginning
36:36
when you said the technology that we had chosen was no longer
36:41
being supported. So it was some open source
36:44
technology that didn't have anybody backing it and let alone
36:49
a company, right? And then it was becoming harder
36:52
to find people that understood this technology and could code
36:56
in whatever this language or could use these frameworks.
37:01
And so as you're thinking about migrating and making a new
37:06
design decision, how are you making sure that that very same
37:09
thing will not just happen again in four years?
37:12
Or do you take that as you know what, in four years we'll
37:15
probably be looking at this again and we need to do it
37:18
again. But for now, the best thing that
37:20
we have is XYZ. Yeah.
37:23
I would say like it's always a continual looking forward.
37:27
Like you would say, OK for now this is the best thing because I
37:30
think specifically in machine learning and also other software
37:33
engineering, things are moving pretty fast.
37:35
Machine learning is moving pretty fast even right now I
37:38
think things are moving pretty like back in 18 Tensorflow was
37:42
the group to production system like everyone in the industry
37:45
who are productizing system. So we have 10s of system, but as
37:50
you are in Tensorflow world, Tensorflow two came in.
37:53
That was a total surprise. But you can go from one to two
37:56
directly. It's not like you can have that
37:57
version upgrade seamlessly. And that was the point when
38:00
printers decided to do like 10 or two versus Pytorch, like what
38:05
makes sense. And then what we did around that
38:08
time was something we had a bunch of eight years working on
38:11
translating small piece of modelling code from 1:00 to
38:14
2:00, a bunch of engineers translate and those same
38:17
engineers translating it also into Pytorch to see what their
38:20
experience was. And after all of those
38:23
considerations, developer feedbacks and stuff, he decided
38:26
OK, try to access and that was a totally like we moved away from
38:31
Tensorflow and moved to another. It's paving off pretty well
38:34
today. I think we see that industry or
38:36
like cadmium definitely is more quite touch friendly today.
38:40
So keep evaluating I would say in that sense for over like all
38:45
the time period I don't think it was, yeah.
38:49
Was there ever the idea of trying to support both of them?
38:53
Did that conversation come up? I think from machine learning.
38:58
So I think The thing is from machine learning platform
39:00
perspective, supporting less languages is better for them
39:05
because then they can focus on particular particular language
39:08
model. But I think it depends like
39:12
there are few use cases who were like no everything is intensive
39:16
flow we want to do tensorflow. But like given that tensor flow
39:20
two was totally different than tensor flow one, you need to
39:23
move to two. So I think that made it easier
39:26
but I think overtime I would say like depending on business use
39:30
cases that exist like they might have like a language decision.
39:34
But luckily for Pytorch at least I think things were favourable
39:38
that most teams are needed once major teams in the ecosystem are
39:42
career. I think the other teams also
39:45
need to eventually agree because I think that becomes the more
39:48
iterated because at this you know there might look the same,
39:51
but as people start to work on those systems you would have
39:54
more functionalities, you would have more things build up which
39:57
you can use for easy and cheaper if you're stuck with like the
40:02
framework that everyone is using.
40:05
Man tensor flow 2.0 the beginning of the end.
40:10
That is so wild to think about. Think back on the history of how
40:14
that played out. Yeah, I think it depends.
40:17
I think some companies are still using it.
40:19
So I don't think it's unsure. But yeah, but at least at
40:22
Pinterest for today, I think it's not there.
40:26
Yeah, that's wild. So now talk me through what
40:30
things you would have done differently knowing what you
40:34
know now, and I don't want to say like, Oh yeah, live your
40:38
life with a little bit of a regret.
40:40
But for the rest of us that potentially are building systems
40:46
and thinking through building these systems, what are some
40:49
things that you know now that you would have done differently?
40:54
Yeah they one thing I would say like when we are in 2018, I
40:58
think improving the way you train your models and also I
41:02
would say like in 2018 like training models were not very
41:07
easy like you need to like copy some core check in that code
41:10
like duplicate copy some code, check in the code, change some
41:13
parameters and that is probably just slows down.
41:17
I know like many companies have a similar thing even today like
41:20
you cannot easily train multiple different models like having
41:24
that flexibility like printers like your internal system
41:27
policies that was developed. I think that was pretty pretty
41:31
useful so that now if you want to chase things, can you do like
41:34
a no code version of it? You could just change
41:37
parameters, change things, don't need to wait for checking in.
41:40
I think that was something if we had earlier on would have been
41:43
useful. The other thing I would say I I
41:47
think it's easy to say that, but I think The thing is can we
41:50
remove the hybrid systems that we had like training in a
41:54
different language, serving in a different language.
41:56
But I think it also depends because I think that helped us
42:00
to go to market faster because that was what existed in 16th
42:04
and 15th to get to the market because of the systems are not
42:07
quite sure. But if quite hours was more
42:10
mature in 1516, I think you could have saved a lot of time
42:13
in those migrations. It's easy to say now.
42:16
But I think practically I think it might not have been the case
42:20
because I think those things are also developing I would say.
42:24
So in that sense, yeah, I think C plus is one language.
42:29
I think machine learning is you don't have that many machine
42:32
learning engineers who understand C say so.
42:36
And that's where I think where they I think in serving systems,
42:40
yes, there are many that you can find people, but not everyone
42:43
who is in Academy are doing machine learning would be
42:46
comfortable doing. CI think that's something I
42:50
don't know. Yeah, I was expecting you to say
42:53
C++ is a language that I never want to touch again in my life.
42:57
Yeah, but I think for lower latency serving, I think you
43:01
still need people to to know that to pull things off in
43:04
general. But I think that's something is
43:06
reducing over time. Yeah, Yeah.
43:09
So then now that we are in 2024 and as you look ahead, I mean on
43:15
all of these different projects or whatever you're working on
43:18
these days, what are some things that you are interested in
43:24
implementing in your stack or or not even implementing in your
43:29
stack, just things that are getting you excited?
43:33
So I think the thing I would say like when you moved from GB 18
43:37
is to like let's say neural networks.
43:39
I think we knew, OK, the first migration we did from GDT is to
43:42
neural network was kind of neutralise.
43:45
Everyone would think that OK, bringing neural networks you'll
43:48
get out of it. It was totally like neutralised.
43:51
It was kind of building the path goal of what we were doing.
43:53
But I would say like advancements that are happening
43:57
in like language processing where you have like a lot of
43:59
sequence of words coming in. I think the same kind of advice
44:02
for instance happening in recommendation systems where you
44:04
have instead of words you have like interactions that the user
44:08
is doing. In those sequence modelling
44:10
techniques are becoming complex and that's where I feel like
44:13
much excited like you can make things like NLP is like kind of
44:17
influencing like recommendation system.
44:20
So at least in the back end I would say like things are coming
44:23
together in general like pick those Transformers are in
44:26
recommendation systems, Transformers are in your
44:29
computer vision systems. They are not that separated out
44:33
industries like the use cases are different but I think those
44:36
techniques are getting adopted widely even implementation
44:41
systems. So that is exciting to see how
44:44
things are coming. Was that in that sense and how
44:47
that research is useful even for our models.
44:51
The other thing is I mentioned before that we used to have many
44:55
intermediate systems which slows you down.
44:57
Let's if you want to train a new sampling strategy or do
45:00
different sampling, it was harder.
45:02
But I think we are moving into integrating most of our systems
45:06
with to do like in Trainer samplings and moving things more
45:10
flexibly into your model training flows and that is
45:14
pretty exciting because why you can do much more things that's
45:18
like a faster velocity and not being bottlenecked by your
45:21
frameworks. Talk to me about this idea of
45:27
Transformers for the basically for the ad rankings and because
45:34
I think I understand it, but I want to hear you dive into that
45:38
a little bit more. Yeah.
45:40
So I think there are two ways you can think about Transformers
45:44
in general. So how this writing models
45:46
works, You have set of features that represent your users, some
45:50
set of features that represent your content or pin at printers
45:54
in some sort of features that represent their kind of
45:57
interactions that you are having with them.
45:59
So typically this recommendation models would have a way to learn
46:03
these interactions, like learn these feature interactions among
46:06
themselves as part of the model architecture.
46:09
And there are things like crossings like cross networks
46:13
and other kind of research there.
46:15
So one way you can also think as transformer is just kind of
46:18
learning how these features interact with each other.
46:20
There's no positional kind of system here, but like just a
46:24
transformer encoder can be used as a feature interaction layer.
46:27
Try to learn whether feature A is related to feature B and by
46:30
what magnitude. So that's one way.
46:33
The other way Transformers are being used is users.
46:37
When they come to the platform, they can interact with lot of
46:39
pins, They might click on some pins, they might hide some pins,
46:42
save some pins. So this is like a sequence of
46:45
interactions you can think about like user journey.
46:48
And if you look at like natural language processing, you can
46:51
think this is like a sentence of words and you can translate into
46:55
same way to say that what is the next probable word that would
46:59
come in this sentence. And say that if this is the
47:01
sequence, what is the next action or like what's the next
47:04
content that you might interact with.
47:06
Just model it very similar to what is in the text processing
47:09
domain and try to and that's what recommendation system is
47:14
like. You have seen everything, what
47:15
is the next you want to see and I think last solver and peak
47:18
comes very close to that and trust over is building block to
47:21
make that kind of like prediction or setup for
47:24
training. And that's where I think most of
47:26
the wins are coming from today, to understand the user better
47:30
and make it more personalised. Yeah, the sequence and mapping
47:34
out a sequence and then trying to decide what the next token is
47:38
like you get with, Yep, ChatGPT or what the next token is.
47:43
As far as what ad you should show, that is incredible.
47:47
And that's what I thought I was understanding.
47:49
And I I know I've seen a few papers on it, I just wanted to
47:52
make sure that I was getting it. Yeah, things are coming.
47:57
Like, I think, I I think that's a good thing about machine
47:59
learning. You take motivation from
48:02
somewhere, you combine it together and it mostly works in
48:05
your domain tools. It's probably just getting
48:07
motivations and trying those things out.
48:10
So the one thing that I have to ask about is because
48:15
recommendation systems, and you were saying it earlier, like
48:19
there is a very stringent need for things to be fast.
48:25
Yeah, and formers aren't necessarily known for the past.
48:29
How do you look at those two things and how do they coexist?
48:32
Yeah. So I think as we move through
48:34
Transformers, I think we started hitting our latency budgets in
48:38
general because they are not that fast.
48:40
One thing is before Transformers we were just doing CPU servings.
48:45
You can't do move to like GPU serving to unlock the similar
48:49
kind of latencies. One thing is it comes at higher
48:52
cost like, but if you're personalising better, you can
48:55
recover some of that cost, like that's one factor.
48:58
But I think at some point you will also hit the ceiling where
49:01
you cannot have the same kind of personalization unblock that you
49:05
might get by increasing infrastructures.
49:07
I think there I think things like new one visitation for
49:11
serving, you're reducing the model complexity in terms of
49:14
sequencing. Depending on how much benefit
49:17
you get, you can control the length of the sequence.
49:20
Then you can optimise and make your sequence because sequence
49:24
length contributes directly to how much cost you would have
49:27
because the larger it is the higher the cost it would be.
49:30
Then you can make the sequence more smarter but not having many
49:33
similar content in that sequence you can properly represent like
49:37
many similar content which has like a representative content
49:40
from that. So you can reduce your sequence
49:43
and be more smarter with your sequence to reduce that link in
49:45
that one. Thirdly, you can start to put
49:49
some of this processing into your offline systems, like
49:52
reducing the amount of things that you computed online,
49:55
because some of the longer sequences beyond let's say one
49:58
month or somewhere there can also be computed offline and
50:01
takes into your system and you can just reference that on the
50:04
fly when you're serving them. So some of those kind of more
50:07
complexities you can address and add into your system in that.
50:12
Sense in going going back to your idea earlier, because it
50:16
sounds like you are definitely not just hitting an open AI API.
50:21
You've got a whole lot of stuff going on behind the scenes with
50:24
your own models that have been trained on these sequences, and
50:28
you're doing some pretty nice optimization to make sure that
50:33
they're not too expensive and they're not too slow.
50:36
Yeah, yeah, yeah. That's totally true.
50:39
So going back to the idea we were talking about before on the
50:43
complexity side and staying simple and then getting complex,
50:48
I'm thinking like it feels like it's not possible to do this by
50:54
just going out and using one of these LM provider APIs because
50:58
of the latency requirements that you have.
51:01
Yeah, I think it's not possible. I think for that purpose
51:05
printers definitely uses let's say π torch for our training.
51:08
But our serving systems are built in house to do all these
51:11
kind of optimizations that you might not get off the shelves
51:15
are serving system definitely is much more complex in that sense
51:18
to handle those things and I think it takes a lot of time to
51:22
build those systems. I would say like the time to
51:26
build might be reduced right now because of all the advancements
51:29
that are there. But like for printers like we
51:32
moved to let's say deep learning based systems and like 2020.
51:37
Then we spent a lot of time and like making our systems
51:40
monitoring better around that time and things sequences
51:43
started coming somewhere in like 2022 to 2023.
51:46
So we have like even when you're making these sequences and like
51:50
when we added the first sequence, it was not transformer
51:52
based sequence, it was very simple like attention models
51:56
like no transformer in there. And then we started slowly,
52:00
slowly making sure our systems can catch up with the kind of
52:04
modelling techniques we want to do.
52:06
That's where I think right now I think we are at a stage that we
52:09
can invest in more complex than what you were doing last year.
52:13
It's. Incredible, dude.
52:14
Well, this has been fascinating, getting to pick your brain about
52:17
all this. I really enjoy how you think of
52:20
things and how open you are to all the learnings you've had
52:23
over the years. It's super cool to see and
52:26
thanks for coming on here. Yeah.
52:28
Thanks for inviting and also thanks for those for listening
52:31
so far.