Latest / Agentic Conversations (formally mlops.community) / "Real-Time" ML: Features and Inference // Sasha Ovsankin and Rupesh Gupta // MLOps Podcast #135

Transcript

"Real-Time" ML: Features and Inference // Sasha Ovsankin and Rupesh Gupta // MLOps Podcast #135 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Sasha, I'm a senior software engineer at LinkedIn.
0:04
Working on machine learning infrastructure.
0:07
I like Chino in the morning and espresso in the afternoon.
0:12
And I lost my obliques and Facebook, and Senior staff
0:18
engineer at Lake. I work on the search and
0:21
recommender systems at like them and I like my coffee.
0:26
So, welcome back to another Emmy.
0:29
Laughs Community Broadcasting. Cast.
0:30
I'm your host to Beatriz. And today, I am with none other
0:34
than the incredible Skyler. We were just talking about tiny
0:39
houses because Skyler, you're living in tiny house these days
0:43
man. Yeah, it's awesome.
0:45
It's amazing. I think I'm going to do more of
0:48
it and everyone is probably thinking.
0:51
I thought we just tuned in to the mo Ops Community podcast.
0:55
Not a tiny home one, but we To chitchat about a little tiny
1:02
house before we jump into the greatest LinkedIn podcast ever.
1:08
I ended up talking for like another hour with our guests.
1:13
After you left Skylar, you bailed early.
1:16
And I got all the information on how the newsfeed works and how
1:19
to kill the algorithm and get all the colors that I need.
1:23
But that is a something that is only privy to myself.
1:27
And I cannot share that with you right now.
1:29
Way. Do you have any key takeaways
1:33
from this chat that we just had? Yeah, I really loved.
1:38
You know how much they focused on making it easy to add
1:42
real-time features. You know, I think like talking
1:45
about real time is been really kind of in the Zeitgeist of
1:50
machine learning recently. I think starting from chips,
1:53
great post, you know, machine learning is going real time,
1:56
you're not ready and she's right, you know, like no one is
1:59
really ready. And like part of the problem is,
2:01
like most of the time real-time features are going to be
2:03
helpful, but it's so hard to get real-time features that just
2:07
nobody does them. So it's really cool to hear
2:11
about how Sasha had kind of like work through that dude.
2:16
That's crazy that you're saying that to me, because I look at
2:21
you as one of the people that is skeptical of real-time and its
2:27
uses and using it all over the place.
2:29
I am, I mistaken in that view? I am a skeptic on most things
2:35
and I think, you know, I think I think it's part of our problem
2:39
is that like real-time features are take a lot of effort.
2:43
And so you could toil away to, you know, get real-time features
2:48
added in and probably all the time you spent doing that was
2:51
time, you didn't spend like getting the low-hanging fruit, I
2:53
just think most of the time, there's like a lot more low
2:56
hanging fruit. Getting to real-time probably as
2:59
like a level. Maturity in machine learning
3:01
that most people aren't at. But hey, you know, I think if
3:04
there's somebody that builds some infrastructure that just
3:07
makes this dumb simple easy. And you don't have to think
3:09
about the real-time nature of it and you can just do it.
3:13
Make everything real time. Do it.
3:16
Oh my God. I never thought I was going to
3:18
hear that from you, man. It's like you've changed sides.
3:23
What is this? Somebody paid you off in the
3:25
background? Something happened.
3:28
You know. It must have been some Asha.
3:30
You know just seeing him smile again and let's switch sides and
3:35
for people that do not know you were working with Sasha back in
3:39
the day at LinkedIn. So, this is a nice little
3:41
reunions that you had. And for me I really enjoyed when
3:46
they were talking about how as you're looking at the news feed
3:51
of LinkedIn, they're trying to balance out what you're looking
3:56
for and what you're engaged in now versus that long.
3:59
Term engagement. And so that was a nice little
4:02
problem that they got into. And if you look at your news
4:06
feed, you do notice that sometimes their news feeds have
4:10
things like oh this person liked whatever or here's an event or
4:16
here's an update from a company and there's so many different
4:19
ways that you can interact with LinkedIn and then on top of that
4:23
layer in job posts. And so if you like job posts or
4:28
if you're seen to be Be liking job post.
4:31
There is that problem of are you liking the job post?
4:34
Because you're looking for a job is it?
4:37
Because you're just want to amplify it because, you know,
4:39
the person that is trying to hire or are you just in to
4:43
hearing about job, post all the time.
4:45
So with that, I thought it was fascinating to counter the to
4:51
the idea of your short-term wants, and those long-term
4:55
wants. Mmm.
4:57
Yeah, Yeah. It's a super hard problem.
4:59
All yeah, so very interesting to dive into the details around
5:03
that. So well you're in your little
5:07
tiny house and I'm in Europe. So basically in a tiny house
5:11
because as we said, everything is smaller here compared to the
5:15
u.s. even the roads. I mean, it's a whole different
5:19
ball game. Man, when I was here, I was
5:22
telling you I got really into just a little tiny house tangent
5:27
before we jump into the actual conversation.
5:29
I got really into the idea of how to maximize the space and
5:36
every square inch of your house and how you can set up different
5:40
things. And so I converted we had when I
5:43
was living in Spain, we had a dishwasher and we converted that
5:47
into a storage place because we never use the dishwasher.
5:51
So instead of having this dishwasher take up space in the
5:55
apartment rental, we made it into a storage unit and Those
6:00
are the things you got to do. I see a microwave behind you and
6:03
I'm super surprised that you have a microwave in a tiny house
6:06
because I feel like that's this. It has a full kitchen like based
6:11
on of Skilling on their tiny house.
6:15
Yeah, it is tiny but it is powerful and I imagine anybody
6:21
else that is into the tiny house, movement knows the guy
6:25
that owns this house. This Skyler is in right now,
6:27
he's kind of famous on YouTube. So That's a funny one that we
6:30
found out. Anyway let's get into this
6:33
conversation with the LinkedIn crew and if anyone out there
6:38
wants to join us in the ml Ops Community, we're doing all kinds
6:42
of cool stuff. We got done with the hackathon
6:45
recently. We've also got lots of
6:47
newsletters, we've got so much going on in real life, beat up.
6:52
Skyler, you're doing in real life stuff in l.a., right?
6:55
Yeah yeah. Come join us out in La even
6:59
though you'll probably Have to drive an hour to meet us living
7:03
in the same city. So yeah, and if you enjoy this
7:09
podcast, it would mean the world to us.
7:11
If you shared it with one friend that you think would also enjoy
7:15
this. Without further Ado, let's jump
7:16
into the conversation. So I don't even, you know, where
7:27
I want to start. I mean this is a LinkedIn
7:32
reunions Skyler's, here rupesh you and Sasha, both used to work
7:36
with Skylar what's changed? Since he is not at LinkedIn for
7:41
the better, or worse, whatever you want to go with.
7:46
Well, it's a, if I remember the meeting Skylar and I Had like a
7:51
few feet from where I sit that was like in the different life
7:57
and it says that that was before David.
8:00
Mmm, right? And and that's that kind of
8:05
change, then that impact that our life, but it is slow slowly
8:10
getting back to normal. You see more and more people on
8:14
campus skeletal when you are in the Bay Area, feel free to, you
8:18
know, jump for for for lunch and see for yourself.
8:24
Yeah, be great. And on the organizational side
8:29
and you guys release feather and we were laughing about that
8:31
before we hit record that, that probably would have made
8:34
Skyler's life a little bit easier.
8:37
Anything else that happened since he's been gone.
8:41
We so what we're going to talk about the whole real-time
8:46
features FedEx real-time would have definitely made scatters
8:52
live better. So yeah.
8:55
So yeah. Waited, till he was gone and
8:58
then he got me if you. So we're working on kind of I'm
9:05
I'll use a bit of a slant here like post Quasar world right
9:09
now. So, okay, that sounds very
9:12
attractive to me. Yes, that's that probably could
9:16
be could be topic for another kind of thing.
9:19
Yeah. Well, let's talk about this real
9:22
time stuff because this is One thing that I have to, I got a
9:28
little beef with LinkedIn when I try and tag people.
9:32
It has a really hard time, recommending who I should be
9:35
tagging, even though I'm replying to a LinkedIn, like,
9:40
comment and and I'm trying to add the person who wrote the
9:45
post but then it doesn't come automatically pop up with that
9:49
person. Even though I write their whole
9:50
name, well, it's going on. Why is it so hard?
9:53
What are you guys dealing with behind the scenes that I don't?
9:56
Understand. I guess rebeis would know all
10:01
about it. Yes.
10:02
Ooh, let me just first set some context to these days.
10:08
As you might know, there is a lot of information that is
10:11
floating around on the internet and most of us have been relying
10:15
on search and recommender systems to sift through all this
10:20
information and to then be most relevant information to us.
10:24
And this is one of the types of information that you just
10:27
mentioned. And now in order to serve the
10:30
most relevant information to users, these recommender
10:33
systems, typically need to know the intent and preference of
10:38
that music and usually no one is going to explicitly State their
10:44
intent and preference and recommender system would need to
10:47
infer the intent and preference of the user based on the actions
10:52
that we use it as performed in the recent past.
10:54
For example, if I apply for web developer job, Jobs in San
10:59
Francisco. Then I'm probably intending to
11:01
find a job and my preference is web developer positions in San
11:07
Francisco. So intent is what I'm trying to
11:09
accomplish and my preference is the preferred way of
11:12
accomplishing. That can take another
11:15
additionally, there is a delay between when a user takes an
11:19
action and when it can be leveraged to adapt
11:22
recommendations for that reason and this is because user
11:27
activity, Is typically processed periodically into features in a
11:31
batch environment and then we will B black window systems.
11:36
So depending on the frequency of these bad jobs, they can be a
11:41
delay of anywhere between a few hours or even days between when
11:46
the member takes an action and then that action can be
11:49
leveraged in recommender systems.
11:53
So, the outcome of this is that a recommender system might not
11:57
be able to Of the most relevant recommendations to you based on
12:01
your current intent and preferences.
12:05
So, for example, let's say that in the past, I've always applied
12:10
for web Developer jobs in San Francisco.
12:13
But today I might start searching for jobs in New York
12:19
and this heavily sorry, or in the lake.
12:24
Yes, yes. And this came to reflect a
12:27
change in my preference, but until this change is detected by
12:32
the recommender systems, I might not.
12:34
The most relevant recommendations, for example,
12:37
content recommend might still continue to recommend me content
12:41
about say that they will job market in San Francisco instead
12:45
of New York. So, the goal, the ultimate goal
12:49
is to reduce the delay between when user takes an action.
12:54
And when it can be leveraged recommender systems to adapt
12:58
recommendations, and we know that if this delay is small,
13:02
then the recommendations for the user can actually We be adapted
13:05
in real time. So I want to preface something.
13:12
Sasha's been looking at your LinkedIn.
13:14
Skyler, you looking for a job in La, he knows what's going on
13:17
behind the scenes, Skyler hasn't told anybody and Sasha's like,
13:21
oh, you're looking for a job like, you know, leg but but one
13:29
thing I realized is that I may have come out a little hot on
13:32
you rupesh. I did not properly state that I
13:35
am a power user and lover of And have been for the last like
13:40
three years. So anybody that doesn't know
13:44
would think that I am trying to take some shots at you.
13:47
I just love LinkedIn so much and I see where there's those little
13:51
tweaks that I sometimes. I'm like ah why doesn't this
13:55
work? Like I wish it would, but I know
13:59
Skylar wanted to jump in with something too.
14:01
Yeah, yeah. Now that you drop that in, I'm
14:03
sure PR is going to approve this Demetrius but Yeah.
14:11
Yeah. I think she's back is most
14:13
available from the, you know, users with experience.
14:16
So it's it's super valuable their feedback from people yeah.
14:21
Sorry go on yeah yeah but yeah the request she gave a lot of
14:26
great information about how more recent information can be useful
14:32
sort of models. But you know something I'm very
14:34
interested in diving into a little bit more detail around is
14:36
like how do you actually what is like the technical details?
14:40
Behind how you actually like, get that more recent information
14:44
integrated into these models. Yes, so, there are several ways
14:49
of doing this, but the most common approach, is to compute
14:54
teachers from the recent actions of the user, and then use these
14:58
features in a machine learning model, and they can be several
15:02
ways again for computing features from recent actions of
15:05
the user. But he most common approach is
15:09
to summarize. The information that is
15:12
contained within these actions that abuser has taken in the
15:15
past into and number or a vector of numbers.
15:19
So a number could be like a flower or a double feature.
15:22
And a vector of numbers could be an embedding type of features so
15:28
we give you more concrete example.
15:29
Let's say, we look at all the job.
15:31
Apply actions that are performed by user in the past, 24 hours,
15:37
and then we can get the embedding which is Vector that
15:41
represents a job for each of these jobs that we use it as
15:45
applied to and then Then just compute an average of all the
15:49
embeddings that we have obtained for these jobs and this feature
15:54
which is a vector of numbers is nothing.
15:58
But a representation of the kind of jobs that this user has
16:01
Preferred in the last 24 hours. And as another example of
16:06
future, could be as simple as the number of times.
16:10
This user has applied to jobs in San Francisco.
16:13
In the last 24 hours or the number of times, this user is
16:16
applied to jobs in New York in the last 24 hours.
16:19
And is actually a blog post on this that we have co-authored
16:23
with the Malik Benjamin B, who is an expert in the job
16:26
recommender happening then? Yeah, it goes into a lot more
16:30
detail about how exactly we take the piece.
16:33
And activity of the user and process it into teachers, give
16:37
it, can I put this in the show notes?
16:40
They don't like that. Yeah, thanks.
16:42
Yeah, have a great. Yeah.
16:43
So, you know, that's a lot of great detail on sort of how
16:48
these work and I think something that's like, maybe might be
16:52
implicit, but I'm curious to dig into a little bit more detail.
16:54
You talk about sort of taking the various like actions that
16:58
somebody might take and somehow summarizing it into number
17:00
Vector, Etc, is Particularly to deal with the sparseness of
17:06
actions or do you think that the action space is like, like it
17:11
how useful is it to kind of, like summarize it with an
17:15
embedding Vector vs just like counting?
17:17
How many times the action occurred like?
17:19
What is the value that they embeddings bringing?
17:22
Yes. Oh ideally.
17:24
We would like to take an action as an actor with an objective.
17:29
For example, I applied for job X or you applied.
17:33
For job, why? And then look at all these
17:38
tuples of actor, verb objects that have occurred in the past
17:42
24 hours, and then try to use these as feature somehow in a
17:47
model. But using these directly the
17:49
model might not be that trivial. So typically we would need to do
17:54
some sort of summarization operation in order to condense
17:57
this information into a few numbers so that it's easy for
18:01
the module. You will learn the Be that needs
18:05
to be assigned to the information that is encoded in
18:08
these recent actions. So and of course, Vector
18:13
representation is going to be better than a numeric
18:15
representation just because more information can be included in
18:20
the website. Complete will number My name's
18:26
Alvin structure and I'm head of machine learning engineering at
18:29
origami energy here in the UK. And I'm bothering you today to
18:33
recommend. You subscribe to the ml Ops
18:35
Community podcast to keep up-to-date with a lot of
18:39
brilliant, goings-on, within the community get Goods.
18:42
Walk, some of the greatest Minds across the amount of space are
18:46
up to So there is one thing that I think we should probably
18:55
explain or parse out to as we start to get into the idea of
18:59
real time. Real time has a lot of different
19:04
meanings for people. I've even heard some people say
19:06
like real real time or things like that.
19:11
What exactly do you mean Sasha, when you're talking about real
19:15
time? What does that look like for
19:16
you? Right?
19:17
So yeah, you're right. There's the real time.
19:21
Times. And there's, there's a real-time
19:23
inference and there's a lot of other things.
19:27
So here, in particular in this discussion, while talking about
19:33
real-time or near real-time generation of feature values.
19:39
So what what we passed was talking about, when we are
19:44
aggregating, the, the events of the, the users that have the
19:48
done in the recent recent time, and And do do it.
19:53
A new neural real time. So near real time approximately
19:57
can think about like seconds to minutes.
19:59
That's, that's the the time range that of the events were
20:05
talking about. So so this is usually happens in
20:09
swimming systems somewhere. This is quite different from
20:16
real-time inference, which is which talks about milliseconds
20:20
and That's not going to be topic from different discussion or
20:23
real-time systems like Robotics and self-driving cars,
20:27
definitely. So yeah so we are.
20:29
We're talking about real-time Futures here.
20:32
Hmm. Yeah, time teacher generation.
20:36
Yeah, yeah, that's great. Yeah yeah.
20:38
Thanks for the clarification. Good called Demetrius.
20:42
You know, one thing I'm curious about is from everything.
20:45
We've talked about. It sounds like there is a lot of
20:47
complexity, maybe until like, actually supporting these kinds
20:51
of real-time features. Just curious like, are these
20:54
always worth that effort? Are these always helpful?
20:58
So I am, of course, tempted to say yes and in general, that is
21:03
the answer to your question but the reality is that we should
21:09
consider adding these. Real-time features do only those
21:12
recommender systems that are already well optimized.
21:16
In other words, we should first focus on are wasting the
21:19
low-hanging fruit, like, optimizing the model structure
21:23
or optimizing. These set of batch computer
21:25
features before we start considering I mean, these are
21:28
good thing features and one thing that I would like to point
21:34
out is just like just like any other feature in the morning not
21:39
every single real-time feature is going to be useful and since
21:43
it can be non tribute to a creative teachers, we typically
21:47
need to have some sort of a framework for deciding what
21:51
real-time features to act and from our experience battling
21:56
than we have seen that real time.
21:58
Versions of important batch. Computed features are usually
22:02
useful. For instance, let's say that you
22:06
have a recommender system, that is currently leveraging batch,
22:10
computed features in the morning for recommending jobs and let's
22:15
say that we already know that. The average, the average
22:19
embedding of all the jobs that we use it as applied to in the
22:24
previous day is let's say the most are one of the most
22:28
important. He teaches of the morning then
22:31
we should try consider adding the real-time version of this
22:37
feature which would be the average embedding of all the
22:40
jobs that the user is applied to in the current date or the
22:43
current hour or be current minute as a teacher.
22:47
And also, by the way, I would like to mention that these
22:51
real-time features should always complement the existing batch
22:55
computer features they should not replace them.
22:58
For example we can get the big embedding of all the jobs that
23:02
we use it applied to in the current day from our wheels and
23:06
teacher pipeline but we should still try to get the applicant
23:10
beginning of all the jobs that the user applied with the
23:13
previous day through our batch pipe length.
23:17
Or previous month? Yes.
23:20
I would love to unpack that a little bit.
23:22
You know, that's interesting. So like can you just dive into
23:25
like why is it? Why is that important?
23:28
So if I know like hey these are the jobs been applying to today
23:32
was important for the models to understand these are jobs.
23:34
You apply to over the last month.
23:36
So, typically like we need to achieve a good balance between
23:41
the long-term intent of the user and the short term in terms of
23:45
using jobs, is probably not the right example, but let me talk
23:49
about the content recommended. So let's sing my long-term
23:53
intent is to be more educated about a year, but in the short
23:57
term, I might be more interested in knowing about Sakura lights
24:00
or some vaccine purple awareness.
24:02
So, in that case, the recommender systems need to know
24:05
what my short-term Kent is, which is gain information about
24:09
coronavirus, but also balance that with my long-term intent of
24:14
gaining more information about AI.
24:16
So, if the recommender system is equal to differentiate my
24:20
long-term intent and my short-term intent, then it's
24:24
able to strike a balance between these two and x. + 1, we you
24:29
identify my short-term and long-term intent and be able to
24:33
discuss differentiate or discriminate between these two
24:35
is play. Of different sets of features
24:38
that represent my short-term and long-term intense.
24:41
Honey, there's also this thing that users with more history,
24:47
like Bobby users, like dementias, right?
24:50
So, they could be, they could be good information about their
24:57
longtime intent but for more fresh users, there's no such
25:02
information. So all you have is like what
25:04
they did right now and then when they come here for the site to
25:08
check it out. And, you know, just Just just
25:12
trying to see what's what's going on.
25:16
We, if we are able to provide them useful information at that
25:20
moment that is that is very very important to to to keep them,
25:28
you know, using continue using this surf when I got so many
25:33
questions that I know will never make it through PR on the system
25:38
that you use for content. But I'm gonna ask him anyway.
25:45
We hear stories and I think one, one thing that I like to see on
25:49
LinkedIn are stories of different companies that have
25:53
made that transition. I think we mentioned before we
25:56
started recording instacart, put out an incredible blog post on
26:01
how they did that. What for you all?
26:05
Why is it that it is hard to do these real-time features, maybe
26:10
we can go a little bit into history.
26:12
So when I joined LinkedIn in 2020.
26:17
Few few weeks before before call it.
26:21
So it was one of the one of the things I want to work on
26:26
real-time features. So you both mentioned that to me
26:31
and I looked into it and it wasn't really clear what was the
26:36
impact of that and the our ability to To figure out the
26:44
impact from that was wasn't wasn't clear.
26:46
And it was on the other hand, we had some some examples of
26:52
features working in this, and we knew that that was hard.
26:55
So basically this like price performance or return on
26:59
investment wasn't clear. But then we have few
27:02
conversations with repairs and we we found some some
27:09
interesting use cases that we wanted to look at Cat and it
27:12
started looking more interesting.
27:14
Like, the, the fact that, that, that we could, we can work in
27:18
this direction and that's how we we got going.
27:21
So, so even, even then it was, it wasn't really clear, should
27:24
we do it all? Should we do?
27:26
And all should we go through this whole whole whole process
27:31
of doing it and then, when we started we, we did it in a kind
27:37
of gradual manner. So, we knew some things that
27:41
That were previously before. So there was Sansa before so I'm
27:46
sorry is streaming processor, that was created LinkedIn.
27:52
Some 2015, maybe maybe even before that by, by the people
27:59
who created Kafka and and the were there was some processors
28:05
that people really did for real-time feature and they had
28:09
some excess. But on the other hand, it was
28:12
really hard for machine. Learning Engineers to run those.
28:18
Those processors. The we can we go in?
28:21
Two more detail that but but it was it was far so.
28:24
So we try to figure out a way. How do we kind of reach this
28:28
balance between? How hard is it?
28:31
And what's the impact? And we figure it out.
28:35
One of the ways to do that via VIA Pinot.
28:40
So, Pinot is yet another system developed at LinkedIn.
28:46
That's That's a low latency analytic system.
28:52
So basically, you collect events into, some sort of a sort, some
28:55
sort of index. And then there's a system that
29:00
is able to aggregate those events and run certain queries
29:03
on that and produce that and give results.
29:07
So, for example, when the match is is on the website looking for
29:12
like I've used to his content or views days profile.
29:16
So there are graphs So this graphs are powered by by Pino
29:22
and there are real time so meaning that there's real
29:26
information that that is behind and and this information is
29:30
created on the slide you know like caching or pre pre pre
29:34
compilation something. So that system was there and we
29:40
we looked into all of what what is able what we will be able to
29:46
do this? Just like what What features
29:48
will be able to do with the system and we started with the
29:52
several use cases and one of those use cases kind of went
29:58
into production and had some some results that's there.
30:02
Was there was job in barracks, right?
30:04
Right. Okay.
30:06
Yeah, yeah. So so so that kind of gave the
30:13
whole like the the fact that there was a door result, there
30:19
were those business impact, the give people confidence that we
30:22
should continue going this direction and that's how we were
30:28
able to continue in, like a bigger effort that that we call
30:34
now. Yeah, Eternal name is Felix real
30:40
time, but I guess that would be called feather real time right
30:43
now. So we, we started this effort to
30:49
to create a real-time processors based on some stuff.
30:56
So yeah. Sorry to interrupt, I'm going to
31:00
sound really ignorant right now but I had no idea a Kafka came.
31:06
From LinkedIn and be, if you note came from LinkedIn, which
31:11
are we did a whole system design, review on Pinterest and
31:16
how they're using to note, and it's so cool to see that.
31:19
It's so, it's just like, mind-blowing to hear about that.
31:24
It's like I just found out that Santa Claus is real and so
31:30
sorry. Go ahead.
31:31
Sasha, I cut you off, man. Keep going though.
31:33
All right, no way. So yeah, I think I've got was
31:36
created. At LinkedIn and confluent was
31:40
started by Jack crabs and two other people from the cupcake in
31:47
for thinking. And yeah, that's, that's that's
31:52
a huge part of infrastructure everywhere.
31:56
And Pinot is actually, also the, the recently started started
32:01
company Kishore, Goldberg Krishna, who was working at
32:04
LinkedIn? Is he started a company called
32:06
Archery and they're doing like that turning this into into good
32:11
business. So that's, that's also, I think
32:15
so. Yeah.
32:16
So, so hard. So, so we had some success with
32:21
with peanut based solution but it has some limitations as well,
32:28
limitations were mainly in the model of like how much data you
32:31
need your personal Target? So, for example, if if you are
32:36
interested in a particular member, Actions of particular
32:40
member that would be maybe dozens of at most hundreds, I
32:45
don't know. In your case the message is
32:46
probably thousands but but for most people is less it is it is
32:54
quite possible to kind of organize it then partition them
32:58
in the way that access and aggregation of this data will be
33:04
fast enough. However, if we are trying to do
33:07
this for Things like jobs or content.
33:11
Then the amount of events that we need to aggregate is is much
33:16
much bigger so it's thousands. Tens of thousands if not
33:19
Millions, right? In this case P no-shows.
33:25
So we do something else and for that we need to go back to like
33:31
pre-computing idea and pre-computing idea.
33:34
Is you run sounds, the processor which Processes event as they
33:40
come and he knows for each event.
33:42
For example, that's, that's the event that Demetrius view this
33:46
particular piece of content, right?
33:48
So it looks at that and says, okay, what are the features that
33:52
in are impacted this? So it has in our situation, it
33:58
has processors running for each one of those features.
34:01
So they pick up this event and recompute this feature based on
34:05
this new information, put this into the The store and the store
34:10
is available for for people, to, for machine learning models, to
34:18
retrieve this information. So, so that's that's how the
34:21
pre-computation works. So it it is, it has wider
34:26
application, right? You, you can apply it and wider,
34:31
spectrum of users. However, there's much, much more
34:36
work. You need to do up front.
34:37
You need to know exactly. Clear all the all this run, all
34:41
these processors, you would say dozens dozens if not hundreds of
34:46
processors for each each feature, right?
34:49
And do a lot of computation and this processors, all of them are
34:53
like a distributed systems with its own little.
34:57
You know problems some containers would fail some
35:02
Hardware would fail or other problems that happened with this
35:08
system. And and it is it is very
35:12
noticeable practically. We know that it's not trivial to
35:16
support system like that. So you need a special kind of
35:23
expertise of in for people to be able to support and run the
35:27
speed people system. So it is it is really
35:30
challenging for people like machine learning Engineers to
35:35
run this. So that's that was a big gap.
35:38
And That's, that's one of the answers to your questions.
35:41
Why is that? Why is it so hard?
35:47
Part of the reason is that, I machine learning engineer, I
35:50
don't care about your infra. Well, I do care about, but I
35:53
don't know that much about the infra and I don't want to spend
35:57
my time. I, you know, I my work is to
36:00
make models. Why do you do want me to respond
36:04
to, to this own called events in the middle of the night?
36:08
So, So, so this is this part of a lot of difficulty challenges
36:15
that are there and it took us a significant time to figure out
36:24
that that a we need. We need to solve this Challenge
36:26
and be how to solve that so we can we can go into more detail
36:30
on that. Awesome.
36:32
Yeah. Yeah I think you know definitely
36:35
great summary of a lot of the challenges of like building real
36:38
time. Teachers and, you know, we kind
36:40
of dug into this a little bit with rupesh on.
36:44
Hey, are these always useful curious to hear about from your
36:48
own experience? Like, what were the successes of
36:50
adding real-time features? You know?
36:52
Maybe some of the examples of where they worked?
36:54
Well, we have actually had pretty good success for the you
36:58
think teaches in central of the recommender systems and I'm
37:03
going to share three examples are soaking our jobs that
37:07
Commander We saw point six percent increase in job applies
37:12
and a 20% reduction in dismissals of a job
37:16
recommendations and 20% the point.
37:21
Six might seem like a small number, but it actually
37:23
translates to a very large. Absolute number, if you look at
37:26
the total amount of job applications that have them on
37:29
LinkedIn on a daily basis, in our D recommended we saw a point
37:35
three percent increase in contributors.
37:38
These are members who like comment share or post in the
37:41
field and we also saw a point four percent increase in revenue
37:46
from sponsored updates in the feet.
37:49
And lastly, never search type of had recommended, which
37:53
recommends, to you autocomplete suggestions.
37:56
We saw a point three percent increase in click-through rate,
38:00
but in general, we observe that the new and less active members
38:05
benefited most from these really And peaches and this was
38:10
expected because these are the users for whom we have the least
38:16
amount of historical activity data.
38:18
If I just signed up on LinkedIn and started taking some actions
38:22
on lignin, there is no historical data.
38:24
So all that the recommender system can use is the activity
38:28
that has been observed within this session.
38:32
To be able to personalize your recommendations will be session.
38:37
So in this type, I had example, the mirror is brought in the
38:41
beginning, we we saw some, some improvements, maybe not now but
38:47
I guess we can work on more dude.
38:50
Well, you got me to start posting more.
38:53
I don't know if it was right around the 20, 20 period, it's
38:58
funny because I was in sales and at some point LinkedIn became a
39:05
way for me to get meetings. So I started I will say it like
39:10
it is I started spamming people on LinkedIn and tried to book
39:14
meetings to get them to check out the tool that I was selling.
39:18
But at some point someone told me they said, you know it's
39:22
actually better if you just post stuff on LinkedIn and then
39:25
people will come to you. And so that was a revelation and
39:30
I started doing that and I've never looked back since luckily,
39:35
I don't really have anything to sell now.
39:37
So I can't spam people even if I wanted to, but going back into
39:41
it and Looking at these, this idea of of these real-time
39:48
features like an in this domain. Are there any other learnings
39:53
that you've had? Maybe Sasha, there's something
39:55
that you want to mention like with this with these
39:59
experiences. Yeah.
40:01
So so one thing I wanted to say once again, we talked about a
40:08
little bit real-time features are kind of Terms of like muscle
40:16
muscle pyramid of needs, there are kind of up there.
40:20
So our best talked about this like lets you need to get first
40:26
Basics, right? Like a data flowing into the
40:29
system and the Machine learnings model working and all that.
40:34
So after that when you when you at the point where you want to
40:40
optimize things that that's where a real-time features.
40:44
Become very helpful and they are very, very useful, but they come
40:50
at a cost and you need to definitely need to take this
40:57
cost and you can't, when you go go for this development.
41:03
So that's that's one thing. Another thing that I kind of
41:11
noticed when we started working on this hole, Real-time, there's
41:17
there's this recurring team of like how do we have this?
41:25
This machine Learning Systems system and there are many people
41:30
who contribute to the system. So there are many hands in this
41:34
in this pot. And how do we make it all work?
41:41
Well, without people. The within all this pot with
41:46
every Everybody kind of doing their part of the of the work.
41:51
So this is this is kind of this really important.
41:54
We mentioned about like machine learning Engineers that care
41:59
about their their part of the saying.
42:01
And then there are application Engineers who care about that.
42:05
They're part of the thing and there are influential nurse who
42:07
care about their part of the thing, and they all together
42:10
kind of tied together into into this machine learning Pipelines.
42:14
Or inference pipelines. So this is, this is a very kind
42:19
of serious serious. An interesting problem that that
42:25
you need to solve in implementing those machine
42:30
Learning System. So one little example that we
42:36
were working on the real-time features like streaming
42:42
processor for a real-time features and And it was the, the
42:48
maintenance problem or support maintenance problem that were
42:52
talking about. So our assuming team came up
42:56
with this idea of managed service.
42:59
So there is the building manager service, so that machine
43:04
learning Engineers can run their feature pipelines, very, very
43:08
easy. And it is managed by why?
43:13
In fact, even the back end so This this kind of separation is
43:17
is a very important. They also they talked about this
43:21
in one of the meet up so we can, we can put this in the show
43:24
notes but that's a, that's an interesting development and in
43:29
general separation. And lastly, I bla machine
43:33
learning. As coming from software
43:35
engineering background, I love machine learning.
43:39
Because when you start working on it, it gets you out of your
43:43
comfort zone. Very quickly.
43:45
So basically whatever are your assumptions you you you will
43:50
likely need to reconsider them. So for example at LinkedIn,
43:55
LinkedIn is historically more lean towards Java so so like all
44:00
the applications are done in Java.
44:04
And so we're essentially Java shop what people say Java shop,
44:09
but with the introduction of machine learning and the need To
44:14
make a good environment for machine gun Engineers.
44:17
We were what trying to, to make to make this environment more,
44:25
make them more efficient in their room, and then we
44:28
absolutely need to use Python. So we we get to like, two
44:33
systems where, you know, machine learning Engineers think the the
44:37
program in Python, but actually that translates to system the
44:41
run and Java. So whatever is the solution
44:44
Jean, we need to think about it. So that's, that's one example of
44:50
of this challenges that the happened and in machine
44:56
learning. And yeah, it's amazing field.
45:02
I'm really excited to work in it.
45:06
Yeah. I just want to add one small
45:07
note as a machine-learning engineer.
45:12
Typically systems that are easy to Use are well-adapted.
45:17
So for instance, a special mention that there is always
45:22
this separation of the boundaries between the word that
45:27
an engineer has to do with an absolute train engineer has to
45:30
do, typically, if you ask in AI engineer or ml engineer to grow
45:37
into the inside domain, and Try to learn and make some changes
45:42
there. There is some resistance, but if
45:44
you make it very easy to say confetti for them to get access
45:51
to these beautiful features, then the adoption rate is pretty
45:54
high. Yeah, yeah, it's definitely
45:57
definitely a great point and it's cool to hear about all the
46:00
challenges that were solved. It seems like if I find myself
46:05
at LinkedIn in the future, I'll have much less frustration.
46:10
But you know, one of the things that you mentioned sort of off
46:15
and Sasha and I was very interested in.
46:18
You talked a little bit about branching, you know, that
46:20
LinkedIn is following a trunk base development and it sounds
46:24
like there was some way that you can up supporting like,
46:29
branching for machine learning loved.
46:30
I would love to hear a little bit more about that.
46:33
Yeah. So so that that is that came
46:37
from conversation with. Another person who me no Alex
46:43
battery. So he was, he was.
46:47
Yeah, he had a vote here. Yeah, yeah.
46:50
So so he was really thinking about how to improve
47:00
productivity of ml engineer. So what they were doing given
47:05
that we're working in track, they were making copies of this.
47:09
Same kind of model code in different directories all under
47:14
the same trunk and and using that for for experimentation so
47:20
different different kind of variance of the model.
47:24
And that wasn't that had multiple problems.
47:28
So, we worked with, with our tools team and just talk to
47:33
them, and it felt felt out of place.
47:36
So, it felt kind of a little bit hacking.
47:39
This this process so we thought can we can we do do any better
47:44
than that. And then the guys is so salty
47:50
toll steam like more thought about the way to use, branching,
47:58
use different branches. So what happens is that you you
48:02
have different experiments on the same model running as a
48:06
parallel branches and you are actually able to to deploy from
48:12
different branches at the same time and then and run this this
48:17
deployment as an experiment and then when you are done
48:22
experiment a you you see the model that works the best and
48:30
the winner. You just kind of merge to the to
48:33
the main track. So that that is that is a very,
48:36
very interesting thing. I I think It is it could be
48:41
interesting. You know, the topic for the
48:43
conversation I can get in touch with the people who do that.
48:47
It's a very interesting development and I think it
48:50
deserves more more people to know about this and think about
48:58
the play applying this in the airport.
49:02
All right guys, so we've been talking about it a little bit
49:05
real. Time is very popular.
49:08
It's a hot topic. These days.
49:10
What are some things that you feel like do not get talked
49:14
about enough when the discussion of real-time comes up?
49:19
I would say, the ease of use typically does not get
49:23
discouraged as much. And I mentioned earlier, as any
49:27
engineer, I want to use a system that is that is easy to use.
49:34
I don't another, I would not prefer to work leverage, a
49:39
system, which has its own intricacies and complexities.
49:47
So I think that is something Which as a community, we need to
49:52
lay an emphasis on and make sure that whatever system we is easy
49:57
to use. Sage advice.
50:03
Yeah my personal pet peeve is portability.
50:08
So in particular portability between streaming and batch so
50:15
when you define a feature and you the same definition can work
50:23
in and streaming mode and in in a batch mode.
50:30
So if you Don't have this or regardless.
50:35
Actually we you when you when you have that feature like
50:40
features that you use in training and you features that
50:45
that you run in and inference you need to have a consistency
50:50
between this two. Otherwise you know, what are you
50:53
doing? Otherwise, your machine learning
50:54
is actually not, not done, right?
50:57
So how do you achieve this? Portability is Is a big question
51:04
people. So I they're not yet like very
51:10
good solution, so that people still working on this.
51:14
There, there are some some problems with that, but but
51:19
there is, there's more to be done.
51:23
So we this problem that was wasn't solved yet.
51:28
So, tell me straight rupesh, does it matter if I, I put the
51:31
link in the comments will more people see it if I put the link
51:34
in the comments, that's all I want to know.
51:35
This has been great, guys. Thank you so much for coming on
51:55
here and talking to us about this.
51:56
Like we mentioned a few times is such a Hot Topic and it's so
52:01
cool to see. See how you all have gone
52:03
through the transition. And I appreciate you sharing
52:05
that with us. Thank you for having us.
52:01
See how you all have gone through the transition.
52:04
And I appreciate you sharing that with us.
52:07
Thank you for having us.