Latest / Agentic Conversations (formally mlops.community) / RecSys at Spotify // Sanket Gupta // #232

Transcript

RecSys at Spotify // Sanket Gupta // #232 · Agentic Conversations (formally mlops.community)

0:00
The higher the hi guys. I'm Sankey.
0:01
I work as senior machine learning Engineer Spotify and
0:05
yeah, when interesting coffee routine, So I love lattice.
0:08
So I used to be hitting all these different coffee shops in
0:11
New York City trying to find the best one.
0:13
But then I'm like after Pandemic, I was like, this is
0:16
just not convenient to do. So I got this coffee machine
0:20
that I brew my own espresso, grind my own beans and pull the
0:24
short and stuff. So it's it's a lot of fun.
0:27
I've been learning a lot about, like espressos, and there's a
0:31
lot of connoisseurs out there. But yeah, learning a lot on that
0:35
fun stuff. Welcome back to the Emblems
0:38
Community Podcast. I'm your host, Dimitrios, and
0:40
today is a very special one for me because I am such a deep and
0:45
dedicated Spotify user. We get to talk with Sankey all
0:48
about the recommendation systems within Spotify and how they've
0:53
built that out so that various teams can leverage the embedding
0:58
store that they have. I was fascinated by the
1:01
different problems and challenges he talked about,
1:04
especially when it comes to evaluating and re ranking and
1:09
what their system looks like. There was some cool takeaways
1:12
that I had from here. He broke down what the
1:15
difference is between the vector store and the feature store and
1:17
what kind of requirements are on each of these.
1:20
He also talked to me about how me as a person who is using
1:25
Spotify has an identity or I have a vector number attached to
1:30
me and how they're constantly updating that.
1:33
And for him, one thing that is very important is how they can
1:37
continuously update that and make sure that the embedding and
1:42
the place that I live in vector space is the freshest and most U
1:48
to date as possible without blowing up the system.
1:50
Because you can imagine, he said on here, there's 600 million
1:54
monthly active users on Spotify. If you're constantly fine tuning
1:58
these embeddings and you are churning them out for each user,
2:04
you really have to think about how much is too much.
2:08
And so he broke down those problems.
2:10
I had a wonderful time talking with my man Sankey, and it turns
2:15
out that he's really into coffee like I am.
2:18
Let's get into it. And if you haven't got your
2:21
tickets to the conference, I would get it now because we're
2:23
almost selling out. All right, man, let's start with
2:31
this. You just told me a viciously hot
2:35
take that I I cannot let go too far from my brain.
2:38
Before we get started, Rexis or Rags, break that down.
2:43
Why do you think that? Got it.
2:45
OK, sure. Let's let's look at.
2:47
We're going right into the system then.
2:50
OK, right. So the reason I think
2:52
recommender systems have a lot of parallels to Rags is because
2:57
of this thing called vector database, right.
2:59
And I mean, it's a fancy word that we are not talking about,
3:02
but these so-called embeddings and nearest neighbour lookups
3:07
have been around for a long time, actually, it's just not
3:10
been applied to text and NLP now.
3:13
But let me give an example, OK? And then we can see how it
3:17
relates back to rags. Say you're listening to a song
3:19
on Spotify and you want to know what are the 10 most similar
3:23
songs to that one, right. So let's say it's Bad Habits by
3:26
Ed Sheeran, right? So now you want to know what are
3:28
the 10 songs closest to bad habits, right.
3:31
So how a human might do it is the human might like, go into
3:36
other songs in which Sheeran, it might, like, look at pop genre
3:39
and do something like that, right.
3:40
But how would you do that at scale, right?
3:42
Like, we don't have human sitting, right?
3:45
We have algorithms. So how would you do that then?
3:47
That's where kind of vector databases come in, right?
3:50
So we have a embedding for bad habits, or vector, whatever you
3:54
want to call it, Vector or embedding, which is just a
3:56
series of numbers, and we do. Appreciate the sorry, hold on
4:01
with the embedding. This is what I've always been
4:03
curious to know is it's off of a description of the song or it's
4:09
off of like all of the metadata, like the tempo and if he's
4:13
singing and falsetto and so it's very much things that are
4:18
describing the song, or it's off of the actual song itself.
4:22
It's a song itself mostly. Mostly.
4:24
I mean there are other aspects to it, but mostly it's kind of
4:28
like the spectrogram analysis of the song.
4:30
So as you said, like the tempo, the Valance, the instrumental
4:33
Ness, the pitch, like all of those kind of features.
4:37
And then you add in kind of the collaborative aspect, which is
4:40
like which people put the songs together in a playlist, right.
4:43
So it's kind of like playlist Co occurrence.
4:45
So it's a combination of those two things which kind of builds
4:48
the embedding. It's a massive system we can get
4:50
into, but the end output is an embedding, right?
4:53
It's it's an embedding. O We've uniquely represent each
4:57
track on Spotify with an embedding.
4:59
So that embedding model is a black box for now, right?
5:02
We can go dig into that, but. But it outputs and embedding,
5:04
right? And then you load all of these
5:07
embeddings in an index, right, which we now call vector
5:11
database, right? And basically now what you do is
5:14
you do nearest neighbour lookups on that embedding, you get back
5:18
like 100 or 200 items back, right?
5:20
That's candidate generation, right?
5:22
And then you do ranking, right? And this is very similar to what
5:25
ranks do, right? Like they do.
5:26
They retrieve like different documents with metadata and so
5:30
on. And then they might like rank
5:32
them based on if it's a New York Times article, it's higher.
5:35
Like, you know, like that's the kind of sending and we do with
5:37
rags, right. And it's very similar here,
5:39
right? Like you get back like these 200
5:41
songs and you're like, OK, so which one are you likely to
5:46
listen to next, right. And and so on.
5:48
So like you kind of do this personalization and like user
5:50
understanding thing, then you rank based on a model you get
5:54
back those 10 songs basically. So that's kind of the approach
5:57
at least the retrieval augmented aspect is very much similar.
6:01
The only thing I guess missing between Rag and Rexis is the
6:04
generative aspect, right? Like you don't generate back
6:07
like a paragraph like explaining why say shivers is similar to
6:12
bad habits, right. They're sung by the same artist.
6:14
They're like, so we don't do that.
6:15
Like you just get the output basically.
6:17
So that's kind of I see how I see Rags and Rex is kind of
6:20
together. But how about when you're
6:23
evaluating the Rexus? That's a whole different kind of
6:27
worms too. Or do you feel like there are
6:28
parallels there, like if it's actually working, how you score
6:32
that, how you get ground truth, what the golden data set is?
6:36
Yeah, I think evals are a little bit same and different, right.
6:40
And I think similarities are in the sense of all those ranking
6:43
metrics like NDCG and MRR and like AUCOC depending on how you
6:49
set up the problem, right. So those metrics are like
6:51
classic like I mean all of ML uses that.
6:54
The difference is I think lie in kind of how we do text matching.
6:57
So I see in rags like they do matching of like exact texts and
7:02
like tokens and like Blues score and rules and things like that
7:05
over here like because we're not generating like we don't really
7:08
go into that. So I think that's kind of the
7:09
difference. Yeah, yeah, that makes sense.
7:13
So you've been doing a lot of stuff at Spotify with
7:17
recommender systems. You're working on a few
7:19
different ones, but is it safe to say that all the new music
7:24
that I've found recently on Spotify is in part because of
7:28
the work that you've been do? Ohh, that's a I could take some
7:32
of the credit, but not all right?
7:33
Like, there's just so many teams working on it.
7:36
So yeah, I think. I think it's a lot of amazing
7:39
people on this problem. All kind of.
7:42
Where's your special sauce? Are you working mainly on the
7:44
platform? On the models?
7:46
On the optimization. Yep, So I work as a Senior
7:50
Machine Learning Engineer in personalization.
7:52
So kind of my focus is has been quite a bit on user
7:56
understanding. So like how do we actually
7:58
understand users in a way that helps out everyone basically in
8:02
the company like in terms of the models.
8:04
So more on models, some like obviously doing a lot of mill
8:08
OPS because otherwise you can't ship anything, right?
8:11
And then evaluations. Also a lot of evaluations.
8:15
Yeah, I imagine your playing with features, you're knee deep
8:20
in features and that's probably your day in, day out.
8:24
Yeah, for sure. Any features that surprised you
8:29
that have a whole lot of weight you would have never expected?
8:33
I think one of them is say, an account each, which is very
8:37
interesting. So account age basically is like
8:39
how old is is an user, right? And we have seen interesting
8:44
patterns like you wouldn't expect like people who have been
8:46
on Spotify longer have certain kinds of patterns than the new
8:50
users, right? So there are very interesting
8:52
patterns there. Is it because of the UI features
8:55
that you feel like the account is different or the account age
9:01
makes a difference when it comes to features?
9:03
Yeah, that's a that's a fair, I mean see, so I'm just giving an
9:06
example of one model, right. So there we have like dozens of
9:09
models. So every model works
9:11
differently, right. I mean, and so on.
9:13
South features are unique to the models.
9:16
But speaking to your cushion, like, I think some of the older
9:20
users may have certain habits, right?
9:22
Some of the newer users may be getting onboarded due to
9:24
podcasts, for example. Some of the older users may have
9:27
joined because of music. So they have certain kind of
9:30
habits, which is ingrained and it's hard to change people's
9:33
habits and so on. So there are those kind of
9:36
correlations also, right? And we recently lost audio
9:39
books, so that's another thing. So all of those things play a
9:43
role, Yeah. Yeah, I've been loving the
9:45
audiobooks, by the way. So there are some really
9:48
interesting problems that you face though, right?
9:50
Like one being cold start and how you can recommend something
9:54
to somebody who is first joining Spotify, which feels a little
9:58
bit like. That's probably the minority, I
10:00
would imagine, because we've all used Spotify in one way, shape
10:04
or form by now. I would assume, but potentially
10:08
no. You have better ideas on that,
10:11
and so how do you deal with the cold start problem?
10:14
Yep. So cold start is a very
10:16
important problem, right? And there are two main reasons
10:19
why I think cold start is very difficult to solve in recess.
10:22
I think listeners who are from ecommerce would relate with what
10:24
I'm going to say, right? So one is that you don't have
10:27
any history that makes intuitive sense, like you just don't know
10:30
anything about the user, right. And the second less intuitive
10:34
reason is because of how the systems are managed, right.
10:37
So we have usually bad systems. So these systems kind of operate
10:41
maybe they infer like once a day for all of users and they're
10:45
running behind, right. Like we have like almost like
10:47
600 million plus users now. So it takes a while for us to
10:51
like churn through all of them. And there are so many
10:53
dependencies and systems. So it can take like one or two
10:57
days for systems to kind of actually catch up actually,
11:00
right. So someone new joins in.
11:02
We don't have anything about them that's bad, but you can't
11:05
do much about it. But our systems ourselves as a
11:07
person starts engaging are also slow, right?
11:10
So we take about two or three days to respond to it, right?
11:13
So there are both of these problems, which kind of make a
11:16
massive difference in cold start.
11:19
And So what do you do? Yep.
11:21
So let's solve the second problem first, right.
11:24
So which is because it's controllable by us is to do a
11:28
more like near real time systems.
11:30
So someone joins on Spotify, they start listening to
11:33
something like say 80's rock, right.
11:35
And then our systems are like in minutes they respond and they
11:39
kind of create new user taste profiles and like these new
11:43
embeddings for them. And then those embeddings can
11:46
then be used across the company right by all the models.
11:49
So that's kind of the systematic aspect which is like go from
11:53
batch to near real time or real time, right?
11:56
But then for the first problem, we have this thing called taste
11:59
onboarding where you kind of select, I don't know if you
12:01
remember the doing that, but you can select three artists or four
12:04
artists and you sign up and then you can also select languages
12:08
and so on. South, it's about kind of
12:10
smartly using that information when it's available, right?
12:14
So it's kind of a combination of both of these things.
12:18
So then you have a bit more information about someone.
12:22
You mentioned that there are a lot of models that you're
12:25
playing with. I am understanding that as
12:29
there's a lot of models in the whole recommender system.
12:33
Yes, there are a lot of models in the recommender systems,
12:36
right? So just an example, right?
12:38
Like search would have their own model, right?
12:40
And then the home is going to have a different model.
12:43
We have models that do ads, which are different models.
12:46
We have Discover Weekly which would be a different model would
12:50
have like as I said like the what are the next 10 songs?
12:54
That's more of a candidate generation thing.
12:56
That's another thing. Then we have multiple systems
12:59
and people. So it's it's a very like even as
13:02
as a as a company like there are obviously going to be a lot of
13:05
people, teams and dependencies. No talk to me about because this
13:11
is just for my own scratch, my own itch.
13:13
Yeah, when I say go to radio for any song, is that a new model
13:19
also? It's going to be a different
13:20
kind of teams managing that, but a lot of them actually tend to
13:24
use the embeddings, right? So the team I was working on is
13:28
building kind of embeddings, and most of the commonality between
13:32
all of that is the embeddings. And that's what I call as
13:35
foundational embeddings, which is the kind of the goal here is
13:39
to not have every team do everything from scratch, right?
13:43
Like the user is the same, so if it's Demetrius, like it's the
13:46
same user, we don't want teams to interpret you differently
13:50
across the whole stack, right? Like, we want teams to have a
13:53
unified and a common understanding of you, right?
13:56
So just just to give you an example, right, say you wake up
13:59
in the morning and you listen to some music and then you jump
14:02
onto a car, you might be listening to a podcast and then
14:04
you hit the gym. You're going to be listening to
14:06
some high energy music. So those are some of the
14:09
patterns that can be used across, right?
14:11
It's not just one team, right. So as a user, you want to
14:15
experience unified and a common thing and these embeddings kind
14:19
of help that. So there is this embedding layer
14:22
in between which kind of branches out across all these
14:24
models. OK, so it feels like the I'm
14:28
getting the embeddings right. Yes, is one of the most
14:31
important things. I I think so.
14:34
I think that that play a massive role especially for candidate
14:38
generation, right, because we were talking about vector
14:41
databases, right? How would you?
14:43
So we have 100 million plus items in our catalogue.
14:45
How would you take an item and bring back 100 out of 100?
14:49
Like that's a very, very tough problem for any model to do.
14:53
So you need some kind of candidate generation, right.
14:56
So that's basically the retrieval problem, and that
14:58
typically is solved by embeddings.
15:00
So one thing that does come up quite a bit even in RAG, is how
15:04
you evaluate the retrieval. And So what are you doing to
15:08
make sure that what you are retrieving is the best?
15:13
Out of those 100 million, did you say 100 million songs or 100
15:16
million songs? How many?
15:17
100 million items, right? So it's a mix of podcast, music,
15:21
audio, books, everything, right? That's a lot, right?
15:24
That's a lot of item. Yeah, kind of.
15:27
And that's why evaluations play such a major role.
15:30
So personally, as an engineer, like, I have spent more time
15:33
evaluating systems than even, like, building the model, right?
15:36
Like, I take the evaluations very seriously and it's a very
15:40
hard problem, right? It's, it's like sometimes you
15:42
need models to evaluate a model definitely.
15:47
So let me give an example. I think to answer your question,
15:49
right. So you said how would you know
15:51
those 10 songs are correct, right.
15:53
So that's one model that got the 10 right.
15:55
Now you need maybe say another model that is going to rank or
15:59
or test that, right. So you're going to see out of
16:03
the 10, which ones are you likely to play next or play in
16:06
the next 4 hours or seven days or whatever.
16:09
And we kind of take an intersection of what you would
16:13
have actually what you have been listening with that one and then
16:16
we kind of do intersections for that.
16:17
So those are the kind of things we do to kind of evaluate some
16:20
of the systems. Now you mentioned as an engineer
16:25
you've spent more time working on evaluation then even building
16:30
the models, right? What is your time working on the
16:35
evaluation system look like when I'm?
16:37
Talking about evaluations, I'm talking about evaluating
16:39
embeddings. OK.
16:40
So let me be clear there. Because there are teams that do
16:42
classification models where the valuations are going to be not
16:45
as difficult, right? So I mean, that's going to be
16:47
classic precision, recall, whatever, right?
16:49
As long as you've set up the labels correctly, you can get
16:52
kind of pretty far with just the basic metrics.
16:56
So I'm talking about these embeddings, right?
16:59
And in the LLM land, those embeddings may be coming from
17:02
say text embedding 3 large model from open air, whatever, where
17:05
you arse in a text and you get back embedding.
17:07
Right. So those are the embeddings I'm
17:08
talking about. And these are very like they're
17:12
very abstract, you can't look at it like no human can look at an
17:15
embedding and say like OK, that's good or bad.
17:17
So basically, right, Like you can't do it right?
17:20
So now you need, yeah. Computer language.
17:22
It's a computer. Language at least numbers and
17:24
these numbers kind of have these non-linear relationships with
17:26
each other, right. So in terms of evaluations we
17:30
kind of break it down into more like sub facets, right.
17:34
So one is intrinsic evaluations, right?
17:36
So we take these embeddings in sitting in the vector space and
17:41
we say like OK so we have a demo three of embedding right?
17:45
Like what are the 50 users similar to Demetrios right.
17:49
So we do the nearest neighbour lookup on that embedding.
17:52
We get back like 50 users. So it may be say me 49 other
17:56
people and then we have these some kind of heuristic system
17:59
which is going to be like OK, Dimitrius likes podcasts, mostly
18:03
podcasts and it's user in US as well.
18:06
Have you been looking at my embedding?
18:08
No. Just good.
18:09
We can't. We can't really look at anyone.
18:11
But I'm just too. Easy to understand I'm.
18:14
Just guessing, just just a guess, but based on those
18:17
heuristics and matching, we're able to then say, OK, I think we
18:20
are embedding is not bad, right? It's able to catch like 30% of
18:24
what we thought Demetrius is similar, users look like.
18:27
So these are the kind of like intrinsic evils I'm talking
18:30
about. Then we'd also do extrinsic
18:32
evals, which would take the embeddings into other models.
18:35
So you build these mock models, which kind of mock some aspect
18:40
of Spotify, right. So as I talked about predicting
18:44
future lessons, so you would actually what you would do is
18:47
you take Dimitrios embedding and then you put it into a system
18:50
and say like what would Dimitris listen to in the next seven
18:53
days, right. So and then you kind of see the
18:56
actual listening and you kind of do overlaps of that.
18:59
So you're kind of like really figuring out if embedding is
19:01
actually good, is even doing his job bright and like that.
19:05
Like you build these basically mock downstream models and you
19:10
try to like evaluate these embeddings by passing through
19:13
those models. So you make you convert
19:15
evaluating embeddings into a classification problem.
19:18
Essentially, yeah, it's kind of what.
19:20
I'm saying, so you're trying to predict and you don't actually
19:23
do anything. You're just putting that side by
19:26
side with my real actions and you're seeing how well you were
19:30
able to predict that. Yep.
19:33
If that makes sense, right? Because embedding itself, you
19:35
can't. As I said, it's hard, right?
19:36
But then you can feed it into these systems, right?
19:40
And these systems are now classification or regression
19:42
systems, which we know from all our love with classical mill,
19:46
right. Like those are they're classic
19:49
techniques. You can easily evaluate them
19:51
through Roc, precision, recall curves and so on.
19:54
And then you when you feed them into, you can evaluate the
19:57
downstream systems. So you're not evaluating the
19:59
embeddings yet, you just evaluating the downstream
20:01
systems and if they improve, you can basically like say
20:06
embeddings are doing better. So it's a two stage process,
20:10
which is what makes it an incredible problem, right?
20:13
And and it's very hard to like tell people like, OK, so you're
20:17
now like building embeddings and you have to evaluate with these
20:19
like mock models which are there.
20:21
So yeah, there's a lot of kind of lot going on there.
20:25
And how are you able to test this stuff out easily?
20:30
It feels like there's a lot of infrastructure in place to be
20:33
able to support that O we. Have internal platform teams
20:37
that kind of help with a number of things.
20:40
And yeah, so it's a whole stack of distributed systems like
20:45
cloud data flow, big table, we use queue flow.
20:49
So all of those kind of things kind of play a role there.
20:52
So a lot of the engineers time obviously is also not just in
20:54
coming up with an idea, but also productizing it in their
20:58
systems. Was it?
21:00
Didn't one of these pipeline tools come out of Spotify way
21:05
back when? Was it Luigi or something?
21:08
Is that still in use? Yep.
21:10
My team actually used it, But yeah, I know a lot of teams are
21:13
now maybe moving to more modern tools, but yeah, Luigi is love
21:17
Luigi, yeah. Really.
21:19
Ohh, that's awesome. Ohh, that's great.
21:22
And so it does feel like there are other ways to do this.
21:28
What are different approaches if you don't use this?
21:30
And correct me if I'm wrong, the style that you would call this
21:35
is like transfer learning. Yeah.
21:38
So the transfer learning exactly.
21:40
Again, let me take an analogy of LLM.
21:42
So I think that will help here. So Open Air comes up with a new
21:46
embedding model, right? And then they create these
21:48
foundational embeddings. But then you work in a finance
21:51
company or say like you are in a law firm and you are analysing
21:55
like a lot of these law techs or like tax, tax docs or whatever.
22:01
So you might want to fine tune them on your data, right?
22:04
So that's basically transfer learning, right?
22:06
So you're kind of tuning the underlying model, you're not
22:08
changing it. And that's kind of the same
22:10
approach we take here where we build kind of these foundational
22:13
embeddings. So our team would build like
22:15
these representation learning based models to build embeddings
22:18
and then all of the downstream teams would on their own task
22:22
like tune them. Essentially what you were
22:26
talking about, personalised DJ. Like they would maybe take the
22:28
embedding and then they would tune it in the voice arena,
22:32
voice setting like what works, what doesn't work?
22:35
Search team would then instead look at like search features.
22:38
So what are you likely to search?
22:40
And they would still use the same embedding, they're just
22:41
tuning it in their own models basically.
22:44
That's kind of the approach of transfer learning.
22:46
Are there ways that you think this could also work at Spotify
22:50
without using transfer learning? Everything has an alternate,
22:54
right? Like something used to be done,
22:56
right. So now like, yeah, so the
22:59
alternate is don't use embeddings, just kind of build
23:01
everything from scratch, right? So we look at last end days of
23:05
history of a user, we look at what devices the user uses, the
23:08
country, the listening like. So you every team starts to do
23:12
everything from scratch and you don't use any embeddings, you
23:16
don't fine tune anything. So again, analogy to ALMS is the
23:20
financial company builds their own LLM from scratch like they
23:24
don't touch open a I, they don't touch LAMP or whatever, they
23:26
just build the whole encoder, decoder transformer like it's
23:30
possible, right? Like it's not impossible, I
23:32
mean, you can still do it, but the amount of difficulty you
23:35
might need to put in like 50 engineers for like 6 months to
23:38
build that, right? So.
23:40
Expensive. Yeah, it's.
23:41
Expensive, right? And GPU costs and so on.
23:43
So you just use someone else's learning and you transfer it to
23:47
your own task, right? That's kind of the where the lot
23:50
of the secret sauce and magic happens.
23:52
So you're. Constantly updating me as a user
23:55
in the embedding database or the vector database?
23:59
And where do features come in? Yes.
24:03
So it features come in during inference, right?
24:06
So we train the model right and and say I'm like, OK, Dimitrios
24:11
has started listening to Jazz and I want to update me too.
24:15
Well man, what is going well, that's.
24:16
Great. OK.
24:18
I have a look at your no Evans. And I am stoked on a Saturday
24:22
afternoon. My wow, my day is made.
24:27
Anyways, I started listening to jazz.
24:28
Yes, and. Our system is going to be like,
24:31
OK, let's trigger an inference for Dimitrios, right?
24:34
And when you trigger an inference, you're going to be
24:36
like, OK, let's build the feature set for Dimitras, right.
24:39
So then those features may be, say, demographics.
24:42
They may be like what what have you been listening in the last
24:45
six months? What have you been listening
24:47
over the last 28 days? What have you been listening in
24:50
the last week? So you have these different time
24:51
frames and jazz kind of comes in into the mix basically, right?
24:56
Right. So then you update the
24:58
representation. So you're not like moving that
25:01
embedding extreme like we just, we just forget everything you've
25:05
heard in the last six months and we just make it like a jazz
25:07
thing. But we just want to like dilute
25:10
it a little bit. We're like, OK, let's just take
25:12
like Dimitrios is like pop and rock taste and just put in a
25:18
tinge of jazz onto it, right. And if Dimitri starts enjoying
25:22
jazz quite a bit, like we will expand jazz in that taste and
25:25
we'll remove some other things, right.
25:27
So that's kind of how it continues to update, yeah.
25:30
And do you have a weight on what I'm listening to right now
25:37
versus what I listen to six months ago or six years ago?
25:41
Because I've been on Spotify for a while, right.
25:42
So how can you balance all of that music?
25:45
I know there are sometimes that I'll see on the UI, something
25:49
will pop up and it'll say like you and all the classic or
25:54
something. I can't remember the exact
25:56
words, but it's kind of like you used to like this.
25:59
Do you still like it? Kind of thing and you want to
26:01
jump back in and listen to it? No, that's a good.
26:04
So I mean those those are against some teams that maybe
26:07
like looking at raw data and and doing that right.
26:10
But for in terms of like the taste profile and embeddings, we
26:14
do have certain factors where we take some of the time, but we
26:17
let them mostly we let the models decide it, right.
26:21
So if you kind of are being very active when listening to
26:25
something from five years back like our model will be that
26:28
higher automatically, right. Otherwise we're going to be by
26:31
default. We are weighing the recent more
26:33
than not, basically. All right, real quick, let's
26:36
talk for a minute about our sponsors of this episode, Making
26:39
it All happen. Lattice flow AI.
26:42
Are you grappling with stagnant model performance?
26:45
Gartner reveals a staggering statistic that 85% of models
26:49
never make it into production. Why?
26:50
Well, reasons can include more data quality, labelling issues,
26:55
overfitting, underfitting and more.
26:57
But the real challenge lies in uncovering blind spots that lurk
27:01
around until models hit production.
27:04
Even with an impressive aggregate performance of 90%,
27:07
models can plateau. Sadly, many companies optimise
27:11
for prioritising model performance for perfect
27:15
scenarios while leaving safety as an afterthought.
27:19
Introducing Lattice Flow AI The pioneer in delivering robust and
27:24
reliable AI models at scale. They are here to help you
27:28
mitigate these risks head on during the AI development stage,
27:31
preventing any unwanted surprises in the real world.
27:35
Their platform empowers your data scientists and ML engineers
27:38
to systematically pinpoint and rectify data and model errors,
27:43
enhancing predictive performance at scale.
27:46
With Lattice Flow AI, you can accelerate time to production
27:49
with a reliable and trustworthy models at scale.
27:52
Don't let your model stall. Visit Lattice Flow Dot AI and
27:58
book a call with the folks over there right now.
28:01
Let them know you heard about it from the M OPS Community
28:04
Podcast. Let's get back into the show.
28:07
One thing that is fascinating though is thinking about those
28:12
people that are like me and you mentioned there's over 100
28:16
million different items. Does that include people?
28:20
Cause I would imagine people are there's you said there's 600
28:24
million users of the app the six. 100 million MAU's
28:29
worldwide, yeah. So yeah, I I'm just wondering
28:33
like there's gotta be a tonne of people that are very much like
28:36
me. I think I'm a special snowflake,
28:38
but if I look in your vector database, I'm gonna find whole
28:42
lot of people That's. The beauty of embedding, saying
28:44
like everyone is a snowflake, right?
28:45
No one person is going to get the same embedding.
28:48
So that that's kind of so special, right?
28:51
Yeah, yeah. That is very true.
28:53
And so then as you're looking at these different embeddings and
28:59
you're looking at ways to make it better in the future, what
29:02
are things that you're focusing on?
29:05
Yep, few things, right? One is definitely feature
29:08
spaces. So like, how do we better
29:11
represent a user, right. So you could.
29:14
So for now, we have been focusing a lot on the music
29:16
side, right? So we have a musical embedding
29:18
and then we have podcast embedding and so on.
29:20
But you can imagine like across content kind of embedding where
29:24
you take Dimitrios's podcast listens along with music.
29:28
So you may be listening to jazz on a day when you listen to say
29:32
like a history of music, podcast, whatever, right?
29:36
So there may be some correlations there which are
29:38
actually very difficult to find algorithmically.
29:41
So to do that, like we need to combine your podcast music and
29:45
all of that taste together and pack it together.
29:48
That's definitely like 1 aspect of improvement.
29:51
Another one is how fast we respond.
29:54
So you listen to something and we like bank like we respond
29:58
immediately and all of your Spotify is updating tragically
30:02
after that. So I think these two itself,
30:03
these two, the two things I mentioned is itself a very tough
30:07
problem and and it takes I think I would say several months or
30:10
even years. Yeah, how hard is it to shift
30:15
from the batch? Like you said, it was like a one
30:18
to two day batch type recommendation cycle to going to
30:24
that end cycle. Yep.
30:27
Fascinating fascinating thing, right?
30:30
The way I see it is that there is a Goldilocks zone with how
30:35
often you infer, right? So I call it Goldilocks zone
30:38
basically, right. So like, I think a lot of people
30:40
think it's either batch or real time, right?
30:42
But actually I think there may be something in the middle which
30:45
is near real time. So by near real time it
30:48
basically means that you don't need to infer for Dimitrius when
30:52
you start listening to jazz and like 30 milliseconds, right?
30:55
Like we don't like your Spotify list.
30:58
You're not like dramatically switch in like less than a
31:01
second, right? That's that's unlikely, right.
31:03
So you instead you can put that jazz listen in a queue along
31:09
with some other people and as a systems like process that so it
31:12
can take a few minutes, but one or two minutes is acceptable
31:18
latency there, right. So that's kind of what I say
31:20
Goldilocks is. It's a good trade off between
31:24
like being responsive but at the same time like you're not
31:27
blowing up. Otherwise like just engineering
31:30
systems have to like scale up to like massive skills and then
31:36
your whole on call burdens you need to put in like I don't know
31:39
how many shares and like do like all kinds of stuff to make that
31:43
a reality right. So that's kind of the dream is
31:47
to train as less as possible but infer as much as possible, but
31:53
not in real time. That's kind of how I see this
31:56
thing play out. O if I understand it correctly
31:59
too, it's that every time you want to update, so the closer
32:04
you get to real time or near real time, That means that you
32:09
have to create new embeddings each time.
32:14
Yep. And let me just give a little
32:17
bit, just to break this in advance a little bit, right.
32:19
OK. So our systems are faster.
32:22
They're still like like the serving is all real time serving
32:26
cannot be like you can't wait around 2 minutes to get back the
32:30
10 lessons, right, like what I said.
32:31
So it all has to be in 30 milliseconds, right.
32:34
So I mean, I just want to be clear there, right.
32:36
So that's kind of the final layer.
32:38
But the question is that that online store, that feature store
32:42
that is online that is serving those recommendations, how fast
32:46
is that being updated? So you have, you have like we
32:50
understand Dimitrios with that understanding maybe of three
32:53
days back, right. So whatever you were doing on
32:56
Spotify three days back is what our understanding is at this
32:59
moment. That's what I see.
33:00
That's still, it's still being sold very fast, like we still
33:03
are serving that at 30 milliseconds, but it's still
33:06
serving things that we know about Dimitrius 3 days back,
33:09
right. So that is a batch features in
33:12
online store, that's what I mean by that, right?
33:14
It's still batch systems because you're understanding is behind,
33:17
but you're still in online. The question is to move your
33:20
understanding your features as much as closer to real time as
33:25
possible, right. So it's you want to move it to a
33:28
level where it's like a few minutes lagging features serving
33:32
in 50 milliseconds. So near real time inference
33:35
served with real time systems, if that makes kind of sense
33:38
basically. It's very clear now, 100%.
33:42
So you have and understanding of mine that's three days old and
33:49
so whatever, I really like that. That put it into perspective,
33:52
because now I see that, OK, whatever I do today, it's going
33:57
to be reflected on my Spotify recommendations in three days.
34:02
Yes. And so one other thing that I
34:06
was thinking about is that the vector database that you have,
34:11
is it just one gigantic database that everything goes into
34:14
because you were saying that you want to be able to have any
34:17
team, any ML team, any recommendation team lay off of
34:22
it or is it a bunch of different ones depending on the location
34:26
or whatever it is? Like how's the system look, I
34:28
mean. We we leverage a lot of big
34:30
table which is essentially a Google cloud product, amazing
34:34
product. And we have like multiple
34:37
different vector databases for like different purposes.
34:40
But for say like tracks, that would be like 1 vector database
34:45
because we want to keep all of that 100 million items or
34:48
whatever that may be, right? Whatever we want to surface like
34:51
in one vector database and then we do nearest neighbour lookups
34:53
on that essentially. OK.
34:57
And then you know which ones to go for depending on what the use
35:01
case is. Exactly.
35:03
Yeah. So that's kind of and we have to
35:05
scale it quite as you can imagine like all of the auto
35:07
scaling and it's a very, very complex system like just to get
35:11
that to function like so that you see no difference basically
35:16
as a user you see no perceive nothing is what our goal is and
35:19
that takes tremendous amount of effort.
35:22
So I do want to shout out to all the all my team members in that
35:25
regard, yes. Shout out yes so correct me
35:29
where I'm wrong. I'm going to try and break down
35:31
the cycle here and you tell me where I'm not understanding it.
35:36
I go on to Spotify, I play my Bill Evans and Chet Baker jazz
35:43
playlist that I absolutely love. This is the first time that I'm
35:47
playing it on Spotify. I haven't played it on Spotify
35:51
because I didn't realise it was on there and I've always been
35:53
playing it on YouTube, but now I play it and it's like ohh new
35:57
information about Demetrius. That information gets taken and
36:03
it gets put into an embedding model to update my embedding.
36:09
And so you're you're constantly gathering information about me
36:13
and if there's new things or whatever the information is,
36:16
you're just adding a little bit to my embedding, which is my
36:19
your understanding of me as a user.
36:22
And once that embedding is there, then it goes and it's
36:26
able to feed that information and that description to the
36:34
online feature store and be able to serve everything very
36:40
quickly. So that in a few days after I
36:43
listen to my Bill Evans playlist I now am starting to see more
36:48
cool great live Bill Evans tracks that I had never heard
36:53
and it's coming U maybe on my if I'm just sometimes I'll let the
36:57
Spotify just play and it will come up with random songs and so
37:01
it'll be spiced in there and then I give it a like and now
37:05
there's like more signals that ohh actually he's really getting
37:08
into jazz and it gets reincorporated in.
37:11
Where I think I'm losing or I'm not so clear on is how does the
37:17
embedding and the vector store go to the feature store and like
37:26
that pipeline or that route, what does it take and how does
37:29
that look? So you said like you listen to
37:32
that playlist that you had heard on YouTube and then you'll come
37:35
on Spotify and listen. So our goal is to not take those
37:38
five days or three or whatever the end days to respond.
37:41
We want to respond ASAP, right. I think we are clear on that,
37:44
right. So that's kind of the goal of
37:46
this new model is like we update like ASAP, right?
37:49
Or rather I should not say ASAP, like to be technically correct
37:52
like few minutes. So we want to respond in a few
37:54
minutes. Now your question was how does
37:58
vector database and features stores kind of connect with each
38:01
other? So it's actually like it's very
38:05
simple, like it's kind of one and the same when you're talking
38:08
in the online setting, right? Because a lot of the times teams
38:13
don't care about embeddings. Great teams want to know the so
38:19
the in a rack system like like a, a person will not care about
38:23
what the embedding is, right? They want to know the documents,
38:26
right? What are the nearest documents.
38:28
Same thing applies here, right? Like teams don't like, they
38:32
don't personally care what the embedding value is.
38:34
So they call the vector database, which you can say is
38:38
like an online feature store because they just calling that
38:40
and then getting back those documents or tracks basically,
38:43
right? So they don't really care as
38:47
much about the raw embeddings when we're talking about these
38:52
candidates generation cases. Now there is a second use case
38:55
which is aligned to more what you are saying is say a team is
38:59
doing ranking right and another ranking is where like you might
39:03
feed in demographics and like are you a podcast listener there
39:08
they fetch the raw embedding from our vector database.
39:11
So that's why I'm like mixing the words so that it's in
39:16
candidate generation, it's vector database in a ranking
39:19
setting, it's a feature store. They're one in the same thing.
39:21
I mean if that makes sense more. Interesting.
39:23
Yeah. I mean, it's a.
39:25
Little, little nuanced, right? But because ranking team is
39:29
building their own models there, they care about the real values
39:33
of embedding. Yeah, they want the real value
39:35
of the embedding. Dimitrius is like whatever like
39:38
128 or 256 dimensional embedding and that goes into their model
39:43
which would then infer, OK right.
39:44
Like I think when you search for queue is it like queen or is it
39:48
like some other other artist? Right.
39:50
Yeah, so. They're like in that search
39:53
model, like your raw embedding matters.
39:57
But when we're talking about the bad habits and shivers example
40:01
like they're the wrong bedding does not matter.
40:03
You just want those 200 candidates or 200 tracks back,
40:07
right? That's where that's a more
40:08
vector database analogy. And in the ranking it's a more
40:12
online feature store analogy. I don't know if that helps.
40:15
Yeah, I like it. It separates it very much
40:18
because I was under the impression that it would be
40:20
like, OK, cool vector database now goes to the feature store or
40:24
the embedding now goes to the feature store.
40:26
But it's more like, no. The feature store is kind of
40:29
doing what the vector database does, and so it there's a little
40:35
bit of overlap there and you still need the embedding, but
40:39
it's not like you need the embedding when you're doing the
40:44
different use cases. And So what I think about then
40:47
is feature stores are very much known for being this power U
40:53
when you need like real time, right?
40:56
And when you're doing that very very fast inference, does your
41:00
use cases with these embeddings and trying to get the embeddings
41:05
out and the candidate generation like do you have, how do you
41:11
make sure that that is fast on inference?
41:14
Because is that still like you're still having to go up
41:17
against whatever 50 milliseconds inference time, right?
41:19
Yep. So we keep everything in these
41:22
fast online stores, which are super fast, 50 milliseconds,
41:25
right? And we update them in the back
41:28
end. That can be one day, 2 day,
41:31
three day or that can be a few minutes, right?
41:33
So, but we still need some notion of Dimitrios in that
41:38
online store so that the search model can call that in 50
41:41
milliseconds essentially. Right.
41:43
Are they getting the best version of Demitrius?
41:45
They can, maybe not, right. They are still getting a couple
41:48
of days behind, but in many majority of cases that's fine,
41:51
right? Even 2 days behind is not such a
41:53
bad, you know, like it's fine like for maybe 90% of the
41:56
systems, they don't really care, right?
41:59
Like they're like, Oh yeah, queue wouldn't mean queen for
42:01
Dimitrios and not like some other artist starting with you.
42:04
Like that thing to be honest is going to be very similar for
42:07
you, right? Like but for another team say
42:10
that is building out the home shelves.
42:13
They really care about up to date because if you started
42:15
listening to jazz, you better want to see jazz there.
42:18
Yeah, Do you want to see? Jazz like right up on the home
42:20
screen and that team cares about it being updated fast.
42:23
So it's really is is again very nice, like which team, what is
42:27
the use case and how fast they want to be updated these
42:31
embeddings. And so the next step on this
42:34
journey feels very much like, how are you informed at how your
42:40
product is going to be used within the Spotify app?
42:43
Because it feels like the whole way you go about creating the
42:49
product would change depending on where you're gonna sit within
42:54
the app. Exactly.
42:55
It's like so many different use cases and complexities.
42:59
I think that's where the we're talking about evaluations,
43:01
right? When you want to make a
43:03
generalised foundational embedding, you have to cover so
43:06
many cases. How do you even evaluate that?
43:09
We can't predict how is it going to be used, right?
43:13
So that's why, as I said, evaluating these systems
43:16
sometimes is harder than even building them, right, because
43:21
not everyone can. I said you can just fine tune,
43:23
but not everyone can fine tune. Not all teams have the capacity
43:27
or ability or desire to fine tune.
43:30
They're like, I'm just going to use your product.
43:32
If it's good, we'll use it, otherwise not.
43:34
Now you really have to step up and build embeddings that can do
43:38
something for everyone, right? And and that's where like doing
43:42
something for everyone in Mill you know like how difficult that
43:45
is essentially because. It's exactly like it wasn't hard
43:49
enough already, yes. She's got.
43:52
Three teams that you're servicing and they all want you
43:55
to go in different directions, yes.
43:58
What are some of these different polls?
44:00
So I guess the way that it works within Spotify on the
44:03
organisational level is that you're creating this product and
44:07
then the UI or UX teams come to you or do you have UI UX people
44:14
embedded in your team to help you and guide you with how it
44:18
would be used? And I would imagine also the
44:20
product managers are helping along the I don't.
44:23
Personally work with UUX as much because a lot of these systems
44:29
are ML. Systems create candidates that
44:32
can basically be a black box. So for a UI, UX person is going
44:36
to be like, OK, we will get 10 tracks.
44:40
How are they created? We don't care, right?
44:43
That's that's like all of this ML that we talked about in this
44:46
show, right? Yeah.
44:46
So for for the UI UX person, it's again a black box.
44:49
They don't technically need to care about it.
44:52
They only care about, OK, once we know the 10 tracks, how is
44:55
the best layout? Do we allow Demitrius to skip
44:58
past it? Do we allow Dimitrios to swipe
45:01
up and add to playlist? So there's a lot of those like
45:03
UI UX elements that you actually interact with.
45:06
But the way we play a role in that is through feature
45:09
collection, right? So sometimes we as ML
45:12
practitioners, we want to know if Dimitrios is skipping past
45:17
some of those things and radio or not.
45:19
And that is something we want to get logged and things and then
45:22
we use those logs in our models. That's kind of where I see the
45:25
loop. So it's going more from the UI,
45:27
UX to Mill rather than the other way, if that makes sense.
45:32
So it's all features, it's all kind of interconnected.
45:36
Wow, so I'm trying to figure out where where to go next because
45:40
this is so cool to think about. How?
45:43
You're basically just saying, hey, here's here's what you get,
45:47
if it's valid, take it, go have fun, plug it into the A, and if
45:53
not, maybe request something else or it's probably not for
45:58
you, go ask another team. Some something like that.
46:02
I mean, it's more nuanced, but yeah, something like that, yeah.
46:05
OK. And it does feel like there are
46:10
constantly new features that are coming U that you could
46:14
potentially want to track. Like I know that there is the
46:20
playlist where you can kind of tune into a playlist by swiping
46:25
through it. And Justice hearing 30 seconds
46:28
of a song, how do you, how do you stay up to date with.
46:31
All right, there's this new feature that's going to be
46:34
rolled out across the app now. You can incorporate this into
46:37
your fine tuning methods. Yeah, fantastic question.
46:41
Yeah, we have some amazing data engineers and data scientists.
46:44
So we typically start this process at data science level,
46:47
right. So we so instead of
46:49
incorporating a new fancy feature like right away because
46:52
you know how difficult productization is, right, we
46:55
first, I mean the way that at least my philosophy personally
46:58
is to 1st use those features in evaluations.
47:01
What you could do is say a fancy new audiobook thing is coming
47:05
out, you could just use that feature to build a cohort,
47:09
right? So you would say, OK, we have
47:11
been evaluating embeddings on podcast listeners, music
47:14
listeners, how about audiobook? And we use kind of those
47:16
audiobook features in that evaluation.
47:19
And if we gain confidence in these kind of notebook
47:23
environments and with analytics and AB testing kind of
47:26
situations, then you know, OK, this feature is not bad.
47:29
You know this feature is pretty good and it's helping us.
47:33
If it's helping us move our metrics in the evaluation land,
47:37
maybe it will help our model we bring it back, right.
47:40
So it's we do will not like typically jump onto anything new
47:45
and fancy unless it's really, really cool.
47:48
So that's kind of how I see it kind of play out like, yeah.
47:52
And are the main metrics you're looking at right now what you
47:55
were talking about with speed and relevance?
47:59
Yep. Relevance is definitely one of
48:01
them. So it would be like essentially
48:05
like things like relevance, accuracy, like all those things
48:10
but also like we have this ranking metrics right.
48:13
So in like imagine like you have a, you have home like where
48:17
there are like multiple shelves. This is a ranking problem,
48:20
right? Where like which one are you
48:22
clicking or which one are you engaging with, right?
48:24
So that's where like things like Mr and NDCG come into play.
48:32
But yeah, you could say like this, that's some form of
48:34
relevance. That's a more of a ranking
48:36
relevance, yeah. Yeah.
48:39
And and that's also where that, like that twin or that
48:43
prediction simulation comes in, where you get to say, let's see
48:47
if we do this or if we show these or whatever, Let's pretend
48:51
like we're going to simulate out what Dimitri is going to do for
48:54
the next three days, 5 days. And.
48:57
And then I throw you a curveball and I listen to Gregorian chants
49:00
and you go, whoa, I was not expecting that from this guy.
49:04
Yes. Exactly, exactly.
49:07
But but some of our discovery teams will be very happy, right?
49:10
They will be like, OK, this is discovered something new like
49:14
can we update Discover Weekly now?
49:16
Because Discover Weekly like follows you, but then they want
49:18
to give you something you've never heard before, right?
49:21
So they really like, they love it.
49:23
OK, now you have heard this something new genre like let's
49:25
pick on that and start showing you so, but some other team is
49:30
going to be unhappy, some other like.
49:31
So it's, you know, it's kind of how like all these different
49:33
systems interplay and work together to kind of satisfy the
49:38
user basically. Man, it's been awesome talking
49:42
to you. I really appreciate how
49:43
transparent you are in all of this stuff that you're working
49:46
on. It sounds like a really hairy
49:48
question when it comes to evaluating this.
49:51
Keeping it fast, but also making sure that what you're getting is
49:57
up to date and it is interesting for the user is very fun.
50:03
Yep. Absolutely.
50:04
Fun stuff. Right on, man.
50:06
Well, this has been a pleasure. Thanks for coming on here.
50:09
Thank you SO. Much for inviting me pleasure.