Latest / Agentic Conversations (formally mlops.community) / BigQuery Feature Store // Nicolas Mauti // #255

Transcript

BigQuery Feature Store // Nicolas Mauti // #255 · Agentic Conversations (formally mlops.community)

0:01
Hello, I'm Nicola lead the envelops at malts and
0:06
unfortunately I don't drink coffee.
0:10
I like the smell, but I don't like the the taste too much If
0:14
you if I try regularly, to be honest.
0:17
And so my morning routine is more about orange juice or I
0:22
have also a very strange habits and my correct on me.
0:26
So I'm a huge fan of min mint syrups with water and so yeah.
0:35
What is up good people of the world?
0:37
Welcome back to another Mlops Community podcast.
0:40
As usual, I'm your host Demitrios.
0:43
Today talking with Nicholas, he got very transparent on how they
0:47
are using Big Query as their feature store and what the pain
0:52
points that they had before they implemented it as a feature
0:55
store were and now the benefits and how exactly they have
1:03
implemented the feature store in Big Query to set them up for
1:08
success. He echoed something that I have
1:11
heard time and time again, which is some of the biggest unlock
1:15
from creating feature stores are decoupling of the feature
1:20
creation, feature generation and the code creation or just coding
1:25
in general model creation. He said it very eloquently.
1:30
I'll let you listen to his whole way of putting it, but basically
1:34
when you're doing one of these two tasks, you're in a
1:37
completely different headspace and he has seen a huge uptick in
1:42
the capabilities from the team since decoupling these two
1:47
tasks. That being said, this is a
1:51
exciting episode for me because it's a prequel to our Data
1:56
Engineering for AI and ML virtual conference that we've
1:59
got coming up on September 12th. Nicholas was a person that
2:05
filled out the call for speakers.
2:06
We didn't have enough room because we're full as far as
2:09
speakers go. But I said, man, this is such a
2:12
good talk. Just come on the podcast, let's
2:14
chop it up and let's talk about it in a more free flowing form.
2:18
And I got to grill him because I had seen the talk I had.
2:21
I knew what he was going to get into.
2:23
So hopefully you enjoy. As always, if you like this
2:27
episode, feel free to share it with one friend and let them
2:29
know how they can get their feature store game on point.
2:40
I didn't mention it to you, but dude, Big Query is all you need
2:43
is a great title. I like that title It's on.
2:47
I know this originally wanted to be a talk and I convinced you to
2:51
do it as a podcast, so I appreciate you being flexible
2:54
with me on this. Let's get into it because you
3:01
can. You break down the scene.
3:03
What is the end goal that you were trying to go for and how
3:08
did Big Query play its part in this?
3:12
Yeah. So it's a pleasure to be to be
3:14
part of this podcast. And for sure it was a talk at
3:18
the beginning, but I think we would have a great discussion.
3:21
And so it's good to take some time to talk about that.
3:26
So yeah, so to explain globally so and how this project started,
3:34
because in fact it's a project to use the query as a feature
3:37
store in our company. So it starts.
3:42
In fact, we train a lot of model at Malts.
3:47
So basically what we're doing is recommender system, NLP task,
3:53
stuff like that, some matching and so this kind of machine
3:58
learning model. And so for that we trained
4:03
several model for recommendation.
4:05
And so before that we did the featuring just before training
4:12
the model. So basically you grab some past
4:16
interaction on the platform and then you will grab some
4:19
properties from your. So we are matching Freelancer
4:23
with projects. So it's a hatch error process.
4:27
And so we are grabbing some information about the freelancer
4:33
skills, DD rates, stuff like that and also from the project.
4:40
And so in fact we did that for all our models and it was ready
4:48
part of the training and we didn't add any source of truth
4:52
or stuff like that. And so it was a very pinpoint, a
4:57
pinpoint and so. It was a pinpoint because you
5:01
couldn't really version it, or you didn't understand you
5:04
couldn't reproduce it. Yeah, exactly.
5:07
In fact, you have several problems.
5:09
So you have some consistency problems because you can have
5:13
two data scientists that train a models that want to use the same
5:17
feature, but it's not computed exactly the same way.
5:20
And so it could create confusion, also confusion with
5:23
the project because if you want to take to the project, OK, we
5:26
are using that in our model, but this has the same name, but it's
5:30
not computed the same way by two data scientist team.
5:33
So it's not it's not very good for for the roject and for the
5:37
comprehension of of our algorithm.
5:40
O There was feature sharing amongst the team, but it wasn't
5:45
actually reliable feature sharing.
5:47
It was almost like, yeah, it's the same name, but it's not the
5:50
same feature underneath the. Hood yeah, it it was mainly copy
5:54
pasting, sharing, I would say but but but for sure, yeah, it
5:59
was it was not very strict. And so it was not one source of
6:03
truth. So we add some tables, but
6:05
sometimes you have to do computation to prepare your
6:08
feature or to compute your feature because it's not
6:11
directly in the table like you want.
6:13
And so sometimes you have to do some coons or stuff like that.
6:16
And you know, the business is really complex.
6:19
It's not so simple. And so sometimes you have some
6:22
different rules to to create the same feature.
6:24
So it was one of our problem. The second one is efficiency.
6:28
I would say it's not very efficient to make the
6:31
computation of the feature on, on, on, on one part of the code
6:36
and the same on other part of the code.
6:38
Let's share this training. Is is this computation?
6:44
So yeah. So because it was, it was
6:46
computing features in each point.
6:49
So it wasn't like caching those features, or it wasn't storing
6:52
the features anywhere that you could pull from.
6:55
No, it was not storing. We, we were not storing the
6:59
feature. We are just computing on the
7:01
flies and training the model. Wow, wow, wow, and.
7:04
You were able to get the desired speed because these recommender
7:10
systems were low latency, right? Yeah.
7:13
So there we are talking only about the.
7:16
I think we we can discuss that. But there is 2 parts.
7:20
There is offline serving for the training the model and the
7:24
online serving to serve the model.
7:27
And so to serve the model, we have a completely different
7:30
approach. In fact, we are just loading in
7:33
memories and features because it's not so big and so we can
7:37
just do it in memory. So it's it's not a problem.
7:42
And so for that we have a daily basis approach and a daily basis
7:46
computation. So each day we recompute the
7:48
feature. It could take, I don't know,
7:51
maybe one hour, something like this, and then we inject them in
7:55
in memory. So it's not the problem.
7:57
The problem was mainly for training the model.
8:00
Yeah. OK.
8:01
So the serving side felt like, OK, we're all right here, but as
8:07
far as the training goes, you didn't have the ability to A,
8:13
get reliable feature sharing happening and B, have the
8:18
computational load or have cut down on the excess computation.
8:24
It wasn't the most optimal way of doing it because you were
8:27
computing features in various different spots and then not
8:30
saving them and going through the code and computing the
8:33
features all over again. And so that, that makes a lot of
8:37
sense. Was there 1/3 pain point?
8:39
Yeah. As a result, but it's linked to
8:42
the two other one, but something that is very important and it's
8:47
very part of the feature store and how you manage your feature
8:51
for training your model is what we call a point in time
8:53
retrieval. So in fact, when you want to
8:56
train a model, so it depends of the model.
8:58
But in your case, we are using past interaction between
9:02
freelancer and projects. And so if you want to train a
9:06
reliable model, so you know that for example, one year ago a
9:11
freelancer said yes to a Java or any language, you want a
9:16
project, but maybe the freelancer has different skills,
9:20
No, maybe you don't want to do Java anymore, or maybe he said
9:25
no one years ago, but no, he have the skills.
9:28
So you have to get the state of the freelancer and also of the
9:34
projects at the time where the interaction was done.
9:40
And so this is very difficult to recompute that, especially if
9:47
you want to do it on the flyer. So you will have basically to do
9:51
some big join between tables that have historical data and
9:57
it's very complex. So it's very error prone.
9:59
And so the data scientist could make some error doing this
10:03
featuring. And it's also very expensive.
10:06
So it's not very efficient, efficient console because you
10:09
have to scan a lot of historical table to do that.
10:14
And so you thought to yourself, there's got to be a better way.
10:19
Yeah. How did you go about figuring
10:22
out the solution? Because I think the solution
10:24
that you ultimately chose, which was using Bigquery as your
10:28
solution and that was all all you needed, right?
10:31
That is a little bit counterintuitive.
10:34
I would imagine most people would start to go and look at
10:37
feature stores as their first solution, or maybe they're going
10:41
to try and figure out some combination of Redis and some
10:47
open source, probably like a feast.
10:50
But why did you say let's see if we could do this all in Big
10:54
Query? So you are right.
10:56
So Regis I think is more for the online serving.
10:59
And as I said, we are doing it in memory.
11:01
So we don't have this problem. But in fact we we think about
11:08
it, we, we check feature store and stuff like that.
11:11
And in fact we realise that none of them fit all our needs.
11:15
We had a lot of needs. We didn't talk about this before
11:18
or so, but one challenge we had also was the monitoring and the
11:24
anomaly detection in the feature.
11:26
And if you do it on the fly, you cannot do analysis on that.
11:32
And so, yeah, so it was a lot of challenges.
11:35
But the monitoring, sorry to interrupt the monitoring on the
11:39
feature creation or what was on the data itself.
11:44
What monitoring are you talking about?
11:46
Yeah, about the data and so about the feature.
11:49
So for example, if you are, if one of your feature is a daily
11:53
rate of the freelancer, maybe you want to assess.
11:58
So you want to assess two things.
12:00
The first one is and this one is more linked to what I think.
12:03
You just want to be sure that when you train a model, all the
12:08
daily rate are not null because there is an issue in the back
12:11
end or I don't know where. And so you want to be sure that
12:14
your data looks right for training the model.
12:18
And also, I think you want to monitor the feature across the
12:23
time and to be sure that your daily rate doesn't increase or
12:28
decrease. It could happen just due to
12:32
inflation or, or, or or thing like that.
12:34
But you want to be alerted by that.
12:36
And, and you want to to know that you have this drift in your
12:40
data and maybe to retrain your model or to check if it's a
12:44
product problem. So maybe just a change on the
12:47
product side or I don't know. So.
12:51
So that's why you want to monitor.
12:54
Yeah. Because at the end of the day,
12:55
if you're getting the wrong rates and suggesting to
13:00
freelancers projects that they are way too expensive for,
13:05
that's going to create a bad suggestion and ultimately not
13:09
lead to matching. And I imagine that there's a lot
13:13
of people that are on the platform and if they get
13:17
recommended three different jobs that the rates are way too low
13:22
for them. If it's too high, I imagine
13:24
they're not going to care. If it's more than their daily
13:27
rate, then that's going to be awesome.
13:28
It's like, well, this might as well try.
13:30
But if it's too low, that's going to create a bad
13:33
experience. Yeah, for sure.
13:35
In fact, it's in both way, it's not very perfect because in one
13:38
way I think you won't be able to match because the delivery is
13:43
too or you propose too low freelancer and the daily rate is
13:48
too high in the project. But there's a way not very good
13:52
also because you will contact very high qualified freelancer
14:00
with a low, low, low, low pricing project.
14:05
And so it's not good also and at.
14:07
The end of the day, yeah, one side is not going to be happy.
14:11
Yeah, exactly. Got matched.
14:13
All right, so, so then you said big query.
14:16
Yeah, walk me through that. Yeah, so big query.
14:18
So in fact, we, so we check a lot of product and we thought
14:21
about, OK, maybe we can use some things that it's already in our
14:25
stack because it was already in our stack for analytic stuff and
14:29
stuff like that. And so in fact, we say, OK,
14:33
let's just build some feature tables.
14:37
It will be 1 table with all the feature.
14:40
And so the data scientist team will have access only to this
14:43
table. So you have one column per
14:45
feature. And this table will be computed
14:49
on a daily basis. So each day we compute the as a
14:53
feature. Sorry for all your freelancer,
14:56
all your projects and just storize this table.
15:01
And so each day, so we compute all our features and then we put
15:05
a timestamp and store it also in the query.
15:10
And so this way, when you train a model on the data set inside,
15:14
you just have to constrict your training data set with your Y.
15:17
What you want to predict the ID of the free answer, the ID of
15:21
the projects and the date of the observation.
15:25
And then you'd have just one join to do on this table and you
15:29
have all the data for all the dates you want.
15:34
And so that's it. To be honest, we are done.
15:38
And if you want to do online serving, so you mentioned
15:42
readies, for example before. So if you want to go with
15:45
readies because in memory it's too large, you can just take the
15:51
computation of the day, put that into readies and then when you
15:55
want to do some live serving of your application, you can just
16:00
ask ready send you have the lattice data available for each
16:04
entity. And so I imagine now you are
16:08
able to version your features because you said you just do
16:12
basically you, you make these computations once a day and then
16:16
you can look back in history and say, what was it on April 17th?
16:19
Yeah. So in fact, it's that very not
16:22
exactly version. So just you have a time stamp
16:25
and so as as you historize each day the comp, the value of each
16:30
feature, you can just go back in time.
16:32
And so maybe I didn't understand well the question, but or or
16:37
your argument that's. That was it.
16:40
That was exactly it. And so I'm, I'm wondering now
16:46
that you have this in play, what have you seen happen?
16:52
So it's like you, you really explained the before to me
16:56
nicely and there's pain points. What's the after and where are
17:01
the pain points? So what's the after?
17:05
So the after is really good now because we have these features
17:09
and so it's work, as I said. So the data centre system want
17:12
to create a model, OK, they have one table, they create their
17:15
data set with the Y with the two ID and the date and they can
17:18
just join and get the feature. They won't.
17:23
So it's worked very, very well. But I would say that we had a
17:28
few challenges with that as the first one is what happened when
17:34
you want to add a new feature that's something that it's not
17:38
already part of the feature table.
17:40
For example, the number of projects that the freelancer
17:43
already done on the platform. For example, this could be a
17:46
feature. And So what how do you add a
17:52
feature? And also how do you Add all the
17:55
history of this feature? So maybe you can you have.
18:00
To backfill it, right? Yeah.
18:02
Exactly. That's exactly the term it's
18:04
backfilling. So yeah, so this is backfilling.
18:07
So for that we develop a mate solution with a script.
18:11
And so basically, so you have two different way to backfill.
18:18
So in factor data warehouse is composed of layers.
18:23
So you have the ingestion layers, then you have some
18:27
transformation layers and at the end you have the feature tables.
18:32
And so at the ingestion layer, we historize everything.
18:39
So basically if you want to add a feature, you can just get to
18:43
these tables and you have all your history and you can do the
18:48
SQL, QR, whatever you want or scan the data the way you want
18:54
just to compute your feature for all the dates.
18:56
If if you have the data for sure, if it's a new feature on
18:59
the product, you don't have the data.
19:01
But for a lot of feature, we already have this data in these
19:05
tables and you can then update your feature table.
19:09
And so for that we just build a small scripts where the sorry is
19:14
a data scientist just have to input their query and it will
19:18
automatically play the query on the feature table and and fill
19:22
it. And it's also versioned and you
19:24
have a change log and everything.
19:26
So if you want to know what happened to this feature at this
19:30
time or when this feature was ID or stuff like that, you can go
19:34
to this change log or check the version of of the feature
19:38
tables. And is there a process like do
19:40
people have to create some kind of PR request in order to get a
19:46
new feature on that table or is it free for all?
19:50
I can anyone can throw up new features?
19:53
No, exactly. So the process I talked about
19:56
before, just before we call it upgrader in your, in your team.
20:03
And so basically, so you have a project on GitLab with the
20:07
upgrader and you have a folder when you can put all your SQL
20:10
scripts to update your feature table and then it's played
20:14
automatically. So not by the CI, but by a
20:17
scheduler. We are using Airflow and so yes.
20:21
And so we have this scheduler that we play all the SQL files
20:25
that were not already applied on the table to update it.
20:29
So for sure the data synthesis team have to write this SQL
20:35
query, then submit a pair and when it's validated we can apply
20:40
it on the feature table. Fascinating.
20:44
And is there a way now that folks are able to go back and
20:52
and share easier? Because you mentioned the
20:55
sharing was a bit of a pain point, but it was more sharing
20:58
was a pain point because two people were using the same name
21:01
for something that was computed differently.
21:03
And it ended up giving them different results after they
21:06
added that feature to their model.
21:08
Now it seems like you've conquered the problem of having
21:14
two different computations of a feature with the same name, but
21:19
I am guessing that it's a lot easier because people can just
21:24
reference the big query table and see, OK, this feature seems
21:28
like what I want. I'm going to add that to my
21:30
model, yeah. Exactly.
21:32
And maybe it's a teasing that it's not already live and we're
21:36
working on on this way. But I would like to develop
21:41
something to help the data scientist team to find the right
21:44
features because know you have that.
21:46
And also in the query, you can so each colon is a feature.
21:50
And so you can add a description for each colon.
21:53
And you know, easily with some, even some LLM or stuff like
21:59
that, you can maybe say, OK, I want to train this model returns
22:03
the features that could be interesting for this model.
22:06
Or just the basic search could be useful also just to search
22:11
the feature on plaintext or even with something like some keyword
22:16
matching just to makes the discovery more easier.
22:20
You know it it it could be a a good feature for the data
22:24
scientist team because sometimes they don't have the same term
22:28
for the same feature too. Yeah, 100%.
22:31
And that feature discovery is huge because if you're going
22:35
through and getting inspiration from what others are using in
22:39
their models, what types of features they're using, then
22:42
that can hopefully translate into you creating a better
22:47
model. Yeah, exactly.
22:51
And so I think it's also a point we discussed before, but for me
22:55
it's very linked to the monitoring because when you have
22:59
your feature, you want to know how it behave and how is the
23:03
feature and do I have a lot of nod in this feature?
23:08
What what are what are the shapes you know of the feature
23:12
and of the data behind this feature?
23:15
And so for that, we also leverage a tool that we already
23:19
had Grafana. And so basically Grafana is
23:24
plugged to be query. And so in Grafana, you can just
23:27
explore each each feature and it will give you some descriptive
23:32
statistics about the features. So the means average.
23:36
If it's a categorical feature, you have the current by category
23:40
and stuff like that. But it's not giving you the
23:45
usage of the features, is it? Do you have any type of lineage
23:48
of these ten models are using these features or this feature
23:53
is used the most by 50 different models?
23:57
No, that's a good point. We don't have that right, No.
24:01
I I get the feeling that is probably kind of hard to
24:04
implement because you're looking at the whole data lineage as
24:08
opposed to just how the the features itself, right?
24:13
You can't do that in Big Query. You can't really know who's
24:16
pulling from what features, I guess.
24:19
Sure. So yes, if I could have some
24:23
data lineage tools that tells us OK, this colon is popular, but
24:29
we cannot, we cannot know that it's popular and used by this
24:34
model in particular. We just know that OK, this colon
24:37
is used, but we don't know if it's used by this model or this
24:41
model or this model. How fascinating.
24:45
Yeah. Because I just know I've heard
24:47
so many horror stories. And I imagine you deal with this
24:50
day in, day out. If one of these columns goes
24:54
haywire and like you were saying, you're monitoring it and
24:58
you're seeing, wow, this column is now not working as we had
25:04
anticipated it to be working, something's going wrong here.
25:07
What downstream effects does that have?
25:09
Which models is that actually affecting?
25:12
So you can know potentially you've got to retrain those
25:16
models. You've got to roll back, you've
25:18
got to do something before the models that are out in the wild
25:23
feel that effect. Exactly.
25:25
And so now we don't have this right now.
25:29
And so you, you don't have, we don't have this at column lover.
25:36
We have this at table level. But it's not very interesting in
25:39
this case because all the model will use the same table.
25:42
So because, yeah, we have dependencies.
25:45
And so we talk about alerting before, but we have also
25:49
automatic alerts. So for that we also leverage
25:53
some tools that we already had great expectation for that.
25:58
And so great expectation, yeah. What exactly?
26:02
Yeah, you might have just about been able been saying this.
26:06
What exactly is it doing? Is it monitoring the data that's
26:09
flowing into big Query? We are.
26:12
Monitoring, so in fact we are calculating each day as a
26:15
feature for all our entities. So entities is freelancer or
26:20
project global name for that. And then when we have this
26:24
table, we will run some test on it.
26:27
So great expectation are like unit test on tables.
26:32
And so if one of the tests fail, it will just stop the word
26:36
process and we won't train any model after that.
26:40
And we will just have slack alert about OK, this column
26:43
doesn't match your expectation. And so for example this column
26:46
is always new or you have a new value in a categorical column.
26:51
So your categorical column must be this or this and you have
26:53
that and so it will block and so we are alerted for that.
26:59
Yeah, it's like this just came in at 14 and usually it's
27:04
between zero and one. What's going on here?
27:07
It's way out of distribution. You might want to go check it
27:10
out. Yeah, exactly.
27:11
But this is, again, I guess the interesting piece is you're just
27:17
monitoring the specific columns for with Great Expectations,
27:21
right? You're not actually monitoring
27:23
the data that is coming in and going through this ingestion and
27:27
then your transformations. And we also have a lot of
27:31
monitoring about that. It's, you know, it's more that
27:36
engineer work than MLS work. But yes, so they have some tools
27:42
basically. They also have Great
27:43
Expectations. So in fact, you have great
27:44
expectation all the way to just detect as soon as possible the
27:49
issues because it's easier to fix and it won't impact a lot of
27:54
downstream jobs or tables or stuff like that.
27:57
So I have Great Expectations. It also have some, yes, some
28:03
scripts or some table just to assess that the data in this
28:08
table are coherent with are in phase with data in this other
28:13
table or stuff like that. So yeah, so they have, they have
28:16
some toolings all along the world chain.
28:19
But in our case for featuring, we are using Great Expectation.
28:25
Talking about Great Expectation is makes me realise that you're
28:31
probably doing something along the lines of continuously
28:36
retraining models, and I write in assuming that.
28:40
Yeah, exactly. Can you break down how that
28:43
works so it? Depends of our models to be
28:47
honest. So it depends about the
28:50
generation of the model and a lot of stuff.
28:54
So if also if it's a recent model, we prefer to train it by
29:01
end for the first time just to be sure that everything looks
29:04
good. But otherwise, we have in most
29:10
cases we have a monthly training of our model.
29:14
So as I said before, so we have some cheques about features and
29:19
if everything looks good, we will just train on your model.
29:23
And if the metric of the model looks good, we will deploy it
29:27
automatically. But and is this all happening
29:29
through GitLab? No.
29:31
It's everything is done in Airflow, Airflow, OK.
29:36
We are using Airflow for the whole scheduling of all this
29:40
task and also for what we discussed before about the
29:43
ingestion and then all the the layers for data warehouse and
29:49
then the feature stable. So it's all all scheduled in
29:52
Airflow. And is it just on a time basis?
29:56
Yeah. So it's, it's on a time basis.
30:00
So we have some so airflow use DAG for graph to to run the
30:06
task. So we have some daily or monthly
30:09
DAG. But if we want, we can just
30:12
create one manually. So if we detect an issue with
30:15
the model or we want to rerun this task or we want to fix that
30:18
in your table you can we can just clear the task and it will
30:23
restore it and fix the whole stuff.
30:26
Because I'm wondering about if you recognise that there's some
30:30
kind of model misbehaving out in the wild.
30:34
Do you have the capability, A to recognise that a model is
30:38
misbehaving, and B just to say, all right, well, let's go
30:41
retrain it and figure out if that solves the problem, Yeah.
30:46
Yeah, Yeah. We have the possibility to start
30:49
a train a manually and to just release a new model.
30:55
Yes, it's completely possible. About the the detection of the
31:03
of the bad behaviour, I would say of the model, yeah, we also
31:08
have some different monitoring. So in fact, when you monitor
31:14
model, I think you have multiple way to, to, to monitor it.
31:21
So we have business metrics. So for example, the matching, we
31:26
know that or the, the average, the average conversion is that.
31:34
So if we see that the conversion is dropping, so maybe it's a
31:37
problem. I I don't like this metric
31:40
because when you, when you learn that your model does not work
31:46
correctly from the from the business, it's a little bit too
31:54
late. I prefer to detect it early
31:58
because when it's a business, it's not very good.
32:01
Yeah. There's some very not happy
32:03
people that are talking to you or hitting you up on Slack.
32:07
Exactly. And I can understand that so
32:10
yeah, for for sure. So that's why we have also some
32:15
earlier monitoring. So just we, we'll just log every
32:21
single return from the model and you can, for example, check the
32:25
score and how it's change over time and this kind of stuff.
32:31
So with that, we try to detect earlier some issues.
32:37
And that's also why we also added some monitoring about the
32:41
feature because it's much earlier as your model is not
32:45
already trained. And so if we are able to detect
32:50
some issue in the future, you can avoid issue at the in the
32:57
output of your model and in the business.
32:59
So. And are you the one that's
33:02
responsible if there is some metric like latency that all of
33:07
a sudden shoots up so you're serving the model and it takes
33:12
one second to serve the model out of nowhere?
33:15
Yeah. So it's also part of my job.
33:19
And it's another kind of monitoring.
33:22
It's more, I would call this platform monitoring.
33:26
So we are monitoring CPU, RAM usage, also latency.
33:31
And so, yeah. And if there is an issue with
33:34
that, I can just discuss with the data scientist team about
33:37
how to solve that. And yeah, so yeah, it's also
33:41
part of the job. And sometimes, yeah, with the
33:44
latency, we can detect other issues.
33:46
Sometime behind the latency is a latency issue.
33:51
You can find maybe a quality issue.
33:53
And for example, you wrote on too much Freelancer.
33:56
And so that's why the latency increase, because your pool is
33:59
very large. And so maybe it's a problem for
34:02
from your model, but you detect it through latency.
34:05
So sometimes you have this kind of effect for sure.
34:08
Yeah. That's a great point that the
34:11
problems when you go to the root cause, it's not as simple as if
34:17
it was just as a software problem because you have to
34:24
think about, well, there might be a reason that this is
34:27
happening that is inherent in the model that just got pushed
34:31
out or just got updated or is going a wire for some reason.
34:34
So you have to look at it from that angle and that's why it it
34:38
makes sense that you would be in charge of that as opposed to
34:42
someone else that isn't as well versed in the ML side of things
34:46
yet. I heard it put one time as the
34:52
monitoring side. My friend Shubi told me this was
34:57
even four years ago. He said I like to monitor and
35:00
think about monitoring on three different levels.
35:02
One is the actual accuracy of the model, another is the data.
35:11
So we're monitoring the data and that seems like what you're
35:14
doing with like the features and what's happening with the
35:17
features and also with Great Expectations and Grafana on
35:20
those columns. And then he said on the systems
35:24
level, so that latency and all that.
35:26
So you have those three different ways that you're
35:28
monitoring that model that is out there making predictions
35:33
because it can go wrong on one of those vectors and it can
35:37
screw everything up. Yeah.
35:40
Totally agree. I would add the also the
35:43
business level, I think it's important and you have a lot of
35:48
factors that could impact the business, but your model could
35:54
be one of these factor. And so even it's not very
35:58
precise and it's not directly linked to your model, I think
36:01
it's very important to put this in place because at the end,
36:06
it's the goal of the model. So to be honest, even if the
36:09
score of the model looks weird, if the business increase and
36:14
it's OK, maybe it's not a very big problem.
36:17
So you you have to balance and so, but at the end you are
36:23
evaluated on the business, not on the accuracy of your model,
36:27
you know. And you don't think how, how do
36:30
you separate in your head the accuracy versus the business?
36:36
So I think it's not separating the accuracy and the business.
36:40
It's more OK if the business goes wrong, let's look at the
36:44
accuracy. And so if the accuracy goes
36:46
wrong, also if you can start to see something on the accuracy
36:51
that seems to go wrong, OK, let's let's deep dive in some
36:55
other. And I think the issue should be
36:57
there. If the business is wrong, but
36:59
the accuracy is very good, maybe we can start to look elsewhere.
37:04
Maybe it's maybe it's a model and maybe you can also invest in
37:07
that, but maybe it's earlier. So it's more gets a lot of
37:13
different metric at different stage to debug more efficiently
37:19
and to be able to spot the right spots of the of the.
37:25
Yeah. It's, it's such a funny thing
37:27
that you say on potentially business metrics are going up,
37:33
all the other metrics are going down.
37:34
So in that case that even though that feels like a very rare,
37:39
rare occurrence, but in that case, don't touch anything, just
37:42
let it run. Right, Yeah.
37:44
Or maybe just investigate, but because maybe you can do better
37:48
if your wallet is better. And maybe.
37:51
It could be better because there as I said there is also other
37:54
factor from for, for the business.
37:56
So maybe it's just the sales teams that works pretty very
38:00
good or there is a lot of other factors.
38:03
So I didn't say don't touch anything, but it's more OK,
38:09
let's have also the business metric to just say, OK, maybe my
38:14
mother is not as good as I think it is, but it works.
38:18
So maybe it's OK. And so say, yeah, it could help
38:23
you to prioritise your work and to know if you have to
38:27
investigate a lot of the model or not depending of of the
38:31
business also. So I think it's, it's, it's more
38:34
it's not business versus accuracy of the model of or or
38:39
or output of the model. It's more OK, let's get a lot of
38:44
metrics from different spot and just cross them to check where
38:49
where the problem could be. Yeah, looking at it more
38:52
holistically, Yeah. And then diving in where you see
38:55
some things that don't feel right.
38:59
Have you done a retro on now that you're using Big Query as
39:05
your feature store? Basically looking at it and
39:12
saying we've been able to alleviate all these different
39:15
pain points, but I am assuming that Big Query comes with a
39:19
cost. And so you recognise we're
39:23
paying this much more in hard cost that we can see from Big
39:27
Query. But we were paying before this
39:31
much in people time that they had to go and recreate a feature
39:37
when they recognise that it wasn't the feature that they
39:39
wanted or they had to. So it's all it feels like there
39:42
was a lot of fuzzy costs that you really couldn't account for
39:48
before, but now that you have Big Query, it's very clear what
39:53
you're paying. Yeah, exactly.
39:56
Also we are paying only once a day when we compute the feature
40:01
for all the entities, whereas before as the featuring were
40:06
done when we were experimenting or training a model also locally
40:13
or testing some stuff. We are paying at each execution
40:17
of the training and of the featuring.
40:19
So now we pay once a day and then the DSNT team can access
40:25
whenever they want and nearly and no cost to the feature.
40:30
We you just have the cost to get once the feature, but it it,
40:34
it's pretty OK. So you have that also, I think
40:37
something that it's not really cost, but something very
40:40
interesting is that now the data centre system is better able to
40:48
split the work between feature engineering and model training.
40:53
And so when they want to train a model, they can just, OK, I will
40:59
need this feature, they can start to implement all this
41:03
feature in the feature table we have.
41:06
And then they train the model and they don't mix both.
41:11
And I think it's more clear in their head when they're working
41:15
about, OK, no, I'm optimising the model and no, I'm doing
41:20
feature engineering because I think it's, and I did some that
41:25
I sent before and I, I trained some model in the past and it's
41:30
very, for me, it's pretty different job, not different
41:34
job. It is very different task.
41:36
So you don't have to think about the same, same thing.
41:39
And so I think it's much easier for them not to split this work
41:44
and to have on one size of future engineering work.
41:47
And maybe they will work for two weeks on that.
41:50
And after that they could start work on the training of the
41:55
model and just think about that, about the parameter of the
41:58
model, about the structure of the model and stuff like that.
42:03
And so I think it's better for them also too.
42:06
It's funny because we had on here the creator of Feather,
42:10
which is a feature store that got open sourced by LinkedIn
42:15
probably two years ago. And one of the things that he
42:18
said was the biggest boon of having a feature store was the
42:24
fact that you could decouple the code from features and feature
42:29
generation. And that's exactly what I'm
42:31
hearing you say is how you're in such different headspaces when
42:35
you're thinking about what kind of features do I want, how am I
42:38
going to create those features, all of that versus what's the
42:43
model doing in the overall coding the model and trying to
42:46
figure out how to make the best model that you can.
42:50
That being said, do you know that the hard numbers on like
42:55
how much money you saved because now you're not computing
42:58
features ad hoc in five different places when a data
43:02
scientist is training a model or when five different data
43:05
scientists are training models and you just compute it once a
43:08
day So you have a clear cost of what it is.
43:10
Did you go back and say what were we spending and what are we
43:14
spending? And now we can give a talk at
43:17
the next Fin OPS conference. And I don't have very precise
43:23
numbers to be honest, also because it was a long project,
43:28
so we started to work on that one years ago.
43:30
And so the team change and there is no more people in the teams
43:34
that are using more big queries. So for sure the cost increase.
43:39
But when we started the project, I calculated that the featuring
43:48
was several hundreds bucks per month, something like this.
43:54
But I think it depends, it's not applicable for other company,
44:00
depends about the volume of data that you that you have.
44:03
Also we train bigger model, no. So it's not very comparable.
44:07
And so, yeah. Well, let's talk about that
44:09
because when do you think this architecture or this style of
44:13
doing it would fall over? So someone out there is saying
44:18
oh maybe I'm going to try this and just make Big Query my
44:22
feature store. Where would you recommend to not
44:26
take this approach? I would say maybe our biggest
44:31
challenge no would be if we want to do some live featuring also
44:38
live computation of the features.
44:41
Like on the fly. Yeah.
44:43
In flight basically with some flink etcetera.
44:47
Yeah, exactly because. I think the Cray is not very
44:51
good to just add data on the fly in it.
44:56
So we are working in batch and I think for that it's worked
45:00
pretty well. But if you good, if you want to
45:03
go with full on the fly pipeline, maybe it's not the
45:07
best way to go. And also as I said, the query
45:12
won't be enough and it's not all you need if you want to do so.
45:19
We talk a lot about training of the model and offline serving.
45:25
But if you want to do online serving, as I said in all case
45:30
we are just grabbing this data from the query and putting it
45:34
into memory. But if you your data is too big
45:39
to to be in memory you will have to use another database like
45:44
Redis like you said before or this kind of low latency
45:48
database. So for sure for online serving
45:52
if it doesn't fit in memory you will have another database.
45:56
And also if you want to have very up to date data and you
46:01
cannot do daily computation of the feature and maybe be queries
46:06
not what you did for that. Yeah, that seems like a very
46:12
respectable answer and non bias. It's like showing where it can
46:17
fall over. So really, if you're using some
46:22
kind of A use case that needs real time or very fast, fast
46:28
computations in flight computations and features, then
46:31
you have to look at a different style of architecture.
46:34
But if you're going with batch, this seems like a really nice
46:40
way to leverage something that I imagine most people have some
46:45
kind of big query type database in their stack.
46:49
And so this could be that first step until you get to a place
46:55
where now your data is too big and it doesn't fit in memory.
46:58
Now we have to re architect. And have you thought about that
47:02
forward compatibility? So once you do hit a stage where
47:08
you need to change things around, where are you going?
47:12
Next, about the word featuring process or if you want to change
47:20
a feature. If you want to so about the
47:23
architecture like yeah, when you want to evolve it, because now
47:27
you, you have different requirements, where do you think
47:31
a, what do you think those requirements will be in the
47:33
future? And B how do you want to?
47:36
Evolve, Yeah. So I think it's the same
47:41
response as before. I think the OR to mention and
47:45
she's will be the size of the data and for sure at some points
47:50
it won't fit in memory anymore. And so for that we will have to
47:58
benchmark some low latency database and to check or we can
48:05
input this data and get this data serving time.
48:08
So I think this is one of of the change we have and the second
48:12
one would be the the life featuring.
48:16
And so for that, we'll already have some Kafka ingestion.
48:21
And so we know that we could leverage that also I think to do
48:26
the computation of the feature. But my main question now is, do
48:32
we output this computation directly in the query in our
48:35
feature table? And we will have to do a lot of
48:39
insertion of the query as I said before.
48:43
And yeah, so or we will manage this data or maybe we will do
48:49
micro batch or I, I don't have the answer right now, but for
48:54
sure it's our future challenges. Yeah, yeah.
48:57
You know, it's coming. But at this point in time, it
49:00
sounds like you don't really need to focus on it too much.
49:03
It can be something. And who knows, by the time you
49:06
cross that bridge, there might be a tool out there that
49:10
services your need perfectly. All right, real quick, I want to
49:12
tell you about our virtual conference that's coming up on
49:15
September 12th. This time we are going against
49:17
the grain and we are doing it. All about.
49:20
Data Engineering for ML and. AI, you're not.
49:25
Going to hear rag talks, but you are going to hear very valuable
49:28
talks. We've got some incredible guests
49:32
and speakers lined up. You know how we do it for these
49:36
virtual conferences. It's going to be a blast.
49:39
Check it out right now. You can go to home dot ML, OPS
49:41
dot community and register. Let's get back into the show.
49:48
Well, thanks dude. This is awesome that you were so
49:50
transparent with me and that you taught me a tonne about how to
49:54
just leverage what you already have and make use of what's in
50:01
your stack to the maximum capabilities.
50:04
Yeah. Sure.
50:06
And it was a pleasure. And so I'm happy if it was
50:09
interesting for you and if you sing that it will be interesting
50:12
for the community. We'll see about that.
50:15
We'll let them give us some feedback.
50:17
If anybody out there is listening still at the end of
50:19
the episode, drop in some comments and let us know what
50:22
you thought. That's all for today.