Latest / Agentic Conversations (formally mlops.community) / Real-time Feature Pipelines, A Personal History // Hendrik Brackmann // MLOps Meetup #46

Transcript

Real-time Feature Pipelines, A Personal History // Hendrik Brackmann // MLOps Meetup #46 · Agentic Conversations (formally mlops.community)

0:16
We have the guest of honor coming to talk with us.
0:20
I really appreciate you coming here and drink.
0:23
This is your kindred spirit because our last names are
0:28
pretty much the same just One letter off almost Minds.
0:32
Brinkman yours is Brockman but you left Germany.
0:37
I think and I'm actually I'm actually stuck know in Germany
0:43
since my fight back with constant.
0:44
But yeah, usually I live in the okay, so I'm working for a small
0:50
business bank hog-tied and and leading up the data function
0:55
there. So trying to build both M and
0:58
other text and data science capability.
1:00
Lettuce. And it's been quite quite a,
1:01
right? Yeah.
1:03
And and for all of the Americans that are with us today, don't
1:07
get confused. Tide, is a financial.
1:10
It's a fintech, right? It is not the stuff that you use
1:14
to clean your clothes. There are two different Tides
1:18
which can be confusing. But now that we've got that out
1:21
of the way, I'm really excited to dive in today with you about
1:27
your journey in machine learning and typically these three
1:31
different jobs that we spoke about before and how you had
1:37
basically an ml Ops understanding and it grew from
1:40
one job to the next and you were able to iterate and make things
1:44
better and make things better and until you landed at where
1:47
you're at right now. So we're specifically going to
1:50
be talking about feature pipelines but anything that you
1:54
want to add a long, the whole ml Ops basically, Is greatly
2:01
appreciated to any insights that you've had.
2:04
We should probably start with just learning about how you got
2:07
into Tech. And yeah, how did I get into
2:10
Tech and kind of very coincidental.
2:13
It's to be honest and I always enjoyed automating things and
2:23
kind of using technology to help improve decision-making.
2:29
Help. Yeah, but data related
2:32
applications. And so I was when I was at Uni,
2:35
I was helping a company to boot, like, option pricing algorithm.
2:41
So, I originally wanted to be a Trader, like doing some kind of
2:44
algorithmic trading and then, through some circumstances ended
2:48
up Consulting with. Yeah, credit companies and was
2:54
building statistical models. For them.
2:55
At that point, I didn't even know that data science was a
2:58
thing and So I only after the fact realized that what I've
3:04
been doing all along was branded data science in kind of a very
3:07
different Community. Oh nice.
3:09
And from there, I was starting to learn more about the
3:13
different ways that people were thinking about problems that we
3:17
previously discussed from very different angles.
3:20
So my background is math econ and specifically probability
3:23
Theory. And so it was yeah very
3:26
interesting transition in that sense.
3:29
Nice. So then you went out, you got
3:32
your first job and this will be like a, I almost look at it.
3:37
Like three epic Journeys that you had along the way.
3:41
Because when we were talking about this and when I asked you
3:44
to come on the Meetup, do you explain to me your different
3:49
roller coaster rides that you've had over the past years?
3:53
And so maybe you can start with that first one, give us a little
3:56
bit of an idea of when it was because I know things have
4:00
changed. The lot since then, and then
4:03
just talk us through what was happening when you were working
4:06
there. And what, what problems were you
4:08
trying to solve all of that good stuff?
4:11
Yeah so and I think like a lot of what you're going to get
4:15
here. It's not what I think modern ml
4:17
Ox architecture looks like it's more around like things that we
4:22
did in the past in the personal earnings that I drew from it.
4:25
And so the first one was around 2014 I think and so I He's
4:30
working for a credit company there and we were trying to
4:35
Mariska says companies that were were coming in and there were
4:40
some interesting aspects to that from like technical point of
4:45
view. So the first one was the
4:48
problem. The primary one is these were
4:50
new companies that we've never seen before.
4:52
So I think a lot of the architecture that front time
4:57
were were a lot of focused on data Pipelines.
5:00
Or companies where you already have data and you later
5:02
warehouse and you are trying to, yeah, score them based on some
5:10
kind of behaviors that you already had previous experience
5:13
for me, because a lot of the data in the data warehouse
5:16
specifically, at that time had quite a huge delay behind the
5:19
the operational system. So it wasn't really possible to
5:24
use that data for your operational scores and that was
5:28
one thing. And then also the just Putting
5:30
models into production has been a lot more difficult and at that
5:34
time than it is today. So at that time, we had a back
5:38
end that was written in C sharp. And there weren't any kind of
5:41
like easy ways to build microservices or we didn't know
5:44
about them anyway, and we wrote most of our model training in
5:49
our at that point. And so we had this big problem
5:51
of and putting the model that was, I mean, the end like
5:55
something that exists in our code, into our back end.
5:59
So what we ended up doing Was he figured out there was a language
6:02
called an F sharp and which yeah it's kind of functional C sharp
6:08
and they had a concept called tight providers.
6:10
You could write F sharp scripts that were then instrumenting our
6:14
code and you could pretty much M access the same object in
6:18
there's when you could exchange objects in between each other
6:21
but it all compiled down to the the same stuff that's your
6:25
Sharpie on the compiles down to. So, and what we ended up doing
6:28
is, we were writing our training Pipelines, In our and then we
6:31
had an F sharp script that was instrumenting, the our scripts
6:35
to run all the training codes. And then we were pretty much
6:39
inspecting. So that point there was like an
6:41
extra boost kind of type of model and and we were extracting
6:49
the the different weights of the different trees.
6:52
More as many early out there pickled these these trees as an
6:55
F, sharp object and then re-implemented the, the
6:57
inference code in F sharp. In order to get some kind of
7:02
model into production so it's been quite a quite a long time
7:07
to and since since then and I think the community has come
7:11
quite a long way and but there are some sorry.
7:14
Go ahead. I expected that I thought I'll
7:16
probably still quite valuable. So what I thought was
7:18
interesting specifically for for that time is that and the whole
7:22
pipeline actually was automated, right?
7:27
So, and everything that needed to be done was running this
7:30
Shop. Stripped, she could even
7:31
schedule and then the whole like and catching a data set from the
7:35
data warehouse running the whole training code, creating this
7:37
Picard file and deploying that into what I did Point ended up
7:41
being a c-sharp library. Was all automated which I
7:44
thought was pretty cool. And yeah, on the, the data
7:49
pipelines, it was kind of interesting in the sense that
7:51
the the contract to The Wider company.
7:57
It's pretty much the C sharp interface so they That is like a
8:01
huge object over and then we use that to extract features and put
8:06
that into into our model scoring code.
8:09
And do you feel there's like was there a lot of Technology
8:14
boundaries at this point or was it mainly just the organization
8:19
itself? What do you think were some
8:21
hiccups that you were running into as you were trying to make
8:25
things better? And so I think a lot of The so I
8:30
think I technology, I honestly don't have a good idea of when
8:34
technology turn up. So I'm always not entirely sure
8:37
whether it's just, we were lacking knowledge about it or
8:38
the technology just wasn't there, but definitely, we
8:41
weren't aware of. I think the options that are
8:44
that are out there today, so we just try to figure it out and
8:49
ourselves and I think a lot of the issues and I think that's
8:53
something that I think in all of technology is, is the case where
8:58
more organizational boundaries, Than there were necessarily and
9:02
technical issues. So if you look at and
9:05
specifically the and the data pipelines, right?
9:08
So the back end was then handing us over this like a data object
9:12
and where we extracted our feature salt so what that meant
9:16
was it was a huge pain to add an additional data sources and into
9:20
our model because then we need to what maybe we managed to get
9:23
it, get the data through some kind of batch process that was
9:27
available in the data warehouse. And we managed to train our
9:31
model based on the new data but then we needed to somehow
9:34
actually get that into into the the other pipeline as well.
9:39
So the backend handing over that the data object and that has
9:44
been a very painful process, primarily not so much because of
9:48
technological restrictions by much more because organization
9:51
need the data team has been quite quite separate from the
9:55
back-end team and trying to get stories into into there.
9:59
Into their system has been like quite quite difficult outside.
10:02
And I think that's something that I still see today where
10:06
it's very hard to convince other teams to expose some kind of
10:10
data that you will need or to get them to fetch some data
10:15
that's not like directly relevant for them or not.
10:18
They're relating to to some kind of use case.
10:21
Yeah, that's something we hear all the time on here and in so
10:25
many guests have come on and said, ml Ops is an Additional
10:29
problem. It's not a technology problem
10:32
and you can have all the tools you want.
10:35
But if you run into problems like you encountered there or if
10:40
you're working on an old structure or whatever it is that
10:45
the organization is not allowing you to have the real data,
10:49
they're giving you replicas of the data and it doesn't give you
10:52
a replica of the full data sets. You're looking at an
10:55
organizational problem, you're not looking at a technological
10:58
problem. So that's super interesting to
11:00
hear and just confirm what what everyone else has been saying
11:06
any other learnings that you got from, from that job that you
11:10
want to pass along to us. And yeah, I think like speak.
11:16
I mean, in the end it comes down to the organizational problems
11:19
as well. But that also meant that was a
11:21
huge pain to actually test the, the feature pipelines.
11:24
So while it was like and technologically demanding to get
11:27
the model into, To production. Like you could actually are
11:31
quite neatly test and what was going on.
11:33
So you can pretty much generate some test data and see whether
11:36
both implementation were returning the same thing.
11:38
Doing the same thing for the, for the features, what huge be
11:41
painful. Like, I don't think we ever
11:43
really managed to do it and that architecture, just because there
11:48
was some data in databases which we could see and then the
11:50
application code was doing some magic and then in the other at
11:53
the other side we were getting some data back.
11:56
So that was it was well Quite interesting learning that
12:01
influence like later decisions that we drew.
12:03
Yeah. And that's interesting.
12:05
And you were saying you were doing our batch at that point
12:07
right. So the training data generation
12:09
was batch but then the in life inference it was a library that
12:14
we gave to our back-end team and they were then on there and
12:17
compiling some data then we were doing like additional
12:20
transformation on our end against it and which then ended
12:23
up with the like huge feature Vector, actually, even that
12:25
time, that was like, quite usual I can remember 160 or so, did we
12:28
had a bit? Oh yeah.
12:31
That sounds like fun and let us know in the chat if anyone has
12:35
worked with F-sharp because that also sounds like a lot of fun to
12:40
work with. So now then you graduated from
12:44
that job, right? You decided.
12:46
All right, I'm going to move on to bigger and better things and
12:50
we'll go into face to what exactly was happening there at
12:55
this second job? Yes.
12:57
The second thing I want to talk about And it was quite
13:00
interesting because when I came there were some things I think
13:03
we're quite nice and but some things will also like really
13:07
really terrible even worse than the first one.
13:10
And so what was nice is they had their own like a python
13:13
microservice in order to and to do the, do the predictions and,
13:19
and also the the whole model deployment with that was also
13:22
quite a lot easier. So if you're using the same
13:24
language, it was very very simple at that point.
13:28
But we also had terrible issues in terms of the data
13:31
acquisition. So what they were doing is they
13:32
were running, they were writing SQL, queries, and stored
13:35
procedures in like in reach replicas of some kind of
13:38
back-end database. In order to create the, the
13:41
feature vectors it was incredibly slow.
13:44
So it took like minutes to create features which like it's
13:49
really not acceptable and but there are some interesting
13:53
aspects of that as well. So what they did is they they
13:57
happy SQL queries, that the road creating inference and features.
14:02
And then the they parameterised, the SQL queries by pretty much
14:07
injecting like ideas and time stamps to create the historical
14:12
data set. So while I think like from a
14:14
kind of computational efficiency standpoint and from a like,
14:17
separation of concerns then point and was really terrible.
14:20
But what was very interesting in in there is that pretty much
14:24
agree using the same code in order to To generate your
14:28
training data sets and generating your your inference
14:34
coat. I thought that was like a very
14:36
interesting idea and something that we try to do to keep also
14:41
afterwards so we later on underwent a proper separation so
14:46
all of that was was killed and we started adopting the back and
14:51
specifically was starting to use and domain-driven design and
14:54
they started to communicate through events with each other
14:57
and then so We managed to migrate all of that to an event
15:01
driven design and that actually worked worked quite well where
15:04
we could, like, have the same idea of having, like local
15:08
projections in that case against the events and then generate our
15:13
training data and by a pretty much doing the same thing of
15:19
creating some kind of transformations of historical
15:22
events. Not only of the events that are
15:24
currently there. And was that how you did?
15:27
You eventually cut? On the time that it took.
15:31
Yeah, we did. So in the end it was, I mean in
15:34
the end it wasn't really significant because we pre
15:37
calculated everything. So the actual time was like,
15:43
less than 100 milliseconds but also that, that wasn't isn't
15:47
quite true. In the sense that we pre
15:50
calculus because it was event-driven, we managed to pre
15:52
calculate everything and when it was needed it was just to look
15:55
up in a database. So there wasn't anything
15:57
happening there. Okay.
15:59
And for those of us that are a little bit foggy on the
16:05
terminology of Transformations, keep break down, what that is
16:08
and why it's important. Sure, what is the
16:14
transformation. So and you have like some kind
16:19
of raw data which can really be be anything.
16:22
So can we I think, what when we looked at it was mostly like
16:25
some kind of Json documents so say like You buy a credit report
16:29
which nowadays comes in terms of for Jason, you get some data on
16:35
like what are crime rates in a particular area?
16:37
You get some data on what other things was there.
16:42
Like I'm like some kind of risk in is that is associated with
16:46
like an email address or something this and usually some
16:49
kind of Jason and then you would want to extract turn that into
16:53
numerical values that you could use.
16:55
So that could be just extracting like a value.
16:58
From a lack of credit report. It could also be things like
17:04
what was the difference between events that were happening to
17:07
hold on? Whom did whatever Lex particular
17:09
screen take and or yeah any kind of like at that time we didn't.
17:18
But you can also use like some kind of a vectorization
17:23
techniques to extract data from yeah, like natural.
17:28
Just so you really try to to turn raw data sources that you
17:32
have available into something that is available as input to
17:36
the machine learning model. Okay?
17:39
Yeah. And that was basically my next
17:41
question was why this is important for the model and why
17:46
it needs these transformations to happen?
17:50
Well, it depends a little bit on what your notion is of model,
17:54
but the, the classical version pretty much takes and numerical
17:59
values as inputs. And with that idea, you pretty
18:07
much have to ensure that all the like what's usually text Fields
18:11
gets and then somehow recorded into numerical values.
18:15
Perfect. Perfect.
18:17
So I've been listening to everyone's feedback in the
18:21
community, I just wrote a ask people to fill out a form on
18:26
what we've been doing good and what we've been doing, we would
18:30
like to see us do more of, and some people said, break it down
18:33
real slow. So I'm trying to listen people,
18:36
I'm trying to get it broken down in a way that everyone from the
18:41
newbies to the expert. People can do it and they also,
18:46
Asked for Q&A sessions. So just in case you don't know,
18:51
feel free to throw any questions of yours into the chat section
18:57
or there's an actual Q&A. That is there.
19:00
And we will try and get to them as fast as we can.
19:04
So, moving along, you did that. Did you feel like there was a
19:08
graduation of sorts from your first job to the second job and
19:13
what were some key things like actually.
19:16
I think what I'm most interested in is what are some things that
19:20
you're like? I'm never going to do that again
19:25
- mm. I think this whole idea unlike
19:28
and having like SQL queries, run against operational.
19:33
Databases is or at least. And this like feature
19:38
calculation and against operational database is really
19:42
horrible one that I've never want to repeat in my life again.
19:46
When those come to a lot of headaches boss from
19:50
computational point of view but also from just separating out
19:54
contracts. So there was also pretty much
19:58
the backend couldn't change their database without breaking
20:01
our code, which is really horrible.
20:03
Hmm, and so that's a definite learning.
20:07
I think I learned a lot of team composition so we discussed
20:14
already a lot of Williston Issue where we are now, I think
20:18
drawing the line between like, we're back and starts and where
20:20
data starts where they has a like wider scope.
20:25
I think we're, where do you feel like that?
20:29
Is because that is something that we talked about a lot and
20:34
the idea of one person being able to run the whole Spectrum
20:38
or an ml Ops engineer really. For me, I'm not sold on that
20:43
idea yet. Like that shouldn't be just one
20:46
person. His job.
20:47
So how did you see things more clearly?
20:52
So I mean ideally, you have this one person who can do
20:56
everything, right? But I think it's just extremely
20:59
hard to find these people. So I think what, what usually
21:03
works much better is and just trying to build teams that like
21:08
have very diverse skill sets. So where we now, have
21:11
application engineer State engineers and the scientists
21:14
grouped together in teams And so that there's much much better
21:19
and communication between them and then, yeah, like trying to
21:25
do like the end product that these people are supposed to
21:27
produce is a service that is running in production and that
21:33
service is not supposed to take like a lot of data directly but
21:37
is supposed to pretty much taken an idea and then return like
21:41
some kind of prediction. Let the whole data pipeline in
21:44
your how do we get the data is the data?
21:46
Actually of good quality. And so on is always in the
21:48
control of the team and because in the end that is what makes
21:53
the team successful, right? So if some some other team is
21:56
has ownership of a data Pipeline and then breaks like it
22:01
influences the performance of some other team which is really
22:04
not. Okay.
22:05
Right. So then let's a team can't
22:07
directly succeed just because there's there's some external
22:10
dependency and I mean in any organization you never get used
22:14
to perfect but we Try to at least an integrate all of them
22:18
into one team, that's awesome to hear that.
22:22
And that's a great explanation to, and that's something I guess
22:27
for another day. And another topic that is when a
22:31
model breaks and it's out in production, who owns that.
22:36
So, the, the next question I guess, or the natural
22:42
progression of this is going to be that you I moved on from that
22:47
job and you landed at where you are now at tide?
22:51
Is that correct? Yes.
22:52
So the the last team thing was actually referring to tight
22:55
already. So okay, and ice.
23:00
But yes, and so at that point it was kind of interesting in the
23:02
sense that we were like creating a lot of teams and quite like a
23:06
high pace. And there were a lot of like
23:09
issues arising from that. But what are we trying?
23:16
To do different this time. So I think the first thing is we
23:21
are trying to really stick to the event driven design.
23:24
I think that was a very good design decision so where we are
23:29
aiming to create a lot of the features that we use for machine
23:35
learning models based on the events that are somewhere
23:37
created in our back end. And then also, as much as
23:40
possible at least use that in order to build our historical
23:45
training data sets. And the special thing you learn.
23:48
Sorry to interrupt. This was something you learned
23:50
at the last job. Yes.
23:52
So I think but I think we scaled it up quite significantly so in
23:57
the, the the first one we only and so a lot of it went through
24:07
through one model and we're here, we are reusing features
24:11
across different models, which is I think, like, That we
24:17
previously haven't explored as much.
24:19
And and we also started doing that like, using the data
24:27
warehouse as a as a primary data source.
24:31
So, in the previous companies and that wasn't really possible
24:35
because it was pretty much a new company comes in and we needed
24:39
to find like some kind of score against them.
24:41
Whereas here we also have at least number of applications
24:45
where there's a lot of information.
24:46
Nation, which doesn't really need to be fresh.
24:48
So we tried to join these two together, that makes sense.
24:54
Yeah, yeah. That makes total sense.
24:57
So we're getting a question coming through in the chat and
25:00
it's Arizona, is asking, have you only worked with teams with
25:04
one programming, language? They're saying there's has three
25:10
and two that are not Python, and it's difficult.
25:16
It challenge to say, the least, for the ml Ops engineer on that,
25:21
was that every your case, I mean, I know that on your first
25:24
job you were working with r and C sharp, right?
25:28
Yeah. So and one of the nice things
25:31
is, since we started the team from scratch, we could
25:34
standardize on python which was very nice on the first job.
25:37
It was and C sharp. In the second one, the back end
25:40
was written in Ruby and we had our own Services there and in
25:43
the third, in our back end is written in Java.
25:48
Yeah it is a pain, right? So what can you say and the the
25:54
only things that I think I think at some point you just need to
25:59
have some kind of obstruction against the microservice.
26:02
So there are already an like tools out there.
26:05
Such as Sage makers that make it reasonably, easy nowadays to
26:09
just spin up an API against some kind of model, but I feel trying
26:15
to do To translate chords directly is is a very painful
26:23
undertaking. If there are like
26:25
implementations of the same like sometimes it works out.
26:27
Sometimes you have implementations of the same
26:30
libraries in multiple languages in which case the Handover isn't
26:33
terrible. But personally, I found it much
26:35
much easier to just have some kind of API extraction in
26:38
realize, doesn't really matter what the languages are.
26:41
Yeah. Yeah.
26:43
And that, I mean, we see a lot of Of these days.
26:47
I feel like in software engineering it's not so much a
26:51
language thing as much. I mean, I see another question
26:54
coming through in the chat. That's asking what's your
26:58
language of choice in ml Ops So yeah, I think like for the, the
27:05
actual Services Python and then for anything where you actually
27:08
are feel like high data volumes, its Sparkle pie spark.
27:14
Okay. Nice.
27:16
All right, let's keep moving. Let's stay on tied for a little
27:19
bit because I know you've got a lot of cool stuff going on
27:22
there. Maybe it would be worth you
27:24
telling us what exactly is going on at tied?
27:29
What are your use cases? And why is it?
27:32
It that like how has this reusing of features with
27:35
different models? Enabled, these use cases, Sure.
27:41
So the ones that spring to - we there a lot of risk related
27:44
ones. So, and trying to assess the
27:50
risk of onboarding to fraud risk, credit risk, invoice, and
27:53
default risk these kinds of things.
27:56
I think, one of the things that I thought was quite quite
27:58
interesting, is we also have a model that tries to match
28:06
invoices to transactions. So whenever you And you do a new
28:11
trend Trend, whenever you have an invoice race against the
28:15
company, you would like to know whether this invoice is getting
28:18
paid. So we are trying to figure out
28:20
for every new transaction that comes in.
28:21
Like, is there a particular invoice that actually pays off
28:24
that transaction or is being paid off by their transaction?
28:29
And in order to help pretty much smes to?
28:35
Yeah. Reduce the, the manual override
28:37
the manual admin by not having to do that.
28:39
Themselves. And I think, the what we've seen
28:44
is that specifically in the, the risk cases, but interestingly
28:46
also, for the invoice and matching one, there were a
28:54
number of features of yet, like, Risk related ones, that could be
29:00
reused. So things such as like invoicing
29:05
patterns or never some transaction patterns that
29:11
Identify the source case. Well then we could actually use
29:14
in different models as well. And I think what, what is
29:20
probably worth mentioning there is in terms of the the feature
29:24
pipelines, one of the things that we did what we started off
29:28
building ourselves but then in the end, when for commercial
29:31
provider is and building up some kind of feature store.
29:35
So we like, as I think, I said previously, the event room
29:40
approach was useful for building our own pipelines and keeping
29:44
the managed but it's actually It even more useful to, to store
29:49
them in some kind of database that can be easily accessed.
29:54
And the reason for that is a lot of is computationally.
29:57
So you want to know, you don't want to do all your feature
30:01
calculations at the point when you want to make a prediction,
30:04
but you do want to do that before and then also you would
30:08
like to. Yeah.
30:12
Fetch. Like they all might have been a
30:14
different point. Sometimes you ideally want to
30:16
standardize on Some particular point in time as well.
30:19
Yeah, future stores are such a buzzword right now in the
30:24
community. I mean, it's not a day that goes
30:28
by that. We don't talk about him.
30:30
I just actually put a video on feature, stores up on LinkedIn,
30:33
because we made some animation videos.
30:36
And I feel like that was the biggest buzz word that along
30:40
with maybe monitoring, which is personally, my favorite because
30:45
it's such a difficult problem to attack and what do you Monitor
30:48
and and how? And And how do you make sure
30:50
you're not getting alerts 24/7. But feature stores are are also
30:54
right up there. It's one of the most talked
30:57
about things in the community so what made you I am interested to
31:02
know like did you try any open source Solutions?
31:05
Or did you just go right out and buy it?
31:08
So we initially and we initially actually started building our
31:13
own but it was actually taking significant time to to scale up.
31:19
And we briefly looked at some open source Solutions.
31:23
And the problem with that is that I think the the scope of
31:27
them were slightly different. And so, I think a lot of the
31:31
open source ones are more like feature, Registries where your,
31:35
her really have to have pre-calculated and your
31:38
features, and then you could register them both in some kind
31:41
of like historical database or some kind of real-time database.
31:46
But what we really wanted to Surf, like some kind of easy way
31:48
of generating new features and that you could then use for.
31:55
Yeah, real time. And as well, for the training
31:58
data generation. So for some kind of batch
32:00
process. So we in the end when for
32:03
product called tacked on that was quite, there was doing that
32:07
out of the box. Also quite new at that time and
32:10
we've been quite quite happy with that decision in particular
32:13
in the sense that it A and reduce the time that we spend on
32:19
generating additional features significantly and then also
32:24
quite interestingly, we see that a lot of the work on generating
32:28
features, even for real-time applications, actually, we the
32:32
data scientists were able to do themselves to speaking.
32:34
Again, of the organizational challenges that previously has
32:39
been done by the engineering team at now, could be done by
32:41
the the data teams for all the data science team.
32:45
Oh, that's yeah, that's a nice benefit that I hadn't even
32:48
thought of. And so when you're looking at
32:52
your different pipelines and you're looking at your future
32:53
pipelines, how do you decide on which pipelines are going to be
32:58
batch? And which ones are streaming?
33:00
And maybe we should preface this by asking you, what does Real
33:06
Time mean to you? Because I've had a few people on
33:10
here and they say, yeah, real time I was talking to somebody
33:13
and they said, oh yeah, we need everything.
33:15
Real time. And then real time to them meant
33:18
they needed everything once a month and so it was a ridiculous
33:24
when they looked at him like, that's real time, what that
33:26
doesn't mean? So what is real time for you?
33:29
How much latency is there? Or how much time do you need?
33:35
What does that mean to you? So it varies a bit from
33:39
application to application, but I would generally think that
33:44
everything less than is Second I would consider real time and
33:47
everything above a second. I would consider near real-time.
33:51
Okay? So there are some applications
33:53
where we actually have how or use cases, where we have hard to
33:56
deadlines. Most notably transaction
33:58
monitoring where and there's an external party doing an API
34:02
call. And we need to come back and say
34:03
like this is transaction, okay. Or not within 500 milliseconds.
34:08
And a lot of the like even structured streaming or some
34:13
doesn't really like work for that properly.
34:16
So you need to get in some way, pre calculate the features for
34:20
for these applications and then have a reasonably lightweight
34:23
process to update that with the information that actually comes
34:27
from the, the transaction itself, which is the new piece
34:29
of information. Yeah, totally.
34:31
So so then it's only when you really need to do it because of
34:38
some third party that you do the real-time and the rest is bad or
34:43
near real-time. Or is there certain like because
34:47
I've heard that it's always easier to do batch.
34:51
I don't know. You can tell me if that's right
34:54
or how you feel about it. And I think it's simpler to to
35:00
reason about and for the most part and oftentimes and like,
35:07
this isn't always true, but oftentimes badge,
35:09
Transformations are stateless. So you have just like some kind
35:14
of piece of data, and then you do some kind of transformation
35:16
and you end up with the the new piece of data and having state
35:19
is, Transformations is very simple to reason about.
35:23
Whereas, if you have like some kind of event driven system, you
35:25
have all these questions of law, It's conditions.
35:27
And what happens if one event happens before the other event?
35:30
And then we need to keep track of state and so on, which makes
35:32
things like reasonably complicated, I think.
35:36
And so in terms of like how do we decide.
35:41
So I think in general everything that works through batch, we try
35:45
to do through match, but if it doesn't work then, yeah, I mean,
35:49
it doesn't work, it's more use case related really than it is
35:52
anything else. and so I'm just thinking like, when you're
35:57
building these real-time distributed feature pipelines at
36:03
scale and you encounter failures How do you deal with that?
36:11
I mean, when you say failures, you mean technical failures or
36:16
you mean like that? Yeah.
36:17
Like mainly a failure with like the pipeline and or anything.
36:22
Yeah. Technical failure that you maybe
36:26
you got to find a bug and you're not really sure where that bug
36:30
is or you have to go and be a detective and find where it's
36:34
coming from. and, So yeah, okay, so if there's an extra so we
36:42
have an we have Engineers on call that would generally
36:48
investigate that and as you say, it can be, quite quite confusing
36:52
to understand where the, the error actually, Originators and
37:00
in the end, we are trying to lock the different parts and
37:07
between our are transformations. Yeah.
37:09
So again, one of the nice things about the event room design is
37:13
that you can actually look very clearly at the inputs to your
37:16
functions and it doesn't always work because sometimes the the
37:21
local state might be updated in there might be some kind of
37:24
weird inference. But yeah.
37:27
And so as you're looking at, like going back to the feature
37:31
store bit, let what stage did you realize or did you want to
37:37
actually get a feature story? Or was it something that you
37:40
came into tied with or was it once you realized?
37:44
Okay, we're going to go with this event, div driven design.
37:47
And the best way to do that that we can see of right now is this
37:52
thing called the feature store. And so, I think that will
37:54
multiple reasons, so one of them was strategic where we are
38:00
trying to in the end, automate every decision within tight.
38:05
And I think the, the way this would work is by reducing the
38:09
Cost per model and with that the cost per feature as much as
38:12
possible. And and that I thought, if you
38:18
look at the cost per model, really a lot of the time goes
38:21
into building the data pipelines.
38:24
So trying to reduce that I thought and we would need to
38:28
have like some kind of yeah feature store solution and Which
38:35
was the primary motivation and that point makes sense.
38:41
And yet, I think it's, I'm still quite quite happy that we went
38:45
down that road. Mmm.
38:47
So then when your, because you've been through these three
38:51
stages and have a look at them in, it's like, roof you were in
38:56
the trenches for a while and as you can recommend to others.
39:01
Now, going through this and seeing these different styles or
39:06
maturity levels of feature pipelines, do you feel that?
39:11
That it's like, how would you recommend when to get a feature
39:15
store or what to do with that? That whole concept because it is
39:21
such a hyped up item right now, right?
39:25
And so, I think it's easy for a lot of people to say, oh well, I
39:29
just need a feature store and then you add.
39:32
I saw someone in the community, drawing a diagram of the Google
39:37
maturity level 2 and then adding all these Tools, that would be
39:43
each piece and if you don't know what what I'm talking about,
39:46
Google put out a paper and given different maturity levels and
39:50
basically how automated your machine Learning System is.
39:54
And they put in the community. This person put this diagram
40:00
with all these different pieces and what they were going to do
40:03
and I was looking at it like wow.
40:06
That's amazing. Good luck.
40:08
How much of this do you already have in place?
40:11
Right? Because it could be crazy to
40:13
setup and especially if it's a small team it could be very
40:17
difficult and take a lot of time.
40:19
So I guess the real question here is you with all your wisdom
40:25
what would you recommend from a maturity standpoint?
40:28
When is that Breaking Point like you should go out and get a
40:31
feature store? So I think the question is gold
40:35
and get a feature store. So I think like as an
40:39
organization you should try to get to A feature store, but that
40:42
can mean very simple things, right?
40:44
It can mean, just having a process of where you create
40:47
tables in your, in your data warehouse, then have a certain
40:51
structure that makes it very easy to extract feature values.
40:56
I think like looking into commercial like solutions to me,
41:00
I think the the only real purpose that is if you need to
41:05
somehow calculate features in real time.
41:07
So if if it's not okay for you to have freshness of features
41:12
that that your data warehouse, can't come facilitate.
41:19
And that to me I think would be the boundary of when you would
41:23
move to to some kind of commercial solution.
41:25
Mmm, so it's really for me around like a freshness
41:29
requirements. Yeah, yeah.
41:31
That's a great answer. And so as you're looking at like
41:35
the feature store, can you summarize like some of the
41:39
greatest problems that the feature store solves for you?
41:44
So I think it's and well reducing the effort that goes
41:49
into creating a feature and putting it into production.
41:53
So reducing the cost per feature and that sense and also reducing
41:59
a hand over our requirements. So I think said earlier as well
42:03
that and now a data scientist can create a features themselves
42:07
rather than having to rely on data engineers.
42:11
And I think there's also a mano During a is much easier.
42:15
If there's one system through which, all your features go and
42:21
basic stuff out of there as well.
42:23
Yeah, that that is also another big thing monitoring that and I
42:28
know that is that is huge, a huge piece of it.
42:33
And then there's a great question in the chat that just
42:36
popped up. It's what problems does a
42:38
feature story introduce if any and so I mean with any
42:44
additional piece of your stack, you are adding an additional
42:49
complexity. So, and I think there's a piece
42:54
on like your in a way and tightly coupling like a lot of
43:01
your Production Services to one particular pipeline, right?
43:04
So in theory of one of these pipelines would go down then
43:07
would create quite a lot of like Downstream impact and then
43:11
they're kind of like subtle iterations on that point as
43:13
well. So specifically, like versioning
43:17
is kind of, like somewhat painful or features where well
43:21
it's not like they actually like process solutions to that as
43:24
well. But you need to at least think
43:25
about it. And where And if you like want
43:29
to make a change to a feature, then you can't just make a
43:32
change to the Future out because that would like create like
43:35
create a different distribution of the future with then would
43:38
mean that your models are not adequate anymore.
43:41
So like and that I think becomes a little bit more difficult, but
43:47
I think the solution is also not particularly difficult in the
43:50
sense that you just don't do it, right?
43:51
You just write new features for every for every iteration, but
43:54
it is something that that needs to be actively managed.
43:58
And and create some danger in that sense as well.
44:00
Yeah, if people don't follow process.
44:03
Yeah, the tightly coupling is, when you talk to Engineers about
44:07
that, they go and so I've got a great question from David here
44:15
in the chat and this one's going to make me David, you're gonna
44:19
make me read a little bit, huh? So, how did you approach Dynamic
44:23
features like rolling? And there's a lot of acronyms in
44:26
here, like, In ew, ma there's probably a special word for that
44:31
which I don't know, dwma online filters or even RN.
44:38
N type models as features and I am thinking about features that
44:44
you will not store permanently in a feature store and might
44:47
involve some State like in a rolling mean or RNN embedding.
44:53
I guess I'm asking about, where does the model end?
44:58
And the feature start from the perspective of the system and
45:01
the engineering as a great question.
45:04
Yeah. And I don't have a Your Grace
45:09
great answer to that and It depends, right?
45:16
And so if you think that these, so I guess in the end, what what
45:21
the this question comes down to is that what has become like
45:25
common practice I would say, is that this kind of stacking of
45:28
models, right? So, the output of one model is
45:30
the input of the kind of the next model.
45:32
So that's it. So in some sense that means that
45:35
the output of one model is also again a feature.
45:38
And it's really a kind of, like a design decision, right?
45:42
So if you, if you think that this feature will be likely
45:46
reused, then bye-bye other things, and then maybe it makes
45:51
more sense to, again, put it into the feature store.
45:54
So it might be reused by other stuff as well.
45:57
And it also depends on how your training is set up, right?
46:00
If you're trying to train, like if your whole training pipeline
46:05
is pretty much training the first Model.
46:07
And then the second model on top, then probably it makes
46:10
sense to not have it in the future store but having more
46:12
closer to your your model. And if you do that, by the way,
46:18
then you have this thing that your features are not strictly
46:21
numerical anymore. But you also have like other
46:23
feature types, which again, is not really a problem but you
46:27
just need to think about them. It can have different types as
46:29
well and so, I don't know if it's not really an answer, but
46:32
it's more David or let us know in the chat if that was at a
46:37
quitter. Not so another great one.
46:41
How many features does the input data have prior to the store in
46:45
an individual industrial setting?
46:48
You might have several hundred features already.
46:50
So what a feature store their serve more to highlight, what
46:55
seems important? I'm not entirely sure I get the
46:59
question to me to me Frank. So how many features does the
47:02
input, the idea to have prior to this?
47:05
Door depending on which store you use, either.
47:10
It's already pre-calculated as feature.
47:12
So it's just the same number like you can, you can have
47:14
different processes pushing in different features, right?
47:17
But it would just be the same if you're using similar to what we
47:20
are using where the calculation or the transformation happens.
47:24
And as part of the future store, then the input data doesn't have
47:27
any features, right? So you just ingest raw data and
47:30
transform it into features. And then the second question.
47:34
So in an industrial setting you might have several hundred
47:37
features already. So what if feature store their
47:40
serve more to highlight what seems important and So, if
47:47
features, so one of the nice things that you often times get
47:51
like basic documentation with that as well.
47:53
So you can actually not only see like what features does your
47:57
organization have. So the discovery piece but you
47:59
can also see what does it actually mean?
48:01
And not, not yet, but maybe it might even make sense to have a
48:06
like, very clear understanding of where it was, particularly
48:08
useful as well, mapping in this, this additional metadata.
48:16
But yeah. So the the documentation, peace
48:18
and Discovery peace is something that I would see as part of a
48:23
feature store as well. I, I just remember that I
48:30
interviewed Kevin the CTO of tecton in June last year and I
48:36
asked him. Hey, do you have any like badges
48:40
for people who create the feature that cost the most money
48:44
or the feature that is the most used or things like that?
48:49
And he was like, no, we didn't do that.
48:51
Dude, I don't think he thought that was something to spend time
48:55
creating, but But I think that would be awesome.
48:57
Imagine you could have a little little bit of challenge with
49:02
your team to see who makes the best feature and who has to go
49:06
stand in the corner because they created the most expensive
49:09
feature. Yeah.
49:13
So yeah, we're getting through. So, okay, to have a data store,
49:18
you really need good data management and owners of the
49:21
various features. I mean, I don't think it has to
49:24
do with having a data store. But I think in Kind of like
49:27
growing organization, you would need to have good data
49:31
management and data owners as well.
49:35
This is more really a problem of scale rather than technical
49:39
issues and foods can can help with that, in the sense that all
49:45
of that is in the end metadata, and you would want to have it
49:47
all in one place. So, if you have a feature story,
49:51
is very good, if you can pretty much see and at the same place
49:54
where you can see the feature and the it is welded to find the
49:57
feature or two metadata such as the description and an owner.
50:01
And but yeah, I I think it's something that just comes with
50:09
scale. Like you need to have like very
50:11
clearly defined ownership of things, really, not only data
50:15
and it's also just becomes impractical to to find the data
50:21
that is relevant for you specifically if there's no like
50:26
Discovery process flashtool. Yeah.
50:29
Alright, last question for you, I am wondering oh, I just see
50:34
another one came through in the chat.
50:35
All right. Second to last question.
50:37
Have you ever used or built any features doors that explicitly
50:40
handle units and does dimensional analysis on derived
50:45
features? I don't, I don't think that
50:49
makes a lot of sense to me, to be completely Frank.
50:54
And so the Okay, let me so personally, I don't think it
51:01
makes a lot of sense because in the end, it goes into some kind
51:03
of model and it should really be part of the model Pipeline and
51:06
to standardize the future in whatever way they want.
51:10
And what's interesting is, well, what is more important is that
51:14
the training Pipeline and the, the inference pipeline have the
51:19
same units? So are and that you ideally
51:24
ensure by executing the same code in both areas, Has then it
51:29
is to, to decide which unit is actually more important.
51:35
It's also, it's also interesting actually with bucks, right?
51:37
So if like, one of your if you have a buck in your training
51:40
data Pipeline and in your inference pipeline, if it's the
51:43
same buckets, probably actually, not that bad for the for the
51:45
whole system because both of them are at least the same.
51:48
It's probably not predictive or it might be predicted for some
51:51
reason. Yeah, I like it.
51:57
So now I want to jump on to the last question, which is how you
52:01
deal with security and how do you navigate these security
52:06
concerns as you're looking at feature Pipelines.
52:10
And so in the end it's mostly access rights and that you can
52:17
well at least the way we are currently doing it have for for
52:22
certain features and yeah I think so.
52:28
I think that security is quite a quite a wide topic is this
52:32
related to like operation concerns or is it organizational
52:35
would be able to get some more operational?
52:40
and in this sense of I'm thinking more about like these
52:52
data pipelines in this data coming through and how you're
52:57
making sure. Yeah, there's the is there like
53:00
our back and you're making sure the right people have access to
53:05
this. Because I'm just remembering
53:07
when you said on that first job that you were at and how you
53:12
weren't getting access to the right data and that was an
53:15
organizational problem. And then could that have been
53:20
fixed, or has it been fixed now, where you're at and how have you
53:24
dealt with that problem? Organizationally I guess. so
53:30
that people are, okay, with giving you the data Okay so
53:36
maybe a couple of things on that.
53:38
So I think and we are currently hiring a better better
53:42
governance manager. And what I think Miami General
53:46
approach is and you can access most data as long as you somehow
53:53
obscure it. And oftentimes, you can find
53:57
some kind of way in which obscure data is still useful.
54:01
In some sense, if you know, You're looking for and I think
54:07
the the key there is more and getting ahead in terms of
54:11
proactively, discussing with the, in our case of the
54:15
compliance team like how you can actually do that rather than
54:21
having them come in late. And then tell this is not, okay,
54:24
this is not okay and so on because that everything you've
54:27
been working on for the last couple weeks, just goes out the
54:30
window. Yeah.
54:32
So I think it's more like Proactively engaging in some
54:36
kind of discussion with them and then finding some kind of
54:39
compromise as well, where it's clearly, not okay?
54:42
To like directly exposed some kind of like personal data, but
54:48
you can usually obscured in some sense that you can use it.
54:53
So, I lied, it looks like the people want a little bit more
54:57
from you. If you have time, there is one
55:00
last question in the chat. What kind of models do you have
55:04
up? At the moment.
55:05
And how do you test and monitor deploy the models?
55:11
And so, this quite a lot of questions.
55:14
So we have a variety. So from the traditional booster
55:19
trees, some kind of neural networks, also have some
55:24
semi-supervised and techniques as well.
55:29
How do you test and monitor the climate models?
55:31
So, testing, there are multiple steps.
55:33
So the first one is After you write some kind of model testers
55:38
unit test. Then there's also and
55:40
distributional test is the accuracy am above a certain
55:44
threshold and depending on, on the use case, or whether it's a
55:51
high-risk use case or not, and we then either put the model
55:55
directly into production in the early testing way, or if it's
55:58
not, so if it's a high risk or sorry.
56:01
Yeah. So if it's a high-risk used
56:02
case, we then have something Which we call shadowing where we
56:06
encourage much. Like if the there's a prediction
56:12
requested we just lock what we would have done with the new
56:15
model and then we analyze the distribution of the new model to
56:18
understand whether it's it's accurate and or not.
56:25
And and then in terms of how do we monitor deployed models?
56:29
So specifically, I guess this is a distributional.
56:32
So there we lock all Our predictions as well and then at
56:36
the moment it's the we built a dashboard for every model so
56:41
it's not. Yeah streamlined yet.
56:44
But yeah. So we have like every prediction
56:50
service creates events based on the predictions that have been
56:53
done which are then end up through various processes in the
56:57
in order to warehouse and we build some kind of dashboarding
57:00
on top of that. Yeah.
57:03
Which doesn't Happen automatically at the moment, but
57:06
that's what we do. Awesome, thank you so much for
57:10
coming and talking to us about this evolution of your ml Ops
57:14
journey. I find it fascinating to see
57:17
what you've gone through. And again, you were in the
57:20
trenches. You were trying to iterate and
57:24
just evolved this Evolution for me is incredible and really
57:28
looking at the evolution of the feature pipelines.
57:31
And then talking about the feature stores, which I A lot of
57:34
people have lots of questions around because it is such a
57:39
buzzword. It's been super insightful for
57:43
me. I want to thank you again and
57:46
everyone out there. If you're not in the slack, go
57:50
ahead and jump in it. We will see you on there.
57:52
If you want to continue the conversation, I know you're in
57:55
it. So, if there's any questions
57:57
that you didn't get to ask in this chat with Hendrick, then
58:03
you can ask him on. Lack.
58:05
All right. Y'all.
58:06
Thanks again hendrik. Yeah, looking forward to hearing
58:09
your questions.