Latest / Agentic Conversations (formally mlops.community) / MLOps with Databricks // Maria Vechtomova // #314

Transcript

MLOps with Databricks // Maria Vechtomova // #314 · Agentic Conversations (formally mlops.community)

0:00
So I'm Maria, I'm Amelops technique at a holder.
0:04
He's I'm also a Co founder of marvelous Amelops company where
0:08
we teach other people about machine learning Amelops and
0:12
Amelops with etheric specifically as well.
0:15
I'm very critical when it comes to my coffee.
0:18
You probably know about that already.
0:20
I like to drink latte with oat milk and you need to have a
0:27
perfect amount of coffee, perfectly grinded with a
0:31
perfectly micro audio got found. So yeah, I can make a better
0:38
coffee than most coffee shops. We are back for another ML OPS
0:43
Community podcast. I'm your host Dimitrios, and
0:47
today I have the pleasure of speaking with Maria from
0:50
Marvelous ML OPS. If you are in the ML OPS
0:54
universe at all, you probably have heard of her.
0:57
This episode does a gigantic deep dive.
1:01
I cannot stress enough how deep we go into Data Bricks.
1:05
The pros, the cons, the good, the bad, the ugly, everything
1:08
about it and stick around. At the end, she talks a bit
1:12
about her course that she's going to be doing and also the
1:15
book that she is writing. So if you've ever had the urge
1:20
or if you are currently working with Data Bricks and you're
1:25
doing some kind of MLAI, you should probably get in touch
1:30
with her because she is by far the expert in this world.
1:35
Let's dive into it. Everything you need to know
1:38
about Databricks and ML OPS on Databricks.
1:48
The last time that I spoke with you, you were telling me how.
1:51
Yeah, I think I'm going to start posting more on LinkedIn.
1:55
And oh boy, did that work out. Yeah, indeed.
1:59
Yeah, it was two years ago. Actually, it was exactly 2 years
2:04
ago we recorded the first podcast together and a lot has
2:09
changed since then. A lot.
2:11
Well, what have you been up to? Well, I think two years ago I
2:16
had this urge to start posting but what I do on Ding Ting and
2:21
also writing articles because I have seen so many experts on
2:27
LinkedIn and everywhere, you know, talking about the mellows
2:30
and what it is. And I thought that a lot of
2:33
things that are there is just not true.
2:35
And I've been doing this for many years, so probably my voice
2:39
also should be heard. And well, I kept going since
2:42
then. It's really, really cool.
2:44
I met a lot of awesome people and yeah, I love it.
2:49
And you focused, I think the majority of your not necessarily
2:55
your content on LinkedIn, but really the content that you do
2:59
beyond LinkedIn deep diving onto data bricks.
3:03
And I think I would love to talk about data bricks, like why you
3:08
chose that? This is not sponsored by data
3:11
bricks by any means, although they are a proud sponsor of the
3:15
MLB community and we love working with them.
3:17
But this specifically, I think I want to talk about the good, the
3:20
bad and the ugly from somebody who's a user, from somebody
3:24
who's done the majority of their certificates.
3:26
And also now you're giving courses on how to better use
3:30
data bricks. I would love to just start with
3:34
like why you even chose data bricks particularly?
3:38
Yeah, yeah, that's, that's really a good question.
3:42
So I've been doing ML OPS for a very long time, like more than
3:46
eight years before it became a thing.
3:48
We built our own tooling around model registry experimentation,
3:52
tracking and using some, you know, Teradata and databases or
3:56
any other SQL database as a back end and G frog artifactory for
4:00
model artifacts storage. And now we were doing it like
4:06
when no one was doing it. And it was really, really cool.
4:10
And since then, I tried doing the ML OPS with all kinds of
4:13
different tools on the clouds on Prem and well, as I said, build
4:18
tools myself. And I, I feel like I've seen it
4:21
all to a certain extent, like how do you connect different
4:25
pieces of your ML setup to make sure it's robust and you can,
4:30
you know, rollback things whenever needed.
4:34
And there are a lot of platforms and there are a lot of tools.
4:37
So when you look at the tools, you can find almost the perfect
4:42
tool for, you know, for each of the components of your ML setup,
4:47
but it is complicated, you know, combining them all together and
4:51
you need to invest a lot into into that and have a big team to
4:56
be able to do that. Platforms on the other side,
5:00
then that's customizable. So there's they have all these
5:04
MLS components which you need like model registry, compute
5:08
serving component, monitoring component, things like that for
5:12
data versioning. They have them all.
5:15
They're not perfect, but he still may need may want to get
5:20
it working for you. Yes case instead of you know
5:23
trying to combine all existing tools.
5:26
If you're on Prem, you probably don't really have a choice.
5:28
You have to go open source and then you have this luxury of
5:32
choosing. But when you are on the cloud,
5:35
if you are in a large organization, bringing in any
5:38
tool is really, really complicated.
5:41
You know, all these sourcing procedures, getting trusted
5:44
vendors, all of that. It's it's a a lot of pain for
5:48
everyone included in the protest.
5:51
So you want to go through this protest on you if you really,
5:55
really want to and have to, the platform's really good, really
6:00
good, good enough. I feel it hasn't been the case
6:03
like 3 years ago, but now platforms are good enough for
6:07
pretty much everything you need to do for ML OPS.
6:10
However, if you look on Internet what people talk, you know how
6:13
people tell you should do things, in my opinion, they're
6:17
just not correct. They're not promoting best
6:20
practices. Like if you look at data rig
6:22
specific training, it's all around notebooks.
6:25
If someone follows me on LinkedIn, you know, I don't
6:28
really like notebooks and for a reason.
6:32
You know, I started as a data scientist myself.
6:34
I used notebooks a lot myself. That was the only way how Python
6:39
to data scientist was taught. You know, in 2016, any course
6:44
was just plain notebooks. And when I started learning
6:50
Python, that was the way. And then I started trying to put
6:54
my first models to production, like build an endpoint.
6:57
That was the use case we had. And I saw how much struggle it
7:01
was to get from a notebook to something in a production ready.
7:06
It was a lot of pain and that was the first revelation for me,
7:10
like why notebooks is not really a great place to start.
7:15
And I'm learning using notebooks is really, really hard, you
7:19
know, because people were using it for years.
7:22
They know how to do it easily, you know, you know how it is.
7:25
If you used to certain tool, getting to another tool, another
7:29
way of doing things is extremely hard.
7:31
And the longer you are doing it, the harder it is.
7:34
It's just like with anything in life.
7:38
And in my opinion, notebooks harm the mellow of life cycle
7:43
much more than anything else out there.
7:45
So that's, that's just how I feel about it.
7:48
So we need to teach data scientists to do, you know, data
7:51
science properly and not using notebooks or, you know, at least
7:55
teach them how to translate a notebook into something that is
7:59
production ready, more or less. And that's, that's what I tried
8:04
to talk about a lot. And also that's what we teach in
8:07
the course. So why data breaks specifically?
8:10
Maybe another question. Well, if you look at all the
8:13
platforms that are out there and there was I think research
8:19
ethical Institute of AI dates and they, they just, it was a
8:24
questionnaire and they asked professionals in the field, what
8:27
tools are you using from what the registry for and now for
8:30
data lake for serving, for training and data rigs came out
8:35
of #1 tool. And it's not a surprise for me,
8:39
to be honest. Data rigs is growing like crazy.
8:43
So data rigs pretty much becomes the tools of tool of choice for
8:47
ML these days in many companies just because it's easy.
8:51
Data engineering is down the data Briggs, all the data is in
8:54
Unity catalog and data rigs. It's just a logical, you know,
8:58
step forward for ML as well to train your models and maybe even
9:03
serve the monitoring on that as well.
9:06
So there are many options out there.
9:09
So, and also on another hand, not it's not just the leading
9:13
platform, I feel it's very different than other vendors
9:16
that I've seen. So they're very open to, you
9:20
know, people criticizing them. And when you tell them you don't
9:24
like something, it's not like they're surprised.
9:27
They actually know that that's the case.
9:29
But it's somewhere in not in their first priority because
9:32
they have other things that, you know, are more important for
9:36
them to fix. And yeah, if I have thought
9:40
about a cool feature on database that might be useful, they have
9:44
thought about it too, and they're probably working it
9:47
already. So that that's something that I
9:50
haven't seen in any other vendor out there and it's pretty
9:54
impressive. So there's so much to unpack
9:58
here. Let me try to go about it
10:01
because the first thing that jumped to my mind was you were
10:04
mentioning how platforms these days feel good enough.
10:07
They weren't three years ago, but now whether you're using, I
10:11
imagine when you say platforms, you're talking about the Vertex
10:14
AIS and the Sage makers out there and the Databricks
10:17
platforms. And so you can get enough from
10:21
that platform to where it is not necessary to go out and cobble
10:28
together many different tools. Yes, indeed.
10:31
The other piece that I think is crucial that you mentioned is
10:37
we've all been through trying to onboard a new vendor and we've
10:41
probably thrown our fists up in the air and said I'm never doing
10:47
this again. I do not want to answer any more
10:50
questions from the dev OPS folks or the dev SEC OPS team about
10:54
this vendor or the compliance and the ISO or the SoC 2,
10:58
whatever it may be. It takes so long.
11:01
I literally just went through this where I chose one tool only
11:08
because they were already accepted into the ecosystem and
11:13
we didn't need to then do the onboarding process.
11:17
Even though I knew it was a worst tool, I wanted to get a
11:20
different tool that I enjoy much more.
11:24
But it's like, oh, if we want to do that, then we're looking at
11:27
like a 2 month lead time to onboard that new vendor versus
11:33
let's just use what we have and amplify the capabilities of this
11:39
tool that's already been accepted.
11:41
And so it feels like that is where a lot of us stand.
11:46
And that also is one of the reasons that you chose Database
11:49
because you saw that survey and you said looks like a lot of
11:53
people are on it. I can only see this growing
11:56
because their capabilities are growing and the stuff that folks
12:00
are doing on data bricks is it's inevitable that it's going to
12:04
grow. If you're doing your data
12:05
engineering there, the data has gravity and you're going to
12:09
probably start branching out if you haven't already on different
12:14
use cases. Yeah.
12:15
So, well, the reason why we did everything in database because
12:18
we already had data bricks and data bricks was not per SE
12:22
perfect three years ago a bit more, but you could do already
12:28
back then even enough things and it you know for it to be pretty
12:32
OK. And it was still a better choice
12:35
in Azure components that we also had access to.
12:39
So we basically the company I work for is on Azure and we also
12:44
use data bricks and if I have to choose between Azure and data
12:48
bricks I would always choose for data bricks just because it's
12:52
way easier in my opinion to to do ML OPS on it.
12:56
That's the reason why. But I also want to know the the
13:01
bad part. You mentioned that Data Bricks
13:03
is pretty open to criticism. There are things, though, that I
13:07
know when I talk to you, you said, yeah, this is how it's
13:12
done in Databricks, but it's probably not the best way to do
13:17
it. Whether that's just trying to
13:20
work in notebooks and pretending like those are ready for
13:23
production or I know we had also talked about like the Databricks
13:28
feature store and how that isn't necessarily the best way of
13:33
doing things. So maybe you have some best
13:38
practices that you found as you've been.
13:41
Yeah. Going through data bricks and
13:44
learning about it or getting deeper on it.
13:47
Yeah, definitely. So I think one of the biggest
13:50
pain points to Until the day is development process and data
13:53
bricks. So as I mentioned, everything is
13:56
around notebooks and I think ML code must be packaged.
14:00
So that's just how it is. If you want to have a
14:05
professionally written ML code, it must be packaged.
14:08
And it is possible to do on data breaks.
14:11
It's just not very straightforward.
14:12
How so? For example data race comes with
14:16
notion of run times. It's pretty much containerized
14:19
version of your environment. It has already pre installed
14:22
packages and other programs, other software and you basically
14:28
want to have reproducible environment locally to develop.
14:32
Because you don't want to develop a notebook, let's say,
14:36
then you can't really reproduce exactly the same environment
14:40
locally. It's just impossible.
14:41
You can get an approximation of that, but because it's an
14:45
approximation, you can only, you know, develop a certain way to
14:49
certain state and to test, you want to test it on data rigs
14:53
again, but you don't want to push between, you know,
14:55
notebooks and local development. So there are other ways like
15:01
using ACID bundle for example. That's something that I
15:04
absolutely love using and that is underestimated.
15:08
I think way of developing on data breaks and they actually
15:11
have a lightning session about it next week.
15:14
Probably it's not going to be come out this broadcast on time,
15:19
but but there will be a recording so we can maybe insert
15:23
the link somewhere. And Acid bundle is just a
15:25
feature of Data Briggs. Well, I said bundle indeed is a
15:30
feature that is developed by Data Briggs.
15:33
It is. Well it, it has a lot of
15:37
components I would say by itself.
15:39
But first of all, it's a way of defining the workflow.
15:43
So if you have an orchestration pipeline in Data Briggs and you
15:48
can define it in Jason file or you can define it in Data Briggs
15:51
Yammer file, which is the definition of the asset bundle.
15:56
And whenever you deploy that bundle, your workflow definition
16:00
gets deployed together with all the assets and you know packages
16:06
and other files that are required for your deployment.
16:11
It was, it used to be really, really hard to deploy things in
16:14
data Rigs because you had to take care of all of that.
16:16
Like that your packages get uploaded to data rigs that you
16:20
know, your Python files get uploaded to data rigs and all
16:23
other files that you need. And you had to build your own
16:27
logics around it to make sure it was all there.
16:30
We actually build the whole thing for it.
16:32
We basically build something very similar to Asset Bundles
16:34
internally. And now we are deprecating this
16:38
because asset bundles does it all.
16:41
But it's not just for the workflows, it's also for
16:44
development. And that's something that people
16:46
don't talk about much. I feel it's people use it for
16:49
deployment, but people don't really use it for development.
16:52
But I believe for development it's also a really, really nice
16:56
feature. So, so that's a great one on
16:59
this development process. What about what exactly was it
17:03
that we were talking about the other day with the feature store
17:05
and how that works? Well, there are many feature
17:11
stores available out there, right?
17:14
It's not just database, but you have a feast, hops works and
17:18
other tools decked on available. They so other tools focus more
17:24
on just feature stores, but database has has all kind of
17:28
components in it. So if you look at database
17:31
feature store, there are pretty much 22 functions like 2 two
17:36
things that you how can you interact with the feature store.
17:40
It's a feature function and a feature look up.
17:43
Feature look up is basically defining how you look up a key
17:47
in the feature table. So I know if you want to look up
17:51
a customer ID and return some values in the table for that
17:55
customer ID you that's the what you could use for that.
18:00
It doesn't really have a fall back.
18:02
So if it doesn't find it, it will just return None, which is
18:05
by itself not very convenient. So to replace that you could use
18:09
a feature function for example if.
18:11
If that is none then look up another thing or return that
18:14
value instead. So feature function by itself I
18:17
think is a quite an ugly construct.
18:22
It's a you have to define your Python function in SQL and
18:26
that's the only way. Like I don't know who came up
18:31
with it in the first place, but I I just not fan of it for many
18:35
reasons. First of all, there is no
18:36
versioning of that thing, right? There is of course version
18:39
control that you have, but when you create this function there
18:44
is no way to point it to a version of code or like a
18:47
version of the function that is used.
18:49
Also, that function will behave differently depending on the
18:52
runtime you use of course, because you know the the Python
18:57
versions of the Python library to use to in this input
19:00
statement in your SQL query. It will behave differently if
19:04
you are in Python 3.10 or Python 3.11 and that Python version is
19:10
defined by the runtime. And when you are running on
19:12
serverless it's even worse because you can't you don't
19:17
can't choose the runtime. But there is a concept of
19:19
environment in serverless, so it may get very confusing for
19:23
people like how the function by itself behaves.
19:28
Also that function has certain limitations when it comes to
19:31
serving. So for example when you there is
19:34
also a thing like a feature serving on data bricks and and
19:38
the idea is actually quite good, right?
19:40
You want to just serve features sometimes and not just models.
19:45
Like you want to look up certain key in the table and return
19:48
things back. It's quite convenient.
19:51
And you can also serve like a combination of this feature
19:54
function and feature look UPS like a stack.
19:57
It's a it's a list that you define and the order of the list
20:01
defines the order of execution of these elements.
20:04
There is no conditional statement.
20:05
So it always, all of these things always get executed.
20:09
And you know, all this together is defined as a features pack.
20:12
And that's that's how it's called and that's what you
20:14
serve. So the feature function there if
20:19
you have to output some complex data types like I don't
20:24
something that is not integer or string, it will fail.
20:29
I hope they will fix it soon because it seems like not really
20:33
intended behaviour. So things like that.
20:37
Also, feature engineering package itself, it only works in
20:41
a notebook or in database environment.
20:43
You you can't run that thing, that Python code on your
20:47
machine. So I can go on and on with this.
20:50
To be honest, I'm not a fan of this feature.
20:55
I'm. It sounds like a mess to be
20:57
honest. Yeah, In my opinion, it's quite
21:00
a big mess and that's too bad because I think there is a great
21:05
potential in it. And another thing that I find
21:08
also one of the most frustrating pieces to be honest.
21:13
So most of machine learning is done in Pandas still, right?
21:17
Pandas, I mean, now Polaris is becoming missing and people
21:20
actually migrate to Polaris, but still not all the libraries
21:25
supported. So Pandas is still still pretty
21:28
much in mainstream. So feature function, so feature
21:33
traceability. This lineage only works when you
21:36
use spy spark. So whenever you convert it to
21:39
pandas or something else, it's not going to work any longer.
21:44
That's kind of weird. Wait, did you tell me though
21:49
that there's a workaround for that?
21:52
Yeah, there are some ugly workarounds for that that we
21:54
actually teach, but yeah, why, why?
21:58
Why would you design it like that?
22:00
Yeah, that's feels like it is part of the downfalls that you
22:08
get with a managed system, but it's also that trade off that
22:11
you're deciding, hey, I would like a managed system, so I want
22:16
certain decisions to be made for me, right.
22:20
And. It's inherent that if you're
22:23
using a platform, they're going to have opinions about how to do
22:27
things. This is one of those times where
22:31
your opinions and the builders of the platform's opinions
22:35
diverge drastically. Yeah.
22:37
And this specific feature, I would, I would say so, but I
22:41
think, well, this is just one of the things that I actually
22:44
really don't like about it. But a lot of other things are
22:47
awesome. Like I said, I said bundles is
22:49
really great. The way how we define workflow
22:53
drastically improved compared to how it used to be.
22:56
I think those are also really good.
22:59
The way how you do training on data breaks.
23:03
But there are a lot of really cool parts.
23:06
I think most of the things are cool parts, so I can't guess.
23:09
That's one of the reasons why I also talk about data breaks.
23:12
If all of that was not great, I wouldn't be.
23:16
Now, are you diving into any of the mosaic side of things or how
23:23
they've incorporated that into data bricks?
23:28
So not in the course that we have now, but we are going to
23:32
launch LLM OPS course and also I'm touching on it in the book
23:36
that I'm writing. So I cover Mlos but also LLM OPS
23:41
use cases where also indeed mosaic part is covered.
23:47
That's what we're going to. We're going to run with that
23:50
term, huh? We're going to use LLM OPS.
23:54
I don't know how you how you would call that.
23:58
For me, everything is in my lops to be honest.
24:00
Yeah, I, I think we are very biased, Yeah, because for me
24:04
too, but the LLM OPS I never felt like was a term that had
24:12
sticking power. And yeah, it's an interesting
24:16
one that. Or AI OPS.
24:17
AI OPS doesn't sound really good.
24:20
No AI OPS. I also interpret with AI for
24:26
operations. So like using AI to get less
24:31
alerts in Datadog? Oh.
24:33
Yeah, yeah, yeah, indeed. Yeah.
24:37
But yeah, I don't, I don't know what we can call it besides vibe
24:40
OPS. That's my new term.
24:42
Of let's just call ML OPS ML OPS.
24:45
I think ML OPS popularity like a as a term is actually quite big.
24:50
So I'm I'm keep pushing for it. Yeah.
24:53
And I do think it it's gone through the hype cycle and now
24:59
it's on the uptick again. Like of course when LLM OPS came
25:04
out, it went down and it was in that through of disillusionment
25:09
and now it's coming back up because I think folks are
25:12
realizing, OK, well, I we need to figure out our production
25:16
environments and no matter what if we're using LLMS or
25:20
traditional ML, yeah, it's kind of similar.
25:23
Yeah, for sure there are way more similarities than people
25:28
would like to think. And well, I give this example
25:32
quite a lot. But if you look at, you know,
25:34
data science hype cycle, data science term studied appearing
25:38
around 2016 fifteen or something like that.
25:42
And it took another three years or so and B5 before ML OPS
25:46
became a real thing. And I guess we are going through
25:50
the same cycle these days, but the cycle will be even bigger.
25:54
I mean, AI popularity has grown way further than ever happened
26:00
with data science in the past. And because of that, I also
26:05
expect that the next ML OPS hype will happen faster, but also
26:10
going to be much bigger than we ever seen before.
26:13
And well, thanks to the Viber coding we will have a lot of
26:19
work to do. That's good job security, that
26:23
is for sure. Yeah.
26:26
Is there times that you have been working with folks and
26:30
you've recommended that they don't use data bricks?
26:35
Oh yeah, for sure, definitely. I think everyone should use
26:39
whatever makes sense for for the situation you're in.
26:44
So for example, we use data bricks a lot.
26:48
Also for model serving, we have some I'm on serving on data
26:52
bricks, but I wouldn't recommend it everywhere.
26:55
So one station I would definitely not recommend it is
26:58
when the whole website is hosted by you, it's running on your
27:01
Kubernetes. And so if you want the low
27:05
latency, you would want to host your model and serve it on
27:09
exactly the same Kubernetes server, not anywhere else.
27:13
So yeah, definitely in this situation, don't use data bricks
27:16
for that. That was.
27:19
Like the anti pattern would be oh we're going to bring in data
27:23
bricks and have it be outside of our Kubernetes cluster.
27:27
It will be anti pattern because it's going to be slower 100%.
27:32
Also data bricks is not going to work for everyone.
27:34
Like I'm not talking about the serving part, which I think,
27:41
well can be quite useful because it simplifies things a lot, but
27:44
it's not going to work for everyone.
27:46
So there is a limitation of 20,000 requests per second for
27:50
the whole workspace and that's, you know, understand my
27:54
assumptions. So realistically it's going to
27:56
be less than that. And that's for the whole
27:58
workspace. If you have multiple endpoints
28:00
on the workspace, it all is covered on this umbrella
28:03
limitation. For some companies it's enough.
28:08
For some it's never going to be enough.
28:11
So then you you just need to pick the tools that make sense
28:14
for what you're doing. That's always going to be the
28:17
case. And for most companies you're
28:20
going to be fine because they don't have any, you know, hard
28:24
requirements or anything like that.
28:27
And most of whatever is done is still badge.
28:33
Yeah. And this 20,000 sounds like
28:36
you've run up against that and we don't need to talk about why
28:40
or how or what, but is there not a way to go and negotiate with
28:45
sales to bring that up? Well, I guess that could be, but
28:50
that's already an increased capacity.
28:52
So there is a default capacity which is lower than that and you
28:56
can create a request to, you know, make it higher up to that
29:01
number. I guess if you are really a big
29:04
customer and they then it might be possible, but I I doubt that
29:10
it would be that easy for any customer to be honest.
29:16
OK. So that's the when not to use
29:20
it. Now, what are some things that
29:22
you've seen? And I think we're both going to
29:24
be at the Databrick Summit Data and AI Summit in June.
29:29
And you're giving a talk, right? I don't know yet actually, so I
29:35
may be be giving talk a devil theater, but anyways I will be
29:39
around there so. What are some exciting
29:42
developments that are on your radar?
29:47
On data brakes, so I actually like that's not per SE a new
29:54
thing, but some things I probably can't talk about so
29:57
that I need to be really cautious with that, but.
30:01
You're you're Privy to insider information?
30:03
I didn't realize you were that cool.
30:06
That's awesome. OK, I can't say so OK, I I can
30:10
tell you things I can tell. So there is AB Genie which I
30:14
think is super cool. What is that?
30:17
So basically it's this AI for BI kind of kind of tool.
30:25
And that's something that within our organization, we also are
30:29
not now going to use more extensively.
30:33
That really simplifies the way how other teams that don't per
30:38
SE have, you know, enough knowledge to code things they
30:42
can interact with the data. And that's something that, you
30:45
know, we try to incorporate in our product teams.
30:50
I think that is pretty cool development.
30:52
It's it's been there for a while, but I think now it's
30:55
getting to a state that it's actually nice to use.
31:01
So we're talking about using data bricks in this utopian
31:06
world where we learn the platform and the insurance and
31:09
outs of it and that is all we need to know.
31:14
And if we can optimize that, we optimize our whole setup and
31:17
system. But for most folks, I feel like
31:21
data rigs is just one part of the stack.
31:24
What have you seen in that regard?
31:26
Oh, yeah, I agree. So I think from what I've seen
31:30
and that's the most common pattern that everything badge is
31:34
happening on data rigs. So basically the whole model
31:37
training which you need to, you know, retrain and probably once
31:41
per week or whatever your retraining cycle is, that can
31:46
happen on data bricks. And I think it will simplify
31:48
your life significantly, especially if all your data is
31:51
in Unity catalogues. It will make it so much easier
31:55
than using anything else. And that's why I think it's very
31:58
smart from data bricks to have Unity catalog in place.
32:03
So what I've seen a lot, and that's what also we are coming
32:06
from and we still have this kind of diversified way of deploying
32:11
things. So we train our models and it
32:15
results in a model artifact that needs to be served.
32:18
So you basically have, you know, model serving in this case, or
32:23
you have batch serving, which means that you just store data
32:26
somewhere in some database and at the request you just need to
32:31
query the database. Or you have a mixed scenario
32:35
where you need to have an artifact plus you need to look
32:38
up some data somewhere. So when you just want to look up
32:42
some data somewhere and without any models, I think data rigs
32:47
wouldn't be my first choice. So there are multiple ways of
32:51
doing that. It's either, you know, model
32:54
serving with some lookup and some other database that that
32:58
would be one of the ways. Or you could use online tables
33:01
and data rigs, but then you're limited with feature serving,
33:05
model serving. There are certain data types
33:07
that are not supported. It's just too complicated in my
33:12
opinion. So I would rather go for, you
33:14
know, serving some faster Pi on Kubernetes and looking up in
33:18
some database like Dynamo DB, Cosmos DB, Mongo DB, things like
33:23
that. And that's like the most common
33:27
approach I see. Also within organisations.
33:29
It makes total sense by the way. And when you do model serving,
33:34
you may want to serve it on database, but also ML for serve
33:40
works with communities. So you could deploy it in the
33:43
same way but on communities and that would be also a nice
33:49
approach. So I think you just need to
33:51
think what doesn't make sense for you for your specific use
33:55
case and you know, go for that database makes monitoring of the
34:01
endpoints much easier. That's one of the, you know,
34:04
upsides why I would vote for model serving if it's possible
34:08
and works for you, like whether it fits the requirements,
34:11
because for all them API calls you can you can look at look up
34:18
that information. So stores an inference table
34:20
which you can enable and that really makes it so nice to
34:25
monitor it. So it it's easier than using
34:30
some other tools to do the same thing.
34:34
That's just what I I think about it.
34:36
We because we tried different approaches with that.
34:41
So yeah, it's always it depends. I'm glad you brought up ML Flow
34:48
right there, because I know that there's been a lot of work on ML
34:51
Flow and specifically around extending ML Flow to new LLM
34:56
capabilities. But when we chatted before, you
34:59
were saying how cool the new updates are with ML Flow.
35:04
I've seen in the ML OPS community that there's been some
35:07
threads going on in Slack on folks that don't like where ML
35:12
Flow has been going, but maybe you can talk to us about what
35:16
you like about it these days. OK, yeah, sounds good.
35:19
I actually became ML ambassador, so I'm probably the right person
35:24
to talk about it. There we go.
35:26
No, I'm also I did feel a bit old for a while, like nothing
35:33
was really happening measure in my opinion for a while and it
35:36
was quite, you know, not very intuitive to use for new users.
35:41
But I feel data commutation is just awesome.
35:45
You it's it's improved significantly in the last years.
35:49
And if you don't understand something, then you can probably
35:53
find that in the documentation. So that's one of the things that
35:57
definitely improved. But also LLM features, that's
36:00
something that you know, also came out recently like they have
36:04
ML Flow traces. Now they have also prompt
36:08
registry, which is super cool and it really makes a lot of
36:14
sense. There is also AI gateway in ML
36:16
Flow. Well, I don't know.
36:20
I don't think there is any other tool out there that is that
36:23
feature reach to be honest, which makes it on one hand a
36:30
really cool tool, but on the other hand, yeah, it might not
36:33
be straightforward to find out how to use it properly again.
36:36
So that's that's probably the other side of the coin.
36:44
When you talk about the ML flow serving on Kubernetes, is this
36:49
the managed version or is this the open source version that you
36:52
were talking about? So open source ML flow serve
36:57
it's functionality that comes from the open source ML flow.
37:02
But basically exactly the same thing is used on data breaks.
37:07
It's just you don't have to run the ML flow serve anywhere you
37:11
need to, you know, use commands to deploy endpoints and data
37:14
breaks instead. But exactly the same thing
37:17
happens actually behind the scenes, which makes it easy to
37:24
deploy pretty much anywhere with exactly the same format.
37:28
One of the things that I find a lesson to each with data week
37:31
serving is that whenever you deploy it, if you just deploy
37:35
it, you wait, things may fail and it's very hard to debug.
37:40
But what people don't realize that it's exactly the same thing
37:43
as a mail flow serve that you could just run locally and debug
37:47
it. So there, there are ways to test
37:50
it. We actually writing A blog about
37:53
it as well now, because it's not very clear for people how to do
37:57
that. But you know, ML flow model, you
38:01
know, format, it's very similar to what Bantam ML is doing in
38:05
certain sense, right? It's just packaging your model
38:08
in a format that can be served in a certain way.
38:11
And you can deploy it anywhere. So you can make an image out of
38:16
it, a Docker image, and then you can use it for serving.
38:20
Or you use data bricks. But data bricks does exactly the
38:23
same thing on the background. So I guess it doesn't matter
38:27
where you deploy it, it just on data bricks.
38:31
I guess it's a bit easier because all of this complex
38:35
parts are hidden from you. So if you had the ability to
38:44
create your favorite stack, we could say using data bricks and
38:51
then plugging in different pieces where needed like that
38:55
aren't native data bricks options and you can extend the
39:00
platform. Let's talk about sort of a
39:04
specific use case. So it's not like, oh, well, it
39:07
depends if you're using this use case and you would want this or
39:09
that. Let's talk about a recommender
39:11
system use case and how would that look?
39:16
What would you swap out, assuming that we don't have to
39:20
onboard any new vendors and do anything new to specific pieces?
39:26
Let's just pretend like the work of actually bringing on the tool
39:32
is non existent. As we already know that is not
39:36
true, but in this hypothetical world it is.
39:41
How would you extend it? So I recommend the systems
39:45
usually are largely pre computed, right.
39:50
It's not something that we are computing at the moment.
39:53
When there is a request coming in, maybe some parts of it, but
39:58
most of it is actually just looked up somewhere already pre
40:01
computed because it's expensive to compute like no one and we
40:05
have some latency requirements. So it means that you have to
40:08
look it up somewhere in the 1st place so.
40:12
That's an assumption that we have.
40:15
Model training can be done on data breaks.
40:17
And for recommended system, I think like at least what we
40:21
have, we use Spark a lot and it makes total sense because all of
40:26
the processes that we run can be distributed for data
40:30
preprocessing. Then also the larger logics of
40:34
what we do for recommended system is also custom made also
40:37
run in π Spark. So and it results basically in
40:42
some something that is looks like a very, very large
40:45
dictionary in the end. And well, that's something that
40:49
you could store, I know in some, in some database.
40:53
And then it makes the most sense just to have some fast API
40:57
running on Kubernetes. Or maybe, I don't know, it
41:01
depends like what are your requirements are, But like even
41:06
Azure Function can be good enough for you.
41:11
So there are multiple options, just you need to to see also
41:14
what you already have within your organization.
41:16
What patterns do you have and choose for that?
41:19
If, if Kubernetes is a big thing of what you do, I would totally
41:23
go for it and then deploy just fast API app and then look up
41:26
and some Cosmos DB, dynamic DB, whatever you have database and
41:32
return the value back. And for monitoring stack then
41:37
you can't use inference tables and networks.
41:40
Then you have to well, what do we use?
41:43
We have App Insights and we also have, you know, permissions and
41:48
refund the setup and that's what we currently use.
41:52
What about the data layer of things like the whole data
41:57
pipelines and processing, moving that around, creating features,
42:00
etcetera? Yeah.
42:02
So, well we, we are, we are on Azure.
42:06
So and that we don't handle the, you know, the raw data part.
42:11
So when the data ingestion part, but there are some limitations
42:15
and don't allow us to use data bricks for that.
42:17
So that's why it's still an Azure Data factory.
42:19
But the when the preprocessing happens, it writes all the data
42:24
and Unity catalogues. So we basically are consumers of
42:29
that data. Our team is consumers of the
42:33
data and the data is produced by another team.
42:36
So if something is wrong with the data, it's not our
42:38
responsibility to fix it. But is that seemed
42:41
responsibility to fix it, which simplifies certain things and
42:46
complicates other things obviously.
42:49
So that that's a data layer that that we we're dealing with.
42:53
Then we have our own custom data preprocessing that is required
42:57
for our models because, you know, data engineering team
43:00
doesn't care about the most specific data transformations.
43:03
So we have our own, you know, data engineering pipeline that
43:07
is used for our personalization stack.
43:09
So that is shared across multiple sub products within our
43:13
personalization domain. This pipeline is with Databricks
43:16
or Airflow. It is on Databricks.
43:18
It's using Databricks workflows and we write back to also to
43:22
Unity catalog and then sub products which is for example
43:27
recommendations in the basket on the product detail page or
43:30
personally offer recommendations as what we do as well.
43:35
These are also separate sub products that run after that
43:39
other pipeline has finished and the result, the results are
43:44
either written to some database where we can use fast API to
43:49
look it up like we actually do serving on Azure Functions these
43:55
days. And the other part we we
44:00
actually have a model as well. So it it's just a model that we
44:07
do data bricks model serving for that.
44:09
But as I said, if you have certain requirements and data
44:12
bricks model serving doesn't fit your requirements, you could use
44:16
a metal flu surf and you know, deploy it somewhere else.
44:21
So I like how I asked you what your ideal stack is and you gave
44:26
me what your. But it is very ideal.
44:28
I really like what we have now. It took long time to figure that
44:33
out and we went through a massive migration.
44:36
We almost done with it. It's a great feeling and it's
44:41
actually really the way we imagined in the beginning.
44:44
So given the situation that we are, I don't think there is a
44:47
better stack for us at the moment.
44:50
That's why, yeah, I like. I liked your answer.
44:53
It's like there's no difference between my ideal stack and my
44:56
current. Yeah, indeed.
44:58
Yeah. Now we are very proud of what we
45:00
achieved. It was a lot of efforts.
45:03
Yeah. What were some of the things
45:07
that you did when you were creating this migration that
45:09
didn't work out? Yeah, we actually wanted to use
45:13
data rigs feature serving and also on look online tables for
45:21
future look up. And because of the limitations
45:23
we faced, we never could use it. So that was one of the downsides
45:28
because it would simplify our deployment stack, right.
45:32
The whole deployment could have couldn't happen just in one big
45:35
pipeline in the workflow on data bricks.
45:38
But instead, now we have, you know, multiple pipelines, which
45:42
is still manageable, but it's less perfect, I would say.
45:47
And you kind of glossed over something that I want to go back
45:50
to, which is you're not the owners of the raw data that gets
45:55
thrown into Unity Catalog. You're just a consumer of it,
45:58
which has its pros and cons. How do you break down the pros
46:02
and cons? Yeah.
46:04
So I think the data quality part is where it is all about intent.
46:10
Of course, there are some, there is some monitoring on the data
46:13
ingestion side and the way how they purchase data and what they
46:17
put in the Unity catalog, they have some quality checks in
46:20
place. However, these quality checks
46:22
are very different than the checks that we do, right?
46:26
For them, it's just like the schema makes sense.
46:28
The values are within certain ranges that are acceptable and
46:31
the count is normal, things like that.
46:34
But what we check for are very different things.
46:37
It's more around, you know, statistical properties of the
46:40
data and the engineering team because we're not the only
46:43
consumers of the data. There are more other consumers.
46:46
They don't really care about these things.
46:49
So it can happen that we see that things are broken and they
46:53
haven't noticed it just because they don't check for things we
46:56
care about. And that's a universal problem
47:00
that everyone has. This feels a lot like where you
47:07
would want to create this data meshy concept Data Contracts and
47:16
have the responsible ones and the consumers.
47:20
So the producers and consumers shake hands and say, all right,
47:23
we agree that these are the quality checks I need or these
47:27
are the things that I'm looking for and I want it with this type
47:31
of freshness and I want it in this style or this scheme or
47:34
whatever it may be. Have you thought about doing
47:37
that? Yeah, we tried.
47:40
But I think it's always about, you know, how big teams are,
47:44
what are their priorities, who they are reporting to.
47:49
All these things matter. And if there is this kind of
47:52
movement from above, then things may change.
47:56
But, you know, if you're just one of the consumers and there
48:00
is a larger team and their priorities are way different,
48:06
it's, it's just hard, you know? Yeah, yeah, that's such a great
48:10
point. It's how do you influence the
48:15
other team who has other priorities to take into account
48:21
that this data that they're giving you, sometimes it goes
48:24
all a wire and you're not able to extract the most amount of
48:30
value from it. So it's almost like you have to
48:32
go up your food chain in order for them to go horizontal and
48:37
then down their food chain instead of you just go in and
48:40
talking to that other team and saying, hey, can we set
48:43
something up like this and that. And, and so you tried the data
48:47
contracts. The data contracts were actively
48:49
put into place or it just the the producers were like, yeah,
48:53
we'll get around to it. And they never did.
48:56
Yeah, second option. That's sounds eerily familiar to
49:02
a few ones that I've heard. So all right, So before we jump,
49:05
I do want to highlight the course that you are creating all
49:09
about data bricks. I think it is obviously clear to
49:12
me as I've been talking to you for the last hour on how
49:16
knowledgeable you are with data bricks from a practitioner's
49:19
standpoint. You've been getting your hands
49:21
dirty with everything data bricks, and you've also been
49:23
staying up to date with everything that is coming out.
49:27
What's the course about? When is it?
49:29
How can I sign up? Yeah.
49:32
So this is a cohort based course that we have on Maven and the
49:36
idea is that we really want to teach everything that we know
49:40
about with Melon Data Briggs. So it's a very highly practical
49:44
course. Every week we go through a piece
49:47
of theory and we actually show the code to people and explain
49:51
how it's done. And everyone needs to create
49:54
their own, you know, code based on their own data set.
49:59
And we review pull requests and you know, every week
50:03
iteratively, we cover another thing.
50:05
And by the end of the course, everyone has a full on project,
50:09
end to end ML project that can be reused, you know, in any
50:14
company pretty much. So.
50:16
And that was always our goal with this, with this course to
50:20
actually build something highly practical and we are super
50:24
active from this court. So people keep asking us a lot
50:28
of questions. Also after the course we keep,
50:30
you know, everyone keeps access to this course and it kind of
50:33
build a community. So that sounds.
50:37
So then the course started, the next cohort starts 5th of May
50:42
and it goes on until 23rd of June.
50:46
And we also have another cohort that starts 1st of September.
50:49
And it will be the last cohort of this specific format of the
50:52
course because as I mentioned earlier, they're going into LLM
50:57
OPS and maybe going to have like extended cohorts.
51:01
We haven't figured that out exactly how it going to be, but
51:05
you know, starting from November.
51:06
So it will be a different course.
51:08
OK, so the course is awesome. I will also mention that you've
51:12
been generous enough to give everyone in the ML OPS community
51:16
a discount code, so you can, we'll leave it a link in the
51:20
description with the discount code and everything in there.
51:24
And that's awesome, thank you. You are writing a book too.
51:27
What's that about? Valen I lost the date breaks.
51:30
What else I yeah, I should have known.
51:36
It's basically all my brain dump.
51:37
Everything I know about ML OPS and data breaks will be in that
51:41
book and that's basically a guide that I always wanted to
51:46
have myself if I were getting started with data breaks.
51:51
Also very highly practical with all kind of considerations.
51:56
So the book will be coming out beginning next year, but there
52:00
was early release of the chapters.
52:03
So the first chapters already are coming out I think next
52:06
week. But also half of the book will
52:09
be around July, I believe, and I will finish writing the book
52:15
around October. That's a goal at least.
52:19
So the chapters will be appearing in the Riley platform
52:22
so everyone can read it, whoever has access to Riley platform.
52:27
So yeah, if you want to have earlier access, that's the way.