Latest / Agentic Conversations (formally mlops.community) / Why is MLOps Hard in an Enterprise? // Maria Vechtomova & Basak Eskili // #159

Transcript

Why is MLOps Hard in an Enterprise? // Maria Vechtomova & Basak Eskili // #159 · Agentic Conversations (formally mlops.community)

0:00
So I'm Maria coming lead Machinery, need you near that
0:03
hotel hazy and very critical about my coffee, it has to be a
0:08
latte in the morning and has to be exactly asian.
0:12
Half grams grinded coffee. That should be not older than
0:16
two weeks and it has to run exactly for 25 seconds and then
0:22
it should be microphone. So it's very hard to beat it and
0:27
I almost never liked coffee. Anywhere else.
0:30
Yes. Hi, my name is Ben shock.
0:32
I work as machine learning engineer at a hotel, Heist.
0:36
I like my coffee. Turkish actually.
0:39
I like I drink. Turkish coffee.
0:40
Every morning. It is strong enough to wake me
0:43
up. Hello and welcome everyone to
0:47
the ml. Ops Community podcast, I am your
0:50
host Dmitry host and today, I'm joined by none other than Abby
0:55
what's happened in a be wonderful if we have a lot of
1:00
announcements. But basically we have an event
1:03
coming up, June 15. At is happening June 15th and
1:07
16th. We've got the large language
1:10
models in production virtual conference.
1:12
So, if anyone wants to learn a little bit more about How to use
1:16
these large language models, that is going to be two days
1:20
worth of non-stop like six hours of streaming.
1:24
I think we've got three different tracks.
1:25
We've got workshops panel discussions, Fireside Chats, and
1:29
of course, talks from some of the greatest in the field, I
1:33
just confirmed the Microsoft deep speed team is going to be
1:36
there. We've also got Matt tape from
1:39
data bricks, giving a keynote and Abby, you're going to be
1:42
talking there, you're going to be on a panel, right?
1:44
Leading the Evaluation panel. Yep.
1:47
But I am pretty excited about recording this introduction
1:51
today where we had a conversation with Maria and been
1:54
shot. Can I just say one thing before
1:56
you jump into who Maria More soccer?
1:58
They hands down. Have the best coffee intro.
2:02
I think that we have heard. They are very particular and I
2:05
like that. They know what they want when it
2:06
comes to coffee. So who are Maryam stock.
2:09
So right now both of them are working at hold which is one of
2:14
the biggest Grocery chains across Europe and Maria for one
2:20
has really interesting background.
2:22
She worked as a data analyst and then went on to bias and
2:26
currently is now manager working in ml engineering a shock.
2:31
She's currently working as a machine learning engineering.
2:33
It will her background again, is as a data scientist and she
2:38
worked at Banks and stuff so they have such Clarity when it
2:41
comes to like thinking about Emma lobs frame.
2:44
First And adoption organizations.
2:47
I was so surprised at how mature they were when it comes to the
2:52
machine learning life cycle. It is so cool to see this
2:55
because they talked about and I don't want to give away too much
2:59
but when they jumped into how they were able to standardize
3:03
and make sure that everyone at a holds was on the same page and
3:09
make sure that there were some kind of standards that people
3:13
were abiding by, and Looking at the maturity levels of each one
3:18
of these groups in their little silos and what was consistent
3:22
across all of these groups? And what were things that people
3:24
were asking for? And on top of that, what was so
3:28
cool is that they were referencing a few times in this
3:31
podcast how they had been thinking about writing a blog or
3:35
that they were, they had this maturity assessment
3:38
questionnaire that they were giving out to their co-workers
3:42
and we asked them in the podcast if they would be open to
3:44
sharing. In it with the rest of the world
3:46
and it is very exciting to say now that that is in blog form
3:52
and it is on the ml, Ops Community blog.
3:54
So we'll leave a link in the description to that.
3:57
In case anyone wants to check out the maturity assessment
4:00
levels that they put together and how you can correctly assess
4:05
where your machine learning efforts are going and especially
4:09
if you're in a big company, this is just like gold, I really
4:13
appreciate them giving this to the rest.
4:15
Us my best part about this conversation with especially
4:19
when we asked Mario on which tool comes out to be expensive
4:23
versus cheaper and she said it's not really about the tool you
4:27
use. It's about how you use it.
4:29
And I think that was one fantastic or at least like a
4:34
very different point of view and probably the right way to think
4:37
about Emma, knobs maturity and emblems Tooling in this place.
4:42
Well, I get into this conversation already, huh?
4:45
Let's go really wanted to get into some of the challenges of
4:55
doing ml in the Enterprise and particularly ml Ops and so
5:01
Maria. And by sock, I'm excited to have
5:03
you both on here to talk about your experiences, and maybe we
5:08
can start off Maria, can you kick us off with just how you
5:11
ended up having such a colorful history and know a little bit
5:15
about This topic at hand. Yeah, that's a long story.
5:18
I think I was always passionate about automating things.
5:22
I started the nine half years ago, as a data analyst data
5:26
science wasn't really a thing so you could not get hired as a
5:29
data scientist but that was building some turmoil was it to
5:32
a telecom company, Netherlands KPM and I was writing scripts in
5:38
our back then so I didn't know by them but they knew some are
5:42
and the be scheduled job. It's on the server standing in
5:47
the room and it could get locked with a key.
5:50
So it was, it was really funny. We didn't have any servers, I
5:53
think fancy but I wanted to do things in an automated fashion
5:57
already. Then I moved to more into data
5:59
science building, my first itchy eyes and there was no one who
6:03
could would things from introduction for us.
6:06
So I started doing it myself. Just learning how would you do
6:09
such a thing talking to software developers and different
6:12
department and get Sing our first API live and we also built
6:19
something called Model Factory back then.
6:21
It was pretty cool. It basically we were in the
6:24
middle of thready six years ago and I lost wasn't the thing
6:28
back. Then Mana Factory.
6:30
It sounds like in ml up tool that I have heard of but I
6:34
can't. Remember where if you Google it,
6:36
you can still find our block on model secretary back then, and I
6:41
think mobile Factory is not really about what tools you use.
6:45
Yes, but it's more how to get it working together.
6:48
So I really love my lobes and I build a version of most
6:52
effective, like, four times all over again with different tools.
6:55
And also one here that will do Heiser.
6:58
So I want to dive into why and how it evolved on these four
7:02
times like you iterated on it. But by sock I also want to hear
7:07
your story. Yeah, - slightly shorter.
7:10
I first started working as a data scientist and I realize
7:14
that Notebooks for being treated as they designs products.
7:17
And many actually projects were remaining at them PB & B P.
7:22
Stage because of operationalization problems, I
7:26
made a decision to become a male engineer because it made more
7:29
sense to fill the gap between data scientist and the Ops
7:32
engineers. And as I do, how I actually
7:35
started working as a male engineer.
7:37
So my previous company, it was a Dutch bank, there I developed
7:43
models, but also deployed models And then I realize the
7:47
standardization is actually key when it comes to Big Enterprise.
7:51
There are we also try to create common solutions for
7:56
operationalization parts. And now I met a whole that heís
8:00
here. We actually managed to
8:01
standardize many things, I think will Deep dive into that later.
8:06
In the podcast, I find it very fun to actually get things in
8:09
production, because notebooks make no value, unless you
8:13
productionize them. It's one of those podcasts that
8:16
I'm realizing I should have blocked off two or three hours
8:20
of our time because just in these opening words, I see that
8:24
there are so many things that I want to talk about.
8:26
Not only what we have that we mentioned with the themes that
8:30
we're going to talk about. But now thinking about
8:33
standardization were thinking about building a model registry
8:37
and the evolution of that and how those two things go
8:42
together. So maybe Maria like, I imagine
8:45
you also think strongly of standardization because I saw
8:48
you bobbing your head and when you were creating these tools,
8:55
these internal tools and you saw the evolution of them, how did
9:00
standardization play a role if at all?
9:03
Yeah so basically model Factory or my mail Upstream work and
9:09
it's all about creating the golden path of how to do things
9:14
because data Most of the models. They just the same so they are
9:18
doing the same thing all over again when it comes to bring
9:21
that to production so they have developed a script, they have to
9:26
run it somewhere and somewhere he's usually just one tool that
9:31
you have to pick. So for example now at that hotel
9:34
case we use data breaks a lot and we run job Sunday the brakes
9:39
before it kpn. We had a server of gold after
9:43
that was the product of teradata.
9:45
ER, but KP and bought it. And we had to use it because
9:47
well, they didn't have any other choice back.
9:49
Then we also migrated that to kubernetes.
9:52
So we were running job from communities and also we were
9:57
trying to do the same thing on AWS using Sage maker.
10:01
So it's there are many tools you could use to do the job.
10:06
It's just picking the one that you have in place already.
10:10
And in large organizations, you already have a lot.
10:14
So basically, you need to A Version Control and that there
10:17
is a ready tool of choice. So we have GitHub.
10:20
For example, that would be his and for IC IC D.
10:24
We use here at GitHub actions at Key, piano, use Jenkins back
10:28
then. So it's really, there are so
10:30
many tools and you just have to see what you have and combine
10:35
them together in a way that is easy to use for data scientists.
10:39
So they don't have to think about it.
10:41
So the way in how, I see that they can use Use the reusable
10:46
workflows that are developed by a central team.
10:49
So for example, our team and a can just have something up and
10:54
running in five minutes, they don't have to think about
10:57
service user service principle. So all of those things it should
11:02
be minimal for them to start working start bringing them
11:07
almost to production. That's how I see it.
11:10
What is also important as Maria mentioned, I think it needs to
11:14
be easy When it comes to a solution design, for AI products
11:19
data science products. It's not about combining 15 tool
11:23
together and then have a very large end-to-end solution.
11:28
It's actually more about you choosing what you need because
11:32
you already need certain functionalities.
11:35
You don't need three tools. That has the same functionality.
11:38
So we would be focused here when you when it comes to standard
11:43
solution is choosing the The key elements and combining them and
11:47
having a very simple end-to-end solution and he provided that as
11:50
a golden path soon. What does like standard tasks
11:54
for? You look like are the reason I
11:57
ask this is because again, depending on the task, you would
12:01
choose very different models. And depending on the models, the
12:04
pipeline's could solve x differ. Yeah, I think it really depends
12:09
on the industry. If you look at that holder
12:11
Heiser, we have two types of Pipelines.
12:15
It's a bed shop. So we run some kind of
12:18
pre-processing and we train a model that maybe once per week
12:23
or once per day. And we create Productions every
12:25
day, for example. So that's one type of a tasks.
12:28
And then this predictions that are created by the model needs
12:32
to be delivered. So some and system so that those
12:36
predictions can be used in sending the emails, for example,
12:39
or they picked up by some server in another country, so that it's
12:43
being used for. Or demand forecast in the
12:47
stores. And the other type of models
12:50
that we have, those are basically apis.
12:54
So when you are on the web site of one of our shops, you are the
13:01
basket and then you get to Justin's of the products that
13:04
you may want to buy based on your basket.
13:07
This is an API that we developed across our API and for that, the
13:14
pipeline will be a bit different, of course.
13:16
Right. You have to train a model, but
13:18
you also need to put That model behind an API, and the right
13:21
now, we have standardized deployment and kubernetes so
13:25
that deploying the same model to another brand.
13:29
For example, is a very easy task, the deployment process
13:32
itself. Of course, you have to still
13:34
fine, tune, the model to make it work because the data is
13:38
different, but the putting the production, it's literally
13:42
couple of minutes. And now we are simplifying.
13:45
It even further by using data brake service.
13:48
By deployment. I know you had a podcast with
13:53
bacon from data breaks recently, on this topic.
13:57
So we really like that solution because the wake of kubernetes
14:02
is used across different brands of our holder case, it's all
14:04
about right. Retail company is different,
14:07
someone some have pushed type of deployment.
14:10
So you skip sea, shell commands and thumb.
14:13
Some have pulled type of deployments with argue CD.
14:16
So you see already here, the difference in how you would
14:20
bring that the production but with data fix it just the same.
14:23
Maria mentioned about deploying to other brands, maybe it's
14:27
actually good to elaborate on that because all the highest
14:30
differs from maybe other Enterprise organizations in the
14:35
sense that it has 19 Brands and All Brands some has data science
14:40
teams. Some has no team at all, but
14:42
they all want similar data science products because they
14:45
all have e-commerce website. They all have a grocery stores.
14:50
So when you come to standardization actually for us,
14:54
it was even a bigger goal to standardize things.
14:57
So that if one brand develops a model, let's also give it to
15:01
other brand and other brand can just leverage it.
15:05
And what does your workflow look like once you start working on a
15:09
project? I know your little bit further
15:11
into the charity right now where you've already built the models
15:15
in. You've already chosen your stack
15:18
but when you sort of started out in this entire journey of
15:22
building the envelope solutions for a hold, what was that?
15:27
How did you choose? What were the key tools for you?
15:30
And what was the tech stack going to look like?
15:33
Or did somebody else previously choose that for you?
15:37
Yeah. So when we started to work at
15:39
our code, okay? So we did mL of maturity
15:41
assessment and there are multiple versions of it.
15:44
You can find on the internet. We've been developing Our own
15:48
questions and we ask this questions.
15:51
Different people within their organization per project.
15:54
So we already based on that assessment and talking to
15:58
people, we knew what kind of tools were used and it was very
16:03
clear that everyone was using data breaks for training models.
16:06
So in data bricks was already within the company for a number
16:10
of years. So it did make sense to stay
16:13
with it. And well, as I need solution, it
16:16
has pros and Cons. But we found it works quite well
16:20
for what we need to do because we use parking lot for
16:23
pre-processing. So that solution made sense.
16:28
Also multiple Brands were using already and GitHub actions for
16:33
ciosed so and also it's a policy of our hello hazel to use that
16:37
for software development. And ising model is kind of
16:41
software in some sense. So we also use keep using that.
16:48
Get Hype was for Version Control and communities, was already
16:53
used for API deployment. So we kind of looked at what is
16:57
being utilized the moment and just to simplify the adoption,
17:02
but we built it makes sense to stay with tool that already
17:05
used. We are also on the asier cloud.
17:08
So we also have some major native solutions that we use.
17:13
And so, I want to dig into this like ml test score that you did,
17:17
because I know, Oh there was that famous Google paper that
17:19
came out back in the day and Microsoft also has their own
17:23
version of it. And so a what did you do?
17:26
You just sent links to people with a survey and you had them
17:31
fill that out or did you go and Shadow People for a day and
17:35
watch how they were doing things?
17:37
And what their workflow was? What did that look like in also
17:40
how many teams did you actually have to interrogate?
17:43
So indeed, we work through the papers available out there and a
17:47
bit. But this paper, send more on the
17:49
organization level, not really on product level.
17:52
So we made it really product level.
17:54
Like do use Version Control for your model, like a simple as
17:58
that move. When you deploy a model, can you
18:00
look back what code was used for?
18:02
What model artifact was used? But data was used for this
18:06
particular deployment. So this kind of question there
18:09
are 70 of them I think right now we have like also standards.
18:15
There is a standard document that we use.
18:18
So we had 70 questions and we just had interviews with people
18:23
with scheduled cold calls with them.
18:26
And we asked those questions. Some also, we sent this Excel
18:33
sheet and we shared Excel sheet in SharePoint and they were
18:36
feeling it themselves. This was, I think one and half
18:38
years ago, we started interrogating people and we're
18:42
going to measure your envelopes maturity.
18:44
Initially they were scared and later, they were also not.
18:47
Not happy to see the results because obviously, among those
18:51
70 questions, the max, I think they achieved was 60% and they
18:55
were not very happy to see that. But I think it was also good
18:59
reflection of their work. So eventually they liked it as
19:02
well. So, purely heathenistic.
19:06
Any chance you would open source those questions to the
19:09
community. I think we can do that.
19:12
Oh my God that would be so amazing because I know how many
19:15
people have come through and asked for something like that
19:19
like how do I measure what a best practices or how do I look
19:22
at what I should be doing? Or I should even be focusing on?
19:26
Yeah, we can definitely do that. We also have standards document
19:30
and we I wrote the best practices of data science.
19:35
She is like 20 Pages document I guess we could open open source
19:40
that because there is nothing regarding our hotel hates.
19:43
Really need. So we'll talk later about that.
19:45
One thing that I I would also like to learn is when you are
19:51
sharing because you mentioned oh we have a model and maybe we
19:54
want to share this model with another team.
19:58
How do you view a data sharing concerns like data?
20:03
Access model sharing All of that headache that I'm sure there's a
20:08
lot of devops people that are listening and they're just,
20:13
they're wondering that it's dying.
20:15
They're dying to know like what is going on in this.
20:17
And in that case, we don't share that data.
20:21
We don't share models. The only thing we share is the
20:24
coat so we have a centralized repository for each type of the
20:31
project. So I'm I was mentioning
20:33
cross-sell model cross from it more.
20:35
Although is deployed for four different brands and there is
20:39
one Central repository with the Logics of the model.
20:42
So and each repository for brand, it's actually using this
20:47
package the central package. And there is a configuration
20:51
file because the data locations, of course different.
20:55
But also High parameters of the model can be different and model
20:59
is trained separately for each brand in separate repository.
21:04
But because it Leverage The central package from Central
21:08
deployment framework. It actually makes it really fast
21:13
to deploy things to add on what Maria said.
21:17
When we develop models, we also keep in mind that multiple
21:21
Brands might use it. So we developed in a way that it
21:25
can be configured in parameterised.
21:27
So that's why the central package is more like a python
21:31
package. Imported in every brand
21:33
repository and used the train and deploy models.
21:37
One of the things I'm fascinated by because you have this very
21:41
interesting structure, imprudent organization itself where you
21:44
have 17 different brands. How does the code review happen
21:49
internally inside the organization and what does the
21:52
process look like? How many people are involved?
21:55
Which is like, how many deer Engineers are there?
21:57
How many eminent years? How many ideas like this?
21:59
Are there? And how long does it take to /?
22:02
Not just technically how long does it take to The modern
22:06
reproduction but how long does the Inception through the
22:09
deployment look like? Yeah I think it really depends.
22:14
So our team is a team that helps the plans to become a
22:21
professional in the machine learning and data science field.
22:25
Some Brands don't have their own data science capability.
22:28
So we have for example, supermarkets in Greece, Serbia
22:32
and Romania. They don't have data scientist
22:34
at all. So, So, our team develops models
22:38
for those Brands and we have four data scientist and to
22:44
machine learning Engineers. So that's pretty much the whole
22:47
team and working on those Solutions.
22:50
And we also have One external machine learning engineer from
22:53
another team that also helps us and we have a platform team in
22:59
the platform team is can be also different per brand but the
23:03
Greece, Serbia, and Romania and Belgium dish Share the same
23:06
platform team. And we have an agreement with
23:10
them. How things are deployed from
23:13
infrastructure point of view because that is also important.
23:16
That's the key actually, to bring things to production fast,
23:19
and there are multiple architectural choices that can
23:22
be made, how those environments are set up.
23:26
So in our case, for historical reasons, we are an agile, cloud
23:31
and so resource groups are shared across the projects.
23:35
Per gram. So, each brand has separate
23:38
Resource Group for environments of production, pre-production
23:43
development, accept the test environment.
23:47
So those have the same service principle that I be used by
23:54
multiple projects and we keep secrets in its organization.
24:00
Sticker that I shared across repositories, but we also have
24:03
some guard rails in place to make sure that That those
24:06
tickets cannot be misused and that's something that we have in
24:11
place is a product of our team provides.
24:13
Our team is slightly centralized as well, given that we are not
24:19
attached to any Brands we are working as more Consultants to
24:25
brands from a global team. So this already gives us the
24:29
chance to actually develop models for multiple Brands
24:33
already since the And you ask for a called reweaving.
24:38
Well our team develops, the models are data scientist,
24:42
develop the models and there are also some data scientists from
24:45
different brands. They can also contribute so it's
24:48
more like a open project within our organization that people can
24:53
contribute and the people contributing to that project
24:57
already full of certain software, engineering best
25:00
practices. And I think it's also important
25:02
to mention that the data is of course the Morton part of the
25:06
whole thing. And there are data Engineers on
25:09
the platform site, that make sure that the data arrives to
25:15
the prepared layer that is be used for the modeling part later
25:18
on the unsung heroes, those data Engineers, making sure it
25:22
arrives in the way that it needs to arrive, and when it needs to
25:26
arrive, all of that, good stuff. So, there is something that you
25:31
all mentioned a few minutes ago that I wanted to do.
25:35
Into also which is not having this tooling sprawl and really
25:40
looking at the user experience and recognizing that the user
25:43
doesn't want to have to choose or go between 15 tools to get
25:49
that whole end-to-end process done.
25:51
And so it seems like you decided on a select amount of tools but
25:58
one thing that I always think about, when I look at the mo of
26:02
stooling landscape, is how vague They get is there's like one
26:07
tool that will have one value prop, but it also does these
26:11
five other things and then you have this other tool that has
26:14
this other value prop. And it does these five over
26:16
other things and there's a lot of overlap.
26:19
And so when you are choosing which tool you want to use and
26:23
how you want to interact, how do you a just end up on?
26:28
Okay, this is the absolute minimum that we should have and
26:34
these are the Geordie or these are the most important pieces
26:39
that we should have. Like these tools are going to
26:41
get us 80% of the way there or 90% of the way there.
26:44
Yeah. So I think we are coming back to
26:47
this ml Lopes maturity assessment questions, right?
26:51
So we have this questions and we want to go as high as possible
26:55
in the percentage of the questions answered as yes.
26:59
So we want to be as mature as possible and then we need to
27:02
Define. How do we do that with the tools
27:04
that we have? Eddie have need to understand in
27:07
the corporate environment is not easy to get yet another tool,
27:11
even though that would cover 100% for you, if that will exist
27:16
it. I don't believe that zero tools
27:20
to all the magic for you that you need to need to do.
27:24
So getting any tool, is a big struggle so you have to do with
27:29
whatever you have at the moment in the corporate environment.
27:32
And so we happen to have data bricks, and actually data breaks
27:37
helped us to cover quite a lot of the points.
27:40
Of course, you need to use any tools, you can use in the right
27:44
way, and you can use it in the wrong way.
27:45
Data bricks, makes it easy to use it wrong, like, you could
27:49
created notebook without any Version Control and you could
27:52
schedule it You could and you can claim well model it in
27:57
production. But it's not really following
28:01
any standards, right? So it's not about the tools but
28:06
it's also how you use the tools. I believe one of the questions I
28:11
had for this out there, are you mentioned briefly about
28:16
guardrails that you have in place?
28:18
Can you go a little bit more and expand on that?
28:21
What are the kind of car dreams that you have?
28:23
While production eyes and your models?
28:26
Yeah. So I think I was mentioning the
28:28
guard rails there. Regarding their organizational
28:30
secrets that we have in GitHub repositories.
28:34
So this Secrets have been shared across multiple projects, but we
28:40
need to make sure that those Secrets cannot be just used by
28:44
any repositories. So we can only add a secret
28:49
repository in the scope of organizational secret if it
28:53
follows certain convention. So, if it has a certain naming
28:57
convention and not Can just add it there it can be only added by
29:02
specific workflow and only certain people can trigger that
29:05
workflow. So you are being checked,
29:08
whether you're part of certain team in GitHub, and if you are
29:12
done only, you can trigger that workflow.
29:15
So, in that sense, it's based on whether you are part of the
29:19
right team to be able to add your repository, to the scope of
29:23
the secrets and also its secrets are getting expired.
29:28
Like we'll see. Secrets.
29:29
Do we have a certain process in place that is run by our
29:33
platform, team to renew the secrets, it's really quite
29:38
technical, I guess and depending on different tools that you use
29:42
your guardrails, may be different, I believe, but that's
29:46
using the tools that we have this, how it would look like.
29:50
For example, that makes a lot of sense because again, this is one
29:53
of the problems that I've seen which is basically secret Sherry
29:57
and the second is credentials, Which has some sloppy processes
30:02
that have seen across organization.
30:04
Very you have to have like defined rules on who can access
30:09
what and why it sounds prohibitive early on.
30:14
When you start implementing things, eventually the value
30:17
delivery is really high. One of the things I wanted to
30:20
know from you guys with were crystal clear from working
30:23
before these envelopes processes were implemented.
30:28
Yeah. Really bad.
30:34
I think I'm coming from a company where I wasn't quite
30:39
luxury position in the sense that well Telecom company.
30:42
I worked for, they were already using all the best practices.
30:46
And what I have seen when I started working here that there
30:49
were some external companies building data science Solutions
30:55
and then they would want to put that in.
30:59
And they would literally zip the code and create the wheel files
31:03
and salad or by email to a devops team the deployed.
31:11
Yeah, I wasn't shocked. Yeah, well you can imagine right
31:14
environment where data scientist, well, create this
31:18
views and run the code, they also be different than the
31:21
devops engineers environment. So the errors would occur and
31:25
then they would send over those errors Methodist.
31:29
In the email back and then you will try and try to reproduce
31:32
them, so it can take weeks right before the issue is resolved.
31:39
It's a nightmare and it took us a year to get all permissions.
31:45
We need to deploy ourselves, like getting this organizational
31:48
secrets for service principles, not defined in GitHub.
31:53
So to agree on that. It took us a year.
31:55
Yeah, in a nutshell, the biggest change.
31:57
We made with Maria. Is to actually break the wall
32:01
between a data scientist and devops engineers and get the
32:06
responsibility of deploying models ourselves because we can
32:12
create the models, we can deploy the models within the same team.
32:16
For the ownership stays in the same team and the errors won't
32:20
have to be emailed from one team to another team.
32:23
And he also had some pick to say about but distrust the distrust
32:27
between the devops and Two beers and various artists are explain
32:31
a little bit more about that as well.
32:33
Yeah. So I believe data scientist.
32:37
They typically come from econometrics background.
32:41
That is take background. You see a lot of them.
32:44
Well what kind of one of them I guess and you see devops
32:48
engineers and all the best practices how to build software
32:53
data scientists. Don't always know about this
32:56
practices so and when they built in Use the data scientists doing
33:01
something that they wouldn't necessarily do.
33:03
There is distrust that appears and you need to build that trust
33:08
between these different teams. That data, scientists actually
33:12
can put things in production themselves because no one would
33:16
give just permissions like that, right?
33:18
As I mentioned, it took us a year so you need to build that
33:23
trust that you'll also can handle that yourself in
33:27
production environment, do you? You think there's a difference
33:30
between a Lopes and develops or what are the key differences?
33:35
According to you guys between both the fields and how does
33:39
that change between the two different kinds of models that
33:43
you're putting honor between the two different systems make you
33:46
have? Where one is the forecasting
33:48
models? In the second is the EPA models,
33:51
were the key differences. When you're thinking about ml
33:54
Ops and put the things yeah. So I believe there is quite a
34:00
big difference between them my lobes and devops and one comes
34:05
with. It's not just the code that you
34:09
need to to run to be able to put things in production.
34:12
So the data part, right? And the data part makes all the
34:15
difference here because even if you cope with the didn't change
34:20
the date, they did change and errors may appear and therefore
34:25
getting access to production. That in development environment,
34:30
might be very important for data science product and that's also
34:34
another struggle, that I believe many people have because of all
34:38
the rules to get the read, access to production data in an
34:43
environment different than production.
34:46
So that's also the struggle that we had, for example.
34:50
And it took some convincing to actually get permissions and
34:54
everything. Another one is checking for the
34:59
quality monitoring check it for the quote of your model when is
35:02
deployed because when whatever software is deployed, there are
35:08
also, of course, some type of health checks.
35:10
If you need to perform here, the quality of your model can
35:13
deteriorate it, the model can get worse.
35:17
And if you don't have those checks in place, you just
35:20
wouldn't No, because well from the software point of view,
35:23
everything is just fine. It wasn't.
35:26
So this two different testing the biggest month.
35:31
The classic question that we tend to ask on the monitoring
35:35
pieces, who gets the call when the model's deteriorate at 3
35:40
a.m. Well I believe that depends on
35:43
the model. So if you have a model that is
35:47
critical for the business in the That if it's not working for
35:52
some reason, you lose a lot of money.
35:54
So we have to use case at the previous company worked at that.
36:00
It was bad, that model people would go and get the description
36:05
for the mobile phones on the website.
36:09
And the model will either accept or reject.
36:12
The person from getting a mobile phone with the subscription.
36:17
And if the model was not answering it, It would cost
36:21
10,000 euro per hour so you can imagine this kind of models are
36:26
critical right? You need to have some checks in
36:29
place and there should be a team that fixes it as tan as
36:34
possible, and I was also called in the weekends to fix it model,
36:38
for example. but we personally right now don't have to choose
36:43
cases where pigs would be needed at three o'clock in the night
36:50
because you've crossed our model is not working.
36:53
Well, there are some costs may be involved but in the night who
36:56
is ordering online groceries anyways.
37:00
Yes probably not for many people drunk that I can imagine and
37:05
drunk people. Well, unfortunately they don't
37:08
did get delivered instantly so it doesn't make sense for drunk
37:11
person to Mordor. Anyways, it would be Leave it
37:16
like in today's neighbor and the next day.
37:18
Oh no, that's okay. So, yeah, the cost of not of
37:23
things not working and not that high.
37:26
The, so in that sense, no one gets cold in the night but
37:33
monitoring is still a key. Yeah, I also working on our
37:36
monitoring component because it's definitely important to
37:40
keep track on how your model is performing in our previous way
37:44
of working. Before we arrived, we I've seen
37:46
models predicting ridiculous, suggestions and not being even
37:52
recognized this cross-sell model for example, imagine an
37:57
e-commerce website and you just keep seeing trash bags suggested
38:01
all the time and why and this is not been even realize for a long
38:05
time so it is not a very expensive cost when it comes to
38:11
not functioning data science product but it is still
38:15
important. To keep monitoring.
38:17
Yeah, I think it's also the trust of the customer, right.
38:20
If the customer sees stupid suggestion, they wouldn't pay
38:22
attention the next time. Yeah, so that's like a future
38:28
cost accumulated water. That comes with it.
38:32
I remember one time we had a one of our first meetups ever in
38:38
them. A lot of community back in
38:39
March, 20 20 Flavio came on and he was saying that at one of the
38:44
companies he was at, they recommended to every single
38:49
user, no matter what, the user was the same item that I think.
38:54
It was like women's high heels and so it ended up costing the
39:00
company a ton of money because the money that they usually get
39:04
from these extra buys. When you have recommended item
39:09
that someone clicks on they lost all of that because they were
39:13
recommending, it not just for like a few hours.
39:15
It was 18 days straight before they realized that anything was
39:18
wrong and that ended up being something that they realized,
39:22
wow, this is actually more important.
39:24
Than we had originally thought. We also had this example where
39:28
we the personalized model gave a very expensive Jean as a
39:36
suggestion to all customers and we had no clue why it happened.
39:41
And since it was also before mL of Standards, we were not able
39:45
to track why? But we don't have a purpose of
39:51
getting everybody drunk, that's for sure.
39:53
It's classic. That's why people are ordering
40:00
things in the middle of the night.
40:01
Now, we know this is because you suggested and and it looks tough
40:06
but there is one thing I wanted to follow up on and it's you
40:12
mentioned before about notebooks and how there is now a process
40:18
around it. And I think that is one of the
40:21
most contested and hot Peixe of the ml Ops community and might
40:26
just be because I bring it up so much.
40:28
It's a little too cliche now. But what does the process look
40:32
like from that exploratory phase in notebooks to actually then
40:36
production izing it so it really depends what kind of notebooks
40:41
use the chance we use. Data breaks notebooks and data
40:44
breaks not exactly the same as IPython notebooks.
40:47
Luckily they are not so they basically just look like python
40:53
files And there is markdown command database.
40:56
Understands it's an old book so it makes your life a lot easier.
41:02
So our data scientists love notebooks is all data scientists
41:05
do and they do start developing The Notebook, most of the times,
41:10
but we encourage them to move all functions and all classes
41:15
that they right outside of the notebook. and database has a
41:19
nice feature called a, The Big Three post that allows To use
41:26
this class soon in such a way and you can also package it.
41:31
So we also encouraged to create python packages as soon as
41:35
possible because the way how we work with data breaches in
41:39
production, we build the wheel and we upload the wheel to dbfs
41:46
But to test it in a notebook, it's just very easy with this,
41:51
with this report feature to be also used, now vs code extension
41:57
for the brakes, a lot. I believe I personally have that
42:00
used that much, but I guess for shark has more experience with
42:03
that. Maybe you can tell a bit more
42:05
about it. Yeah, we allow our data
42:07
scientists start from notebook. We don't have any rights or
42:12
notebook at. They can start exploring.
42:15
Especially in later breaks where they have direct access to data,
42:19
but we also would what we also tell them is to start writing
42:24
production-ready code. I think this is also, one of the
42:26
important takeaways, from what we learned is that train, your
42:32
data scientist, in a way that they write modular functions,
42:36
they write production-ready codes, they start learning some
42:41
software best practices and with That's when it comes to actually
42:48
going to production. We can easily migrate our code
42:53
from the notebooks to a GitHub repository here.
42:57
Also get proposed plays an important role because it allows
43:00
us to synchronize it up repositories to database
43:04
environment. Now, also with another feature
43:08
recently, released from data bricks called vs code extension,
43:13
they can also develop things locally, but the running
43:17
database clusters, we obvious code.
43:20
I think this also allows them to already start writing in an IDE
43:27
while running on database clusters.
43:29
So this already, I encourage them to write more mature codes.
43:35
Yeah. And the way also, of course,
43:37
emphasizing the importance of unit testing, I think that icky
43:41
and also documenting things properly one the power plants is
43:46
also doing checks on code quality so you basically cannot
43:53
deploy if you code does not look well and but if you use our
44:00
workflows that we provide so that's something that we also
44:04
kind of trying to enforce I guess and and what does the
44:09
process internally look like when you tell somebody, you
44:12
know, you want to write production number code, do We
44:15
provide people a template internally oven for a good code
44:19
for a modern look like, or is it too like constant feedback on
44:24
their existing code? Yeah.
44:27
So when a data scientist start with the project we recommend to
44:31
use our cookie cutter template and it's not like using a
44:35
cookiecutter template. It's a workflow that they can
44:38
run from the repository that we have of quick data repository
44:42
and it would create repository for you without permission.
44:45
Ready. So basically your repository
44:48
will be added to the scope of organizational, secret, and it
44:51
will have code that can ban. It will have mended by with
44:55
Hello World in it, and then, they can deploy it already.
45:01
This fellow World code, but they can also adjust this code.
45:05
And then there is the I pipeline to check the quality of the code
45:10
that that will run. So those things we already
45:14
provide As part of this template, we also have of
45:18
course, examples of the existing projects where they can see.
45:21
And we also have some conference documentation from how to do
45:25
that and let me try to talk to them a lot and I think that's
45:29
also important piece and having examples is already good
45:34
guideline for them but I think there's also a bit of trust
45:37
between us and them. We rely on them to write need
45:43
codes and follow the guidelines. We show them.
45:46
So When you look at the whole process that you have now and
45:51
what you've been able to accomplish what are the areas
45:56
that you feel like still need work, or you have blind spots
46:00
in. Yeah.
46:03
So one of the things that we already started working on is
46:07
the cost management. I think that's a very important
46:11
thing when you are on a cloud because cost can go high without
46:16
you even knowing about it so special.
46:22
Yeah, well it can also get expensive if you choose the
46:27
wrong type of clusters anything can actually So we tagged things
46:33
properly, because the resource groups are shared across
46:35
multiple projects, this workflows that we have the tag
46:39
tag things. So we tagged at the big jobs and
46:43
when data Rick's job jobs attack, they will promote the
46:46
tag, do whatever resources data, breaks job creates to be able to
46:50
run the job. And then you can see in Azure
46:54
cost management how the costs were associated with different
46:59
rounds. That's something that is now
47:02
being handled by another team but we would ideally want to
47:05
have our own dashboard with all the information about all the
47:09
Rams with some metadata about the rounds but also with the
47:12
associated costs. So that's something that we want
47:16
to still improve on. And I think overall monitoring
47:21
there is Yeah, scattereth about tooling for monitoring the being
47:27
used across the organization. So some use a lot graph Anna.
47:33
Prometheus as your data Explorer, elastic storage is
47:37
also part of the stack, some team.
47:39
So it's really everything you can find.
47:42
We have and I think it is important first for the path
47:48
from Team to make the decisions on what the tooling we're gonna.
47:52
Have because standardization is key.
47:56
Also for monitoring and we want to build integration layer with
48:00
this monitoring tool of choice for the platform.
48:04
But because we are waiting for this decision to be made on what
48:08
tools are gonna adopt for the whole platform.
48:12
It's a bit dirty. At the moment, I would like
48:17
monitoring to be improved but here for that you need to have a
48:21
tool of choice. First and that goes beyond the
48:25
mallove's. So this tooling just for
48:27
everything right now just for machine learning.
48:31
In addition to the course management.
48:34
Well, actually, since we started focusing on the course
48:36
management, we started seeing how much, actually a model
48:39
training cost, literally in euros.
48:42
And this also motivated us to calculate how much value we
48:46
actually bring. Because if we are spending more
48:49
money than we actually are making money, then what's the
48:51
purpose of hanger having a data science model?
48:54
And when it comes to recommendation systems, it's a
48:57
little difficult to calculate the actual You you get because
49:01
it's not a direct transaction, but we still found our ways to
49:07
calculate our kpis. The value in terms of Revenue on
49:15
our websites. And this already allows us to
49:19
have a clear image of how much model calls and how much value.
49:23
It brings not every model brings as much as well.
49:28
You as models like the bad forecast for example.
49:33
But it's still nice to see and compare and and we did already a
49:38
lot of cost saving by the way, because we mentioned Were Model
49:42
developed by the third parties and the wake of they were
49:45
deployed. It just crazy there was a
49:48
process in place and as your function that would create and
49:52
kubernetes cluster and it will then it way way too big bigger
49:56
than its needed to run a thing, then it will run a thing and
50:01
then it could be discussed will be killed.
50:04
So in every day and you could be described, the will be created,
50:07
but if a job failed then can be discussed up, wouldn't be
50:11
deleted. David.
50:12
So you end up having those giveaways got through without
50:15
any monitoring, having huge amount of costs.
50:19
So by doing the mail Lopes in the way, in this standardized
50:23
way, he's in the framework that we build.
50:25
We were able to reduce the cost by 90 percent 90.
50:31
That's huge. And also in general we have seen
50:34
a lot of processes where we could improve like we've seen
50:38
processes where interactive clusters and I being used for
50:41
The brakes. So it makes sense to move them
50:45
all to job clusters because of smaller cost on the debit side.
50:50
And also, to optimize for the VM sizes and BM Types on Azure.
50:56
So we want to move from V2 types, to V4 V5, which have
51:01
better performance price-performance ratio but also
51:07
pick just, the right size of the cluster, to run a job instead of
51:11
having Being a bigger cluster that can accommodate for running
51:14
multiple types of processes. So and by that, we also are
51:19
going to save a lot of money. I think it's a good moment to
51:23
repeat with Maria said, initially I need to can be used
51:27
bad or in any tool can be used good.
51:31
Any tool can be very expensive Oriental can be very cheap if
51:34
you use it wisely. So database can be very
51:39
expensive or you can actually find a way to make it cheaper.
51:44
Such good wisdom. That is so cool to hear about
51:48
and really, I love that you are taking this approach of.
51:54
Hey, we need to prove our worth and we need to show people that
51:58
what we're doing is actually making the company money.
52:01
It's not a cost center. It's a, it's a profit Center and
52:05
that gives you so many more resources Horses to be able to
52:08
Champion new projects and show your worth.
52:12
I want to know how many people are coming to.
52:14
You asking to use Ai and chat GPT on new projects.
52:19
Oh well, actually hero never heard anyone asking us to use
52:26
chat the GPT at. However, we have a teammate on
52:30
the principal data Sciences, she loves judge ability, and she's
52:34
using it a lot. For pretty much everything.
52:40
But I believe it has not created that hyper yet within our
52:45
organization. At least, I believe with many
52:48
other visitation. Indeed, though, we also had a my
52:51
lips conference recently. And we had one talk on how chat
52:56
GPT is being used now at Ikea. So there is some change
53:01
happening in Turkey because of chargeability, but I cannot tell
53:04
you about our Own company. I started using co-pilots on
53:11
GitHub. I think that's a really nice
53:13
tool but indeed we haven't heard many Church ability.
53:17
Rumors in our organization. At least not yet.
53:22
So you're still Untouched by the entire generative.
53:26
LM hype. Well, we are touched but not not
53:34
as organization, I believe. Not yet.
53:38
Fantastic. So I think this is a good time.
53:40
We should wrap up this call or into B virus.
53:46
Really wonderful talking to both of you.
53:49
I II think I can comfortably say and I've not said it live yet,
53:54
which is this was my favorite interview to the state and
53:57
Stone. Thank you, kiss.
54:00
You guys were so open about all the things including where your
54:04
processes look like as brothers. Is talking about more than
54:08
maturities. These are and these are the
54:10
things that I do find a lot of people talking about a candle
54:14
Revenue. Question is something everyone
54:16
talks about witches. He has a BS.
54:18
I steam, we need to make revenues but nobody really puts
54:21
down perspective, some to, but it still.
54:24
Are you find that missing? At least from a big picture,
54:27
brain doesn't make sense. So I really love this
54:31
conversation. Thank you, your.
54:35
I also really enjoyed. It was really fun.
54:38
I definitely enjoy Is Skyler. I lead machine learning at
54:47
Health rhythms, if you want to stay on top of everything
54:51
happening in ml. Ops, subscribe to this podcast
54:54
now. Now, now, now now now Now.