Latest / Agentic Conversations (formally mlops.community) / Engineering MLOps // Emmanuel Raj // MLOps Meetup #69

Transcript

Engineering MLOps // Emmanuel Raj // MLOps Meetup #69 · Agentic Conversations (formally mlops.community)

0:02
Betsy. All right, I think I'm going to
0:11
go with this. Hmm Manuel is a fin land-based
0:28
senior machine learning engineer.
0:34
He's passionate and every search software engineer, speaker and
0:44
off the other. He's also of machine learning
0:53
engineer at Ito every and a researcher at our kitty.
1:04
University. Of applied sciences in Finland,
1:14
with over six, plus years of experience, building machine
1:20
learning Solutions in the industry, he has worked on
1:29
multiple domains such as Healthcare.
1:36
And Manufacturing. I was the No Doubt looking for
1:40
retail e-commerce. An aviation.
1:49
All right, hold on. Let me just let me give that one
1:50
another go because it was so close.
1:52
I got a little distracted with over 60 years of experience,
2:00
building machine learning solution.
2:10
He has worked on multiple domains such as Healthcare.
2:21
Manufacturing Finance, retail, e-commerce deviation, etc, etc,
2:28
etc. That is Emanuel.
2:31
Welcome to the show. Hey, thank you to me, that was
2:34
splendid. Ironhide your channel.
2:38
Hopefully you heard that. Let's uh let's get into it, man.
2:42
Hopefully, you enjoyed that all. I couldn't hear all.
2:45
You were saying is Praises for the intro that you just had, I'm
2:51
just gonna pretend that's what you were saying.
2:53
Well, well, I wasn't able to hear and now we are going to get
2:59
going. I think we've got enough people
3:01
here and I know they're all very interested to listen.
3:06
Into your wise words and what you have to say and also see a
3:09
bit of a coating demo. I know you've got something
3:12
planned for us that we kind of organized last minute.
3:16
So I'm excited for that. Have high hopes and I think we
3:23
should start though, just by hearing about how you got into
3:27
machine learning. And you have all of this, all of
3:30
this experience which we just heard about in the song, if it
3:34
was paying attention to the lyrics.
3:37
And so give us a bit of background on yourself.
3:42
Yeah. Surely at that mean, first of
3:44
all, I would like to say and I already said that you can hear
3:47
that but no one introduced me so well, it was so Soulful.
3:52
I really enjoyed it. Thank you.
3:55
And yeah, but just a bit of intro about how I came to be in
4:00
machine. Learning it all started back
4:03
seven years ago when I was a student back in India, during my
4:08
batch Bachelor's in computer science, engineering, I must
4:11
admit that I was not the best of students until I found this
4:15
class of artificial intelligence where I discovered that software
4:19
doesn't have to be deterministic.
4:22
It can be without rules. That can be non deterministic.
4:25
So that got me excited. When I heard fuzzy logic and
4:29
neural networks and all that then since then I've come to
4:33
Finland and then it's a wonderful place.
4:36
A lot of innovation in a is happening, especially grew up
4:40
here so I get To be part of that and worked in multiple companies
4:44
over here. And something that gets me
4:47
excited is when I see machine learning work in production, so
4:53
that kept me going on this machine learning engineer role
4:56
or sort of that kind of throws in the last six plus years.
5:00
And now I work at the company called theater.
5:02
Every that I get to be part of multiple Industries and
5:06
companies help them get machine learning to production and make
5:10
it work so that they get the Best value out of it.
5:15
That's awesome. And as we mentioned before, for
5:19
those who are just joining right now, you've got a new book out
5:23
which is amazing. I know how hard it is to write a
5:26
book, so congratulations on that.
5:28
We can put a link to that in the chat.
5:31
I'll do that next time. That you answer a question and
5:35
we also should mention that in case you're not in the MLS
5:39
Community already jump into our slack Channel.
5:41
Where it's going off. And that is all that I want to
5:45
say, because I forgot to say that before.
5:47
But let's talk a little bit about what you've seen over
5:51
these different years when you've been working with
5:55
different companies and in your opinion, why do you think
6:00
machine learning projects fail. So much as they do, we've all
6:03
heard that stat where it's like X percent of data or machine
6:09
learning projects, never get put into production.
6:11
And we've heard it so much. It's become a little cliche and
6:15
I've spoken out about that before.
6:17
But what is your advice or what have you been seeing?
6:22
And what do you, what do you reckon is?
6:24
Why these machine learning projects?
6:27
Don't, don't get put out. And they fail.
6:31
Yeah, that's that's an exciting question.
6:34
And lots to learn over the years because I've seen some projects
6:38
fail in front of me and especially since so there's this
6:42
boom of infrastructure and a lot of data being created in the
6:46
volume, velocity and variety and so forth.
6:51
So we've seen that we can integrate machine learning into
6:56
our current a software project. So while ml was coming into
7:02
software, we wanted to infuse with devops and that's where the
7:06
birth of Emma loves happen and during this course of couple of
7:09
years, then we have this Standard process of ml Ops
7:14
before that, many people are data.
7:16
Scientists used to build models in the notebooks and deploy it
7:21
in a very last title on some cluster or deployment Target.
7:25
And they used to be that used to be the draw normal approach, but
7:31
there are some problems that we could clearly see.
7:34
At least, I noticed that many companies to begin with to
7:39
personalize ML on. Air Products, they lack the
7:42
systematic. Way of developing the AI
7:44
software, which is what Emma Lopes is a lot about.
7:49
And there was a lack of continual learning abilities in
7:52
production because companies start with the mindset of having
7:58
a POC to do, a quick POC and validate, its business value.
8:03
And well it when it goes to production, they don't focus on
8:08
continuous learning and retaining the models.
8:10
So that is Reason why it projects tend to fail if that's
8:15
not the mindset to keep you continue learning in the mind.
8:19
And also, when it comes to production, when they have
8:24
continued learning ability, they tend to go with wrong model race
8:29
strategy, sometimes which doesn't satisfy the business
8:32
kpis, or doesn't get the best business value for that problem
8:37
or sometimes there's a lack of Wrong governance.
8:42
In terms of data models and auditing the project and really
8:47
inspecting quarterly or monthly what is happening in terms of
8:52
operations and backtracking the lineage of their operations and
8:56
seeing if there are any areas that can be better worked on to
9:01
make the whole flow better and sometimes they also lack testing
9:07
for real-world. One thing is to test with unit
9:11
tests. Do integration tests before
9:13
going into production but also important to do testing for the
9:17
real world scenarios such as load testing or for the business
9:21
problem and so forth. So these have been some problems
9:25
and before this cool technology called open your network
9:30
exchange. The Onyx came for
9:33
interoperability there used to be this way of developing a
9:38
machine learning models using multiple Frameworks.
9:41
Um, data scientist uses tensorflow, someone despite
9:44
heart. So, at the end to deploy these
9:48
models, there wasn't this uniform way of realizing the
9:51
models and packaging them. So that used to be one of the
9:55
struggles as well. But thanks to such Frameworks,
9:58
such as open your network exchange and that are helping to
10:01
tackle these problems. So, these are some of the
10:04
problems that I noticed while my experience over these years,
10:10
Okay, I got to follow up questions on that one, first.
10:12
One is simple. Yes or no?
10:14
Do you feel like Onyx has solved the problem there and the next
10:19
one is I want to hear you weigh in on why jupyter notebooks
10:24
aren't ready for production. Right.
10:29
Good questions, Onyx. I mean it's not the full One,
10:34
Stop solution that addresses all types of Frameworks.
10:40
Well, at least it started with addressing common Frameworks
10:43
such as sklearn or tensorflow or pie thoughts.
10:46
It's not the one stop solution. So I would say not everything is
10:50
developed by using all these common framework, sometimes
10:53
their custom models that can't be serialized using onyx.
10:58
So there lies the limitation but usually roughly an estimate.
11:03
Maybe 70% of the models could be serialized using Onyx so it
11:09
helps somewhere. And over the time I see that
11:13
developing and helping solve this interoperability the model
11:16
interoperability problems. So at least we have something to
11:21
work around and coming to jupyter notebooks.
11:25
Well, jupyter notebooks. They are basically used to
11:29
provide a web interface to see your output while you're
11:33
developing and when you put you Peter notebooks in the cic, D
11:38
pipeline, that's not ideal, because, in CI CD pipelines, we
11:42
usually use scripts like Python scripts or other files that go
11:51
with the software engineering development and that can be
11:56
deployed and used by the ICD pipeline.
11:58
So, there lies the limitation. It's good for development, but
12:03
when we take it to the cic D pipelines, that's where it
12:09
loses. Its essence of providing a web
12:11
interface because we don't need them.
12:13
It's better to go as lean and simple as possible in the form
12:17
of scripts. Well, since you're talking about
12:21
cic pipelines, maybe we can go into a little bit of your book
12:27
and how you mention in that how to build robust.
12:31
Eicd pipelines. Can you go into that a little
12:34
bit more and just tell us what your best practices have, what
12:38
you found to be best practices. Yes.
12:43
Yeah, I'm building robust, eicd pipelines.
12:46
I mean, that's all that is and it's a mindset shift from
12:53
building a model focusing on building a model to building a
12:56
pipeline and not seeing model as your final product, but seeing
13:01
python as your final product. So that's really important to
13:05
have a robust Pipeline and some other components of robust
13:09
pipeline are too. Proper process of quality
13:15
assurance. For each phase traditionally, we
13:18
have Devon vitamins to a and deployment environment, but a
13:25
lot of things don't need to be completely automated because
13:29
that might end up, end up, failing in production.
13:34
And there needs to be oversight, especially in the QA
13:38
environment. And especially Q&A is important,
13:40
because There we do a lot of load testing stress testing
13:44
integration testing before we release.
13:46
So having these tests as part of the CI CD pipeline is crucial,
13:53
and making sure that you also have a proper model release
13:59
strategy. When it comes to your business
14:02
problem is also an important aspect for your CI CD Pipeline
14:07
and these models really strategies can be Like A/B
14:11
Testing Shadow testing, Cannery testing, whatever suits, your
14:15
business problem try to integrate them in a systematic
14:20
way. So that the workflow addresses
14:22
the business problem and try to automate as much as needed.
14:28
And try to involve the human in the loop where the quality
14:33
assurance comes in. So these are some components of
14:36
robust eicd pipeline. I mean, whatever technology we
14:40
use but these These principles should be applied to make sure
14:44
that the governance is done properly and quality is assured
14:48
before we take it to production. And also in monitoring we need
14:53
to ensure that there's a good monitoring strategy for the
14:56
business problem as well. Yeah.
14:58
And that instinctively makes me feel like a little bit of what
15:06
we had, who was it? Josh Tobin on here while back
15:10
and he Talking about how you need to monitor each phase of
15:15
the whole machine learning development life cycle and it
15:18
kind of sounds like, that's what you're hinting at to is making
15:21
sure that your your monitoring everything as it's going along
15:25
and your, you've got a little bit of the human in the loop.
15:28
But you're also automating what you can, but it's able to really
15:33
get that get to that point where, unless something is fit
15:37
to move on to the next step, it doesn't go there.
15:41
So you don't have tragedies where something is put out
15:45
before. It is even ready and it's fully
15:46
baked. Yeah, exactly.
15:49
And also there's something that is seen to be developing which
15:55
is a i Ops and clubbing. That together with ML Ops ml Ops
16:01
is about applying devops practices to your machine
16:05
learning deployment and monitoring.
16:08
But AI Ops is about applying a i Do your devops practices though,
16:14
when we take that angle even it makes the CI CD pipeline more
16:18
robust. And do you have an example of
16:21
how that would work with in cic pipeline?
16:24
Because I know that I have some friends working at a regular
16:29
devops monitoring company and they use AI Ops within their
16:34
company, to make sure that you're only being alerted for
16:39
things that actually matter. So, they're using machine
16:42
learning or AI to make sure that you get the alerts that are
16:48
really important and you don't get stuck in alert hell as they
16:51
call it. And so, one thing that I'm
16:55
wondering is on the CI CD pipeline, how can you bring
16:58
machine learning into that and create an AI Ops within the cic
17:04
pipeline? For your machine learning?
17:06
Yeah, that's it's critical to get the right alerts.
17:09
Otherwise, it doesn't not have a lot of impact.
17:12
So there are some use cases that we can look into some common use
17:17
cases that can be applied to traditional ski ICD pipeline, a
17:21
lobster such as making a predictive model to monitor, the
17:26
releases that you make. So for instance if you're about
17:29
to release a model you can use another model predictive model
17:34
to see how likely it is that this model would succeed in
17:39
production or whatever you're deploying.
17:41
And the train that model which is used to predict How likely it
17:46
is to succeed or not the training data that can be used
17:49
is all the historic data of your releases, for example.
17:52
So it needs such kind of data and documenting.
17:57
And storing such data is also crucial in order to get proper
18:01
alerts. So the better your predictive
18:03
model is the better. It is able to say, if x y, z
18:07
release is going to be successful or not.
18:10
That is one use case. Ace and secondly when we have a
18:14
model in production and we are monitoring its logs, maybe we
18:19
can develop an anomaly detection model to use which is train
18:24
based on all the locks previous, historical log books and detect
18:29
an anomaly before it happens. So that we know that this kind
18:35
of error can be foreseen. And also to automate error
18:39
handling, we can bring In some some scripts as part of the Ci
18:46
City Pipeline and in case error is detected using anomaly
18:50
detection model. Then that skip the script would
18:54
be triggered to address that problem, so that before the
18:57
problem arises, we already have fixed it.
19:00
It can be something like a relearning, the deployed service
19:04
or redeploying, another model, or whatever that problem might
19:09
be. So these kind of of common use
19:11
cases can be applied using a Ops 2 that develops workflow and CI
19:17
CD Pipeline. And this kind of an approach can
19:19
give a good rate of automation for the whole work.
19:24
Yeah. Yeah, it's fascinating to think
19:26
about and I had never even looked at it that way.
19:30
So thanks for opening those doors in my mind.
19:34
There's a question that just came through and I think it's
19:36
one that many people struggle with, so thank you.
19:39
Whoever it was for asking this, hi Manuel, what advice do you
19:44
have for someone who is trying to start building the ml Ops
19:48
capabilities of their company, where to start, considering they
19:53
already have some Models in production.
19:58
Right, so I'm assuming these models are built using not using
20:04
envelopes, way of breaking a zoom, that they're just put in
20:09
production by some data scientists.
20:11
So good way to start is to look at the legacy of the company.
20:16
What kind of tools are being used because ml Ops is such a
20:20
technology that can be used using, usually, any kind of
20:24
tools that are being used in your software stack, maybe you
20:27
To add one or two depending on your use case.
20:29
So I would suggest getting acquainted with envelopes
20:33
workflow process and try to build an ml platform first where
20:41
you have a way of working to provide, get for your data, get
20:47
for your source code. So you have a lineage for data
20:50
and code and then also the models.
20:53
So that kind of a platform gives you a place where you can get a
20:57
bird eye view of the operations before deployment and then that
21:01
would be the first phase where you build your ml models using
21:06
ml platform where you have a proper lineage and the second
21:09
phase would be the deploy phase where you can take those models,
21:13
which have been trained serialized and then packaged
21:16
them to a micro service, or whatever format suits your
21:21
deployment needs and package them.
21:24
That would be the second phase so I would suggest a test go.
21:27
Creatively in a modular way you build your models first, using a
21:31
new platform. Then go to the deploy phase
21:34
where you use your email platform to deployment goes and
21:37
then go to the monitor face. So there are tons of resources
21:42
coming out these days to learn about this.
21:45
And there are big companies, or startups working as well, so I'm
21:49
sure you would be able to find your way, but keep in mind, the
21:53
legacy of your company, the business is profit, business
21:55
problems, and try to make the best.
21:57
Best workflow for it. So you said something really
22:02
interesting there that I'm always grappling with and of
22:06
course in the community were exposed to all the new tools and
22:11
everything fresh that's coming out, right?
22:14
And it's easy to go down that rabbit hole and think that you
22:17
need XYZ as whatever insert new tool here.
22:24
But when you're talking about keeping in mind the legacy of
22:27
what the company has in house, I can be useful and really trying
22:31
to build off of that. So you don't need to go out and
22:34
shop around for the newest, brightest shiny tool.
22:38
Can you go into that a little bit more and and what tools you
22:41
think are are really important or most useful?
22:45
And can you get away with doing it all, on the Legacy tools?
22:51
I mean, we've had Daniel stall on here, who works at Regions
22:56
Bank and he broke down his Tech stack.
22:59
Yeah, and it was interesting to me because there was no fancy ml
23:04
Ops tool in the whole Tech stack.
23:06
And he, he boasted that he has 100% of his models, get put into
23:11
production, and so that was like a little bit of a circuit
23:15
breaker in my mind thinking. Wow, no, no special ml obstacles
23:19
yet. He is doing the what most ml Ops
23:24
teams or what most machine learning teams can only dream of
23:28
doing so. What kind of tools for you are
23:32
the most important? And how do you feel about that
23:35
the Legacy and bringing on new shiny tools?
23:39
Yeah that's a really good question because almost all
23:42
companies have a history of software development if it's not
23:46
a start-up if it's a start-up you can of course look at tools
23:49
and try to figure out the best one.
23:52
But in this case I agree with Daniel what he said that you get
23:59
a proper way. Workflow, you can use whatever
24:02
tools or multiple tools to make it work and get 100% models
24:09
working in production and get it working for your business
24:12
problem. So somewhere I would like to
24:16
point out is focus on the devops stack that your company has in
24:21
terms of legacy and try to build off of that.
24:25
If for example, you're using develop swelling from Amazon On
24:30
or GitHub, or get lab, whatever it is, or as you're trying to
24:35
build off of that, they might, there's a likelihood that they
24:38
might have services for ML Ops or tooling around it.
24:44
Try to plug in play with that. If it suits you, then go with
24:47
it. Don't fall in love with tools,
24:49
for in love, with the most optimal way to solve a problem
24:53
and that's the way we want to look at.
24:55
And also keep the cost in mind because using the Tool or using
25:00
the most, coolest tool out there is really cool but also try to
25:04
optimize costs for getting the most optimal value for your
25:09
software life cycle. So keep this in mind.
25:13
Keep this in mind. Try to iterate with with the
25:17
Legacy tools if there are there. Otherwise, there are tools.
25:22
I would advise try to look at some tools that keep you
25:27
independent or give you. Liberty in terms of not having a
25:32
vendor, for example, if you're using some Cloud, you're logged
25:37
in to continue using that. So, there are other services
25:41
which startups are working on, which are orchestration, tools,
25:45
and I can recommend something like value High, which is a
25:49
finish start up here. Also, working in America took
25:53
off, they have customers and it's an open source tool for
25:58
some of the features. And And try to build off, open
26:02
source tools as well, because there are a lot of Open Source
26:04
tools such as ml flow, which are quite hid out in the market.
26:08
And when you try to build microservices, you can look at
26:12
fast API couple with Docker to create these micro services and
26:17
using such tools. Try to go open source as much as
26:21
possible. And if you have a legacy where
26:25
your devops tools are being used from a particular, vendor try to
26:29
Plug and Play. That works well for you.
26:31
That's good but fall in love with the with the problem the
26:35
way you solve it and optimize it not tools so it will take a bit
26:39
of trial and error and exploring and taking feedback and support
26:43
to figure out the right workflow.
26:46
And Tool. So yeah, speaking of Open Source
26:49
tools and working with them with in machine learning, the one
26:53
that always comes to my mind is Prometheus and anger fauna using
26:58
those together. And for the most part they do
27:02
the trick, right? They work, I'm wondering if you
27:06
have experience with them and how you feel about them.
27:09
And what really because I think Lena from the community said
27:14
this best that, It's not important what tool you're
27:17
using. It's what your, what kind of
27:19
metrics you're monitoring and so most tools, most monitoring
27:24
tools, specific monitoring tools are able to measure all of the
27:29
metrics but it's you, the machine learning engineer or the
27:33
one who built the model for needs to know what the metrics
27:35
are that, you're really looking for and and how you figure out
27:39
how to retrain from those and, and all that.
27:42
So, love to hear your thoughts on that.
27:44
Yeah. I mean Those tools Prometheus
27:47
and Griffon are getting quite some users, especially in a
27:50
monitoring site and yeah, it depends on you what you want to
27:54
Monitor, and there are two things to monitor especially
27:58
which is the model performance and the application performance
28:02
on the high level, and everything else comes under
28:05
under under these. So Prometheus and Griff Wanna
28:08
Give You a good bird. Eye view on both and depends on
28:12
you, what you want to customize. So I think those are Tools,
28:16
especially for monitoring stack so building off of something
28:19
else or cic the pipeline and try to Club these Prometheus
28:24
circular fauna or there's also some other tools such as
28:27
evidently AI or these kind of startups that are really
28:31
building my software and services for monitoring as well.
28:35
So it depends on what you want to monitor.
28:39
The keep your business problem and business value in mind and
28:44
try to get critical. Call metrics such as response,
28:48
time, failure rate, and the application performance metrics
28:54
and so forth. And try to even look at logs and
28:57
try to also apply a. I Ops if you want to take it a
29:00
step further to predict if something's going to fail or how
29:05
you want to handle it. And I think it's also important
29:11
to ask you about security and I know there's probably Like,
29:16
going back to this question that was asked and how to bring the
29:20
MLS capabilities into a company where you have some models in
29:23
production, I know that security can be a roadblock for a lot of
29:28
teams. How have you faced that?
29:30
And how you overcome that challenge.
29:34
Yeah. When it comes to security,
29:37
especially if your machine learning engineer, they designed
29:39
this, sometimes it's hard to foresee all the possible attacks
29:43
that might be coming. We might be be very focused on a
29:46
Model part of it. Maybe we can anticipate some
29:49
adversarial attack or poisoning attack or reverse engineering or
29:53
band or back so we can work for that.
29:56
But real life security is another thing and I would advise
30:01
to take help from your Dev Ops teams and system experts to
30:07
foresee, and have have a workaround with firewalls or
30:12
making sure you accept you're in.
30:15
Rinses from particular addresses or keep track of your operations
30:20
to keep your services secure and also Ops can be sprayed help to
30:26
monitor your system logs and predict certain patterns.
30:30
If there's an adversarial attack coming in or poisoning attack
30:33
coming in it will alert you before the impact is really
30:38
fair, so making use of all this and at the end it's a team game,
30:42
one man or woman can't do everything.
30:44
So Try to build the best of all. No, it's, that's great.
30:50
So before we get into the live coding, that you were going to
30:57
do for us, I am wondering in your opinion.
31:01
What is next? What comes next for?
31:03
ML Ops? Where Do We Go From Here?
31:06
Yeah, and The Lofts has really taken off in the last couple of
31:11
years and we've got where we've got and it's a nice development
31:15
where we have this nice workflow to automate Our deployment
31:20
operations and monitoring operations, but I think there's
31:23
quite some work still needed to be done in the monitoring angle
31:27
and making our CI CD pipelines robust.
31:29
As we talked with hops before, and also, plugging in a Ops with
31:35
ML Ops in terms of predictive capabilities to predictive of
31:40
release, is going to be successful or not.
31:43
But monitoring your logs for a potential attack or this kind of
31:47
capability. So clubbing.
31:48
Nai Ops and automating error, handling security, and this kind
31:53
of operations together. With a ops+ envelopes I think
31:58
is, is the next Frontier that we have in a lapse and if both
32:02
these come together can be a very powerful technology.
32:07
Hmm. Where do you feel like right now
32:09
is the most mature section of envelopes because it is such
32:15
like you're saying it is such a wide.
32:18
And vast discipline and you're mentioning how see ICD is
32:24
probably not so mature. And we need some, we need to do
32:27
some work with monitoring. What pieces?
32:30
Do you feel like are doing much more than other pieces?
32:37
Yeah, I think if we look at em, llp's process on a very high
32:40
level modular level. We have these three blocks where
32:44
we build models deploy models monitor models.
32:48
So I feel there's quite some nice work done with the bin spot
32:53
where their MO platform tools available out there.
32:57
Big cloud providers are there. Many startups are there where we
33:01
can have this nice governance strategy till we build the
33:04
models and package them where you can trace.
33:07
That the data that was used to train the model source code,
33:11
that was used to train a model parameters that we used to train
33:14
the model. And so so I feel like build
33:17
module is really The Taken good shape.
33:20
And you can use many tools out there and they have this really
33:24
well in casein deploy. Module is the second one which
33:27
has taken nice shape. And there are a couple of
33:31
deployment targets that are becoming quite popular, thanks
33:35
to kubernetes Docker. And these kind of technologies
33:40
that are taking us leaps in that.
33:43
So there is nice development in deployment as well.
33:46
So now the Third This is the monitoring module needs.
33:51
Some work to make sure that Mo services are robust secure and
33:58
scalable whenever needed. So tools, like Prometheus
34:02
co-founders are taking us there, but there is still some work to
34:06
do when it comes to really customizing inference to the
34:11
business problem and knowing what to monitor having the right
34:15
model release strategy and Being a strategy to header handle and
34:21
so forth. So there's work to get down
34:23
monitoring module but build and deploy are in nice shape.
34:28
Excellent. So what have you got for us
34:31
today? You want to walk us through what
34:32
you were planning on doing and making in front of our eyes,
34:37
right in front of our eyes. Yeah, definitely.
34:41
Thanks for this opportunity as well and always excited to code
34:44
and and show what can be done Hands-On.
34:48
So, today we'll be looking at one of the important aspects of
34:52
the inlet flow, especially for the Q&A or module, We test our
34:58
service before we release it. And one of the important test
35:02
that we should do is stress test or load, testing before you push
35:08
out your model to production making sure that your service is
35:13
able to handle X number of requests or X number of users,
35:18
or whatever scenarios are out there.
35:21
So making sure you load test for that.
35:24
Before you take it to production, I can be Quite
35:27
handy. So you should of course, to
35:29
integration tests and unit tests, before you build, and
35:33
this kind of test but load testing is an important one.
35:36
Which will ensure robustness of your end of service options and
35:40
let's go. Let's just pretend I'm one of
35:43
those people that doesn't do that.
35:45
What are some potentially bad things that can happen?
35:51
Then we gonna end up with the lot of failures or if there's a
35:56
highwomen Volume of requests. Your model will not be able to
36:00
serve it. Hence your users are unhappy,
36:03
your business value will be reduced and they'll be unhappy
36:07
customers and yeah you simply feel in some parts of delivering
36:14
business value if that doesn't happen.
36:17
Yeah, I think testing is something that is so underrated,
36:22
especially when it comes to ml Ops and trying to get the
36:27
Testing mentality is something that Alfredo does that talked
36:31
about two or three weeks ago. I can't remember exactly when
36:35
you was on here, but he was mentioning how important it is
36:38
to test and really try and catch everything you can before it
36:43
gets to production. And so he was he also said
36:46
something that I felt was was straight wisdom and that was you
36:50
want to get you want to catch things as early as possible
36:56
because the closer that you catch something to production
37:00
the more expensive, it is That's a good thought.
37:04
I agree to that. Yeah, we want to be able to let
37:09
catch ya. So yeah, let's see what we have
37:15
today for the Hands-On stuff. So, today, I'll walk you through
37:20
a table where we have a fast API microservice.
37:25
We may deploy it on cloud or run it locally.
37:29
But then load test this service using an open-source tool that
37:33
can be easily plugged in to your envelopes workflow to to load
37:39
test in terms of what is the response time.
37:43
Or how are your request doing and how many users you want to
37:46
test against and so forth. So let's have a look.
37:50
I will share my screen. Awesome and hopefully, it should
37:55
work. Yep.
37:57
See you in it. You see the fast, Eddie ice
38:00
cream? Yep.
38:02
Yes. So here is the code.
38:05
Really try to keep it very clean and simple because it's better
38:10
to understand. That way.
38:13
So we have this git repository you can find it with my name and
38:18
my labs Community moved up and it also stood in the chat right
38:21
now. Yes please do so.
38:24
And so there's this fast API microservice which is very
38:29
simple. It it predicts the weather if
38:34
it's going to rain or not and this kind of a problem is
38:38
inspired from my book where I Walk through the a business
38:44
problem, a real life business problem where we need to predict
38:47
the weather at Port, where ships are coming with heavy logistic
38:53
cargos. So we need to plan traffic and
38:56
so forth to optimize the port. So one of the aspects is to
38:59
predict whether they're so it's the same same code from there.
39:04
Basically, I've trained support Vector Machine model to classify
39:11
whether the given input. Put, which is coming in form of
39:15
temperature, humidity wind speed, it been baiting degree
39:20
visibility in km pressure, millibars and current weather
39:25
condition. And we use these this input
39:27
value to predict whether eight hours ahead of time.
39:32
So this support Vector machine is a Time series model, which
39:36
predicts eight hours ahead in time.
39:39
And basically, here we start Fast API object, and we take the
39:48
model scalar, which scales the incoming data bit between minus
39:53
1 to 1. And then we have the model
39:56
loaded as well. And here at the end, point
40:00
predict, we take in the input data in terms of these variables
40:05
and we simply scale it using the model scalar and then we predict
40:11
it. It's a rain or no rain.
40:14
So using this information in the hypothetical business problem,
40:18
the port was able to train so that was the idea of this web
40:23
service of this fast API server. So I'm do you see my terminal
40:30
now? Yes, yes, yes.
40:35
So yeah. I'm going to run that fast API
40:41
microservice locally and we'll test it using locus of how
40:46
robust is So there's no Docker container running at this point.
40:54
I have run that image in my PC. Local in am spinning up a
41:00
container. Now for that past API service
41:04
and when we go to the service now it should work as it is with
41:10
the predict endpoint that we saw before we make a post request
41:15
and for the Post request, you can have your input data
41:20
whatever. Mission.
41:21
There is and it should give you a response.
41:23
I have a model input data that we can drive it.
41:32
Try it out. So you can see the values
41:38
temperature, decreased, immunity, and all this.
41:41
So we input this data to the service and we get the
41:44
prediction if it rains. Yeah it says it rains for this
41:48
kind of condition so this is what we're basically doing with
41:52
this past API service and now let's check it out.
41:55
Check out how to load test this if it's able to handle 50 users.
42:01
Users what is the scenario? What happens really when we give
42:06
this service in production? For example, to the users.
42:10
So imagine this service is not running locally but it's running
42:13
in product, I'm just running it locally now just to demonstrate
42:18
to you. So we'll have to just deploy
42:21
this Docker container in your deployment.
42:24
Target to get it in production. But here for load testing.
42:30
Use something called Locust IO which is an open source tool and
42:36
it comes to the form of SDK which can be integrated to your
42:39
ml Ops workflow. And you can see the script is
42:43
very simple here, where we use these three methods from Locust.
42:50
So we use Locust, and use these three methods, HTTP user tasks.
42:55
And between, and we take the dump data, which is the input.
43:00
Data. So, the HTTP user that works
43:05
using the Locust will bombard the API service that we deployed
43:10
in production, or we're running locally now with this dump data.
43:14
So the data would be constant but you'll make multiple
43:17
requests that's the idea. So here we create a class with
43:22
your name of your choice and and pass the HTTP user that will
43:28
make requests to your endpoint. And And here you point to the
43:34
service to make a post request and you sending the data and the
43:40
header, which is adjacent content type.
43:43
So here you don't mention the endpoint.
43:47
Now will spin up the Locust or IO server and it's a server,
43:51
it's a web service that runs your in your infra, but it can
43:56
be also a script that can run and give you output.
44:00
So you In that output, you can work around in your CI, CD
44:04
Pipeline and say if it passes a load test or not.
44:08
So let's spin out those local store IO.
44:23
So I'm starting the Locust server which spins up the web
44:27
interface. It has a cool web interface so
44:31
it runs on localhost 1889. So here you have a choice where
44:40
you should mention your end point that you want to test.
44:43
So I'm going to use my Local Host 8080 because I'm running
44:48
the service locally you can deploy it on some endpoint on
44:52
cloud or Deployment Target and give that endpoint address yet
44:56
tested. So I'm going to simulate a
44:58
scenario where I will try to have 50 users testing and have a
45:06
spawn rate which will be not s number of users per second.
45:10
So this will let say we spawn up 10 users for each second until
45:15
it gets 50 users and bombarding the endpoint with 50 users.
45:20
So let's start the load test. So, here you see that the test
45:27
has started now for the predict endpoint and there are some
45:31
requests going with the dummy data or the data dump, we had in
45:35
the Locust script and there are zero fails for.
45:40
Now this is the aggregated data. You can see and We can also look
45:48
at the median time for the request response time and look
45:52
at charts as well. So the charts charts, look
45:56
something like this. We can see what time requests
45:59
are coming to the end point. And at what time, how many
46:03
number of requests are being done per second.
46:05
As you can see 15.2 16 requests in a second being sent to the
46:12
service and you can see the average response time And the
46:17
number of users exponentially over time.
46:20
As we requested, 10 users to respond up here 50 user.
46:24
There's are constantly bombarding the service so it is
46:29
a HTTP users and you get critical information like this
46:33
total requests per second and how many have passed as you can
46:37
see, there's one failure here. You can, of course, go back to
46:41
your logs and see why it has failed that kind of information.
46:45
Can be useful to handle such an error if it happens in
46:51
production so you can see your failures here.
46:56
So there's also locks that come in here.
46:59
It is connection error. Something with my Local Host.
47:05
Yeah and you can also see exceptions.
47:09
Here, there are some things that happen.
47:11
You can download this data and further inspect but I wouldn't
47:14
recommend this. RC IC D pipeline.
47:17
I would recommend using using SDK to to logout the outputs of
47:25
the test and let's say, if you are, you have a, I will stop
47:29
this test now and let's say, you have this test, go on for two
47:33
minutes and you have the end results and you see that all
47:37
your requests are passing. There's no fail.
47:40
Then you can take it to production otherwise just have a
47:44
look at your Logs, why is it failing and try to address that
47:50
before taking it to production? So this kind of a tool can be
47:54
useful, especially in quality assurance also interface where
48:00
you can load, test your microservice before you deploy
48:04
it to production. So, you get a idea of what it
48:07
can be with X number of requests and X number of users trying it
48:12
out. So, Yeah, that is what we have
48:14
for today. So, is there a way to automate
48:19
this in your, in, your pipeline, or in, in your whole strategy?
48:26
Yeah, the way to automate would be to have this script log out
48:34
or send locks if for a number of requests that have been done,
48:39
how many have passed? And how many have failed this
48:41
kind of statistics. And you can have a script like
48:45
this part of your Q&A, for example, where you do test, and
48:50
this test would tell you X number of failures and if there
48:55
is no failure, then automatically deploy the service
49:00
to production. If there is failure, and of
49:02
course, you would need human intervention and inspection to
49:06
see what's the cause and so forth.
49:08
So having this script as part of Q&A or Dev can Give you an idea
49:14
of the kind of load and how the performance of the microservices
49:19
before taking it to production. Amazing.
49:27
So I think that is That's all. We've got unless there's any
49:34
more questions. This has been awesome and I had
49:37
no idea about Locust and I think you instantly see the value of a
49:43
tool like that and and why you would want to use it.
49:47
So that's really cool to see. I want to thank you for coming
49:51
on here and is there anything else that we we should know
49:55
about you? And I know you're in the
49:57
community slap but if somebody wants to reach out, what's the
49:59
best way to do it? Yeah, first of all, thank you
50:03
Danny for having me here and I was, of course, looking forward
50:07
to have discussions, especially all the awesome discussions that
50:11
happen on envelopes community. And I'd like to point out that
50:16
the book is out, engineering analogs, you can learn all these
50:20
tips and tricks of having a robust cic pipeline ml apps in
50:25
general. See two such as Locust ml flow
50:29
and all the tools that Usually handy and it comes to Emily Ops
50:33
workflow. Check out the book.
50:37
But otherwise it's been a pleasure to have the demo and
50:40
have these awesome descriptions and every time we have this
50:44
discussion, there's quite some nice Reflections happening.
50:47
So thank you for that and it's been a pleasure.
50:51
Yeah. And for those that do not know,
50:54
I'm throwing the book in the chat again, one last time.
50:58
And are you going to To create a course ever.
51:03
Right? Also that's in the pipeline as
51:05
well because some people like to read books, some people like
51:09
online content, so the video content is ongoing and of course
51:16
should be out on udemy next month and somewhere where I
51:20
cannot third week that was fast. Really cool.
51:24
Well, keep us posted when it comes out and we'll put it in
51:27
the description in YouTube for those who are watching this in
51:30
the future. As for now.
51:32
Thank you everyone for joining us.
51:35
And we will see you next week. Same place, same time.
51:39
Yes, thank you. It's been a pleasure.
51:41
All