Latest / Agentic Conversations (formally mlops.community) / Deploying Machine Learning Models at Scale in Cloud // Vishnu Prathish // MLOps Meetup #60

Transcript

Deploying Machine Learning Models at Scale in Cloud // Vishnu Prathish // MLOps Meetup #60 · Agentic Conversations (formally mlops.community)

0:16
Yeah, this new you're normally in Canada, you're back in India
0:20
right now, I think it is a good start.
0:24
If you can tell us exactly what brought you to Canada, how you
0:28
got into Tech, And what you're doing right now.
0:34
Interesting. So I I don't remember a time
0:38
when I'm not in Tech So it's 15, 20 years of encoding.
0:45
I've been programming since I know it grade but my entire
0:53
degree, my undergrad Masters, everything in software
0:58
engineering, so I'm not a dentist.
1:00
Dissed by discipline, and that's probably what I'll bring to the
1:06
ml Ops Community as well, right? So there is a wisdom instead of
1:10
wisdom that is coming from the software engineering world.
1:13
That's very relevant to them L Ops community in general and how
1:20
how we built our department pattern.
1:22
So we built our pipelines. These are largely a mess of data
1:29
scientist. Workflow combined with
1:32
traditional software, engineering, paradigms, and
1:35
wisdoms. So that's what I bring in that
1:37
has been what I've been working on for over 10 years now.
1:43
What what I do right now? I lead a team of data scientists
1:49
and engineers at, in a wise, solve a very cool problem.
1:54
We we work in the water industry obviously like like, We bet the
2:02
fresh water supply for the world is limited like extreme weather
2:08
flooding around the rise and the demand for like fresh water has
2:14
grown dramatically and our capacity to deliver them is
2:17
limited to a growing population. Safely affordably reliably,
2:23
right? So this is where the noise comes
2:25
in will build all these software.
2:27
That's that Makes water infrastructure happen.
2:32
Again, simplest words, but this is one of the Legacy Industries
2:38
and we have our own problems. Like there is aging
2:42
infrastructure. Some of these pipes and assets
2:48
that I work with are like 100 years old, right?
2:51
So, there is one instance where we actually dug up a wooden
2:55
pipe. And like if you're modeling the
2:58
vibrational characteristics of a wooden by B company, Three
3:00
different from like a regular bike.
3:03
So you can imagine the kind of data science problems will go
3:07
through just because of this aging infrastructure, right?
3:12
The workforce that is operating these distribution networks or
3:17
Wastewater management plans. They are all like largely
3:20
tiring, baby, boomer generation. They have this tribal knowledge
3:26
that has been built over decades.
3:30
That's not translated well into into the next-gen, right?
3:35
So there is a lot of scope for automation.
3:37
So this is the we are coming in at the time where the see this
3:42
industry is starting to consider automation, right?
3:46
Like many of them have, but there is still like a trailing
3:49
large list of people who have not moved move to Cloud our
3:56
bodice as product, for example, or the security model is old
4:00
things. Like that and well I try to
4:04
stall. And it is, this industry comes
4:07
with its own unique, set of data challenges, it set of quality
4:14
problems as I told. Wouldn't I problem, right?
4:19
So these are the issues that we deal with every day.
4:23
And we have built a solution to optimize and solve all of these
4:29
things at it. About what are so yeah, I need a
4:32
team. Yeah.
4:33
Yeah. What are some of these use cases
4:36
that you're working on? I mean, there's the wooden pipe,
4:38
but you said that, you make basically the water, the fresh
4:43
water happened. Is it just around the
4:46
instruction or you purifying it? Also, what's going on there?
4:50
All right, so then that is an oversimplification but we, you
4:56
know, Isis exclusively software company, we largely bill.
5:00
Build digital models like the term digital twin is, is
5:07
familiar to many who are working in the industry.
5:10
It's basically a virtual representation of this physical
5:14
actual tangible assets, right? So, if there is a distribution
5:18
Network that takes water from a lake, I'm seated to a reservoir
5:24
and maintains like a certain level of pressure and then it
5:28
supplies to people house people So we call it a water
5:31
distribution Network reality. That is way more pumps way more
5:33
Lakes, way more first wires. But ultimately there is like an
5:38
aggregate demand, which will be like a curve, right?
5:42
So there is people are waking up in the morning and they will
5:45
require X liters of water, but as the day passes it reduces.
5:50
So there is like a demand curve that we need to meet all the.
5:56
While there is a tariff, Ocean to the electricity that these
6:02
companies are using to pump the water in from the lake to those
6:05
who are, right? So the first thing we do, we
6:09
take this network, we digitize it build this digital twin and
6:14
what it means to be still doing is we can virtualize its
6:17
operation so we can input certain way the pump should run
6:23
over the next twenty-four hours. And we can see whether it will
6:26
need the dream and how much energy we consumed in doing
6:29
that. It's so in a wise, the matches
6:32
my org. Then basically, optimizes this
6:36
operations, we provide customers with a schedule of how to run
6:42
their Network or the system in a manner that will save them
6:46
energy cost optimal. Right?
6:48
So all these problems I talked about with aging like operators
6:55
and all of that goes away by adding like an AI based
6:59
automation into this, the actual details of how that works, is
7:03
fair. We involved discussion and I can
7:06
get into if anybody's interested.
7:07
Yeah. Well button it involves.
7:09
Yeah. So let's talk a little bit about
7:12
the platform that you've built for data scientists and the
7:16
whole machine learning. And so maybe we can talk about
7:21
some of the like where you started when you got into it and
7:25
then how its evolved, and what does the platform look like now?
7:29
Why is it unique? All right.
7:32
Yeah, so like just, all right, so there are certain problems
7:37
that comes as part of the territory working in this
7:40
industry. First sensor, placement is bad,
7:45
right? These are old unreliable sensors
7:49
placed across this network, but some of them are reliable, but
7:53
not many of them are exerted. A lot of sensor anomalies and
7:57
another issue. That is the data is very sparse
8:00
like some of these Chemical concentration.
8:02
Some of these measurements of quality water is taken, like
8:08
using manually collected by human being every day sent to a
8:13
lab after X number of hours, the cells come back.
8:17
And this is your one data point in certain cities, right?
8:21
So you get four of them a day and we need 20,000 of them day
8:25
to build reliable models, right? So, these are like unique
8:30
challenges that comes with Working in this industry and we
8:37
buy by large. We started out to figure out
8:42
whether there is a way to even like build a digital 12, build a
8:47
reliable representation of this network because they did I so
8:51
bad, right? So we figured it out, but that
8:56
goes into all right babe, but the ml Ops and the a pipe Line
9:01
where it really helps this, it it kind of enables this
9:07
configuration driven data, cleaning and data,
9:10
Transformations, and feature, engineering that you apply on
9:13
these data points, consistent and reproducible like across
9:19
various other points, the pipeline.
9:21
So let's start with somebody building models in like jupyter,
9:25
notebook, applying certain feature, engineering
9:28
Transformations and data cleaning operation.
9:30
They are dropping. Interpolating data are dropping
9:33
sensor anomalies and we apply a lot of them, right?
9:37
So this needs to be reliably replicated in real-time pipeline
9:41
in batch inference in retraining, right?
9:45
So Eric a conflict driven there's like a configuration
9:50
element to that that we need to build so when we started
9:52
building it out like this is handcrafted models and they were
9:56
working in theory but it's Ever materialized in production like
10:03
when we started with our first customers.
10:05
So that's the first time you thinking about Emma loves ho
10:07
okay. How can we solve this problem?
10:11
The first thing we started doing, we started building like
10:13
a model repository kind of thing where we can build and deploy
10:19
the models along with Associated metadata, the talks about how
10:25
feature engineering was performed.
10:26
How direct leaning was performed.
10:29
Hmm. And and we try to reproduce that
10:33
in real time, but it turned out a lot of like manual custom
10:37
building work. And we tried a few other
10:41
out-of-the-box Solutions, data robot data IQ.
10:45
There are a few provide us out there that claim to provide an
10:50
end-to-end solution to this problem.
10:53
But ultimately we kind of built a bespoke solution which is kind
10:58
of them. Loves that everybody has, right?
11:00
Nobody has February. There's no standard and a lobs
11:05
package out there that works for everybody.
11:08
It's always bespoke. But we were we leverage stage
11:13
maker and Amazon AWS to do a lot of these building blocks.
11:19
And we it's still a work in progress, but we right now have
11:24
decent Deployment pipeline. That involves end-to-end data
11:31
science, workflow starting from feature engineering, all the way
11:35
cleaning all the analysis you need to do, ability to do a
11:38
continuous retraining. And we are reaching a point
11:41
where we can do automated real training.
11:43
That's good, I see ICD pipeline, that promotes the artifacts from
11:49
like a development environment to QA to production and in
11:52
between all the checks and balances you need to get real.
11:55
Time monitoring in production or all the sweet stuff that comes
12:01
with the yeah. All the seats that comes with
12:06
good mL of sight line. Can you walk us through some of
12:10
those like that that pipeline architecture that you're talking
12:14
about? And obviously, yeah, verbally
12:18
here but talk us through and also I'm really interested about
12:23
why When you did POC with some of these other companies that
12:29
you mentioned before it didn't work out, what were some things
12:33
that and then you said, alright, well, let's just do and you
12:36
landed on stage maker. Why did that one work?
12:39
In the other ones, didn't? Let's stop that, right?
12:43
So there is really like two patterns out there and I want to
12:50
call it Cloud, agnostic, and Cloud native with the Eternal
12:54
debate in the industry. Which we Which way to go people.
12:57
Take both sides, but I am generally a big fan of cloud
13:03
native solution. So an Amazon at a place is Big.
13:07
Obviously, the largest provider bigger than next three providers
13:11
combined, right? So Amazon do has have a large
13:15
array of tools that facilitate some elapsed, that the the The
13:25
idea is built around like a centralized IDE integrated
13:28
development environment, that it's like your jupyter notebook.
13:35
But this Federal right? It just connected to everything
13:37
Amazon. So if you want to feature
13:42
engineering, like, if you want to do say, feature selection,
13:45
from a list of 200,000 features, your local machine that you're
13:51
running the jupyter notebook in is isn't just powerful enough to
13:54
do that. You can Outsource that to
13:56
Amazon's one click. You can do training, highly,
13:59
parallelized neural networks training in Amazon's compute
14:05
Cloud, another one, click, you get apis, to do all of that and
14:09
then from the cic, D is also, there is a change maker
14:13
pipelines to do reproducible training, right?
14:18
So once you once once there is a messy process, right?
14:22
Once you get to A good model. Sometimes you lose track of how
14:28
we arrive in there, right? So it's important.
14:33
It's important to be able to reproduce that and we always
14:38
like redo the whole thing starting from raw data or
14:43
Transformations that you apply it gets saved, the conflict
14:45
driven manner, it gets rebuilt and we trained.
14:52
In addition, you also get some free hybrid parameter during in
14:54
that and we are But ultimately, the model gets rebuilt, and the
14:59
artifact that gets rebuilt is what gets used in other in
15:04
production, from now on, right? So a lot of these, nice, cool
15:09
things. Came with sage maker, like and
15:12
Alternatives. We if you, if I were to go and
15:16
Link Cloud, agnostic approach, and which a lot of people do,
15:20
right, and there is, the definitely is value to it.
15:23
Not denying that. It's just a lot more devops.
15:27
Like their designs is a very multi disciplinary process as a
15:34
result, there is multiple several tools involved in it,
15:38
right? So, there are open source tools
15:42
out there that do one thing really well like dq'd has unit
15:45
testing for data ml flowed as experiment.
15:49
Tracking Jupiter, lab does like a shareable notebook mechanism.
15:55
See, icds thermoform and 200 other plays of doing it.
15:58
All of these are separate separate tools now.
16:01
You bring all of them together like that, same full-time job,
16:06
right? So, and what by going, like,
16:09
Cloud, native, picking up tools that are built within Amazon and
16:13
Sage maker you're basically cutting short, that all of that,
16:16
you're way more agile. You're way more fast in figuring
16:22
out the like it's business speed basically right here, your your
16:27
go. You reach your Value Point
16:29
faster. And like, if you want to fail,
16:33
like Emma loves, you said, still an evolving discipline, right?
16:37
Not many people have figured it out.
16:39
So you need to try and fail many times before you figure out
16:44
what's right for you. So this enables you to fail
16:47
fast, like, you don't want to figure out after maintaining and
16:51
buying a huge, cavernous cluster, even it's in Cloud
16:54
paying it for it after 3 months, then you realize it doesn't work
16:56
for you, right? So, serverless patterns.
17:02
Things like that, keeps your cost minimum and while get a
17:07
while enabling you to build your custom.
17:11
Bespoke analysis pipeline faster than typical.
17:15
So, can you talk about why you didn't go with open source
17:22
tools? And I see there's a question
17:23
that came through. So I'm going to get to that
17:25
next. I promise, I will get through
17:26
it, but I'm wondering about like open source and if you played
17:30
around with anything, Thing in that Arena that is that is open
17:35
source Alternatives or everything we do.
17:39
And end the, to be honest, select lot of this Cloud
17:41
providers leverage, those right? These are all safe, I think
17:48
seismic experiment, tracking is built around him and flow, but
17:53
and what happened? Wasn't sure about that.
17:54
But the South Tower is largely leverage open source tools,
17:58
right? So the question is, whether you
18:00
want to do that in the Aggression yourself or let the
18:05
cloud providers. Now, case Amazon do it then, use
18:08
it, right? So we did evaluate a few of them
18:13
and realize that there is, there is value in it.
18:17
It's just that when, when you start evaluating Amazon, we
18:21
realize that we can I can hire one last devops person to build
18:27
and maintain this entire fight crime.
18:31
Yeah, that makes sense. So question coming through is
18:34
what's the platform that you're using to virtualize?
18:37
The physical system that virtual twin which lies just gives us
18:43
some. So that's I wanted without going
18:47
into too much and details. Over IP, it's basically modeling
18:53
each asset involved in in this network.
18:57
Let's say I talked about a lake and pump and a destroyer and
19:03
people supplying getting orders from edified.
19:06
So we more than all of these components and and the model
19:10
that you pick, for each asset, depends on type class of acid.
19:14
A lot of them are random Forest models.
19:17
Some of them are neural networks.
19:20
Some of them are very autoregressive, models are
19:22
forecast demand, for example, so but they are Semi-auto Amal
19:29
style pipelines that allows you to pick the right model that
19:34
solves. That particular asset problems
19:36
of a pump is probably the simplest of of this list of
19:41
assets side. So it's if your pump is on based
19:44
on a pump curve, you need to determine what the flow is going
19:47
to be from the but that as you add chemical process into this
19:52
flocculation erase that gets complicated and there needs to
19:55
be advanced mode. But ultimately, we model all of
19:58
these individual components using an array of machine
20:02
learning processes. And build collected, diag
20:08
directed acyclic graph of this models that can that that
20:13
essentially becomes a digital tool.
20:14
So know how in terms of the platform that is used it is
20:19
entirely in Amazon stage maker. We use various libraries that is
20:25
some of them natively provided. Some of them we developed for
20:28
certain type of assets. But ultimately building models
20:33
that represent the Lisa. Ourselves gets deployed to Sage
20:36
maker inference engine and we leverage real-time and batch
20:42
Insurance capabilities provided by teaching Michael along, with
20:46
their custom-built framework. On top of it to build this bag.
20:51
I don't know that answer that question.
20:53
If that doesn't, please follow up.
20:56
Yeah, that was pretty clear to me.
20:59
I'm wondering about the data quality issues that you
21:02
mentioned before and how you People like this scenario is
21:07
wild to meet you, have people manually taking the test and
21:11
then you get four of them a day and really what you would prefer
21:16
to have our 20,000. So how are you attacking that
21:21
problem? Are you just going in and trying
21:23
to lobby for more sensors to be put in?
21:28
Or like, what can you do there? Like so, well, not all.
21:34
Data science, problems are solvable where there is a degree
21:36
of Science in it. You can crush it and solve it.
21:41
But so, well, there are instances where we were unable,
21:47
the model systems purely because of lack of data.
21:51
But there are really some really cool methods.
21:55
You have developed that helps mitigate this problem as well.
21:59
Like, there are methods of building virtual or soft sensors
22:03
that Rely on the sparse data but also leverage one of the other
22:10
sensors that has more frequent data points to build the soft
22:14
sensor. And there are some proprietary
22:16
algorithms in doing that. But this conversation not about
22:21
data science, right? So in terms of Emma lobs, what
22:25
we really need to do is enable data scientists to do whatever
22:31
they want. And make sure that it's
22:34
reproducible in a pipeline, that's the key, right?
22:38
And this this, what I call as config driven data
22:43
Transformations for it, it needs to be it.
22:50
Needs to enable the messy data science process of building
22:57
could call. The features are going to a
22:58
model from like a bad Road. Data.
23:05
And capsule that in the configuration.
23:08
Make sure it's available for retraining.
23:11
Make sure it's available for real-time inference.
23:14
When you want to do the predictions, make sure it's
23:16
available for other match predictions if you're doing any
23:20
it. So that's one like, really good
23:23
way to draw Bus Time. Elapsed pipeline helps in
23:28
solving this problem, but the actual way of solving itself is
23:33
Extremely design, C. Hmm.
23:36
So this idea of reproducibility is obviously really important to
23:41
you and the team. Why is that?
23:45
Have you had problems in the past where you couldn't
23:48
reproduce something and then it came back to bite you?
23:50
Or is it just something that you're bringing from the
23:52
software development world? But you produce ability is
23:58
obviously important for but everybody, right?
24:03
The and And most of the Frameworks out there, do that
24:06
for you. There is a but it really starts
24:10
from legacies and grabbing and onwards the cleaning and
24:15
preparation steps that involves. Calculation of calculated
24:22
virtual sensors calculated, virtual Channel streams of data
24:26
or droppings and so normally'. They re producing all of them,
24:32
reliably and correctly between environments, takes like little
24:36
bit of work. It's little bit of software
24:39
engineering, there are tools out there that does that Republic.
24:42
Amazon itself has features Stone.
24:44
It goes all the way up from a ton of Interest cleaning but it
24:49
clean Go some raw data. You are, you'll be able to build
24:53
and store and save features that you can reuse across your
24:59
multiple modeling exercises. Yeah, that's a fascinating one
25:04
for me because I know in the, in certain sectors, reproducibility
25:09
is a must right? Because of the law.
25:13
And so, it's really interesting to hear that for you
25:18
reproducibility Is there because you just need to?
25:21
It's like the core business I guess is what it sounds like.
25:26
Like, if you can't reproduce it, then your core business is not
25:30
going to be able to work that one.
25:32
Yeah, that is what it is along with the deployment pattern that
25:38
we need to follow. Like so one thing I might not
25:40
have mentioned, well, is this digital twin building exercise
25:43
is custom for every customer like they Different thought I
25:51
would advise like talk about, but they have different networks
25:57
different assets different problems.
25:59
So this it hand curated exercise of building models.
26:05
So, it's not as separate. See a CD pipeline that is built
26:10
per customer. It's 11m elapsed pipeline
26:14
through which like a models pass through, right?
26:17
It's not like a Cic, D script that gets that changes between
26:23
customers and you have the ability to customize it any way
26:26
you want. If the day start giving that
26:28
kind of flexibility during friends framework needs to
26:31
change as well. Right there is software that we
26:34
use is this leverages this prediction will need to change
26:37
as well. So we needed to build like a
26:39
certain degree of standardization around this
26:41
process which and and there are like hundreds of models passing
26:48
through this pipeline as well. So, Yeah, so is that does that
26:56
mean that? It's like you have various
26:58
customers and then these models are all going out.
27:01
But each model is individual to each customer.
27:04
You have largely. So right.
27:06
Yeah, the digital training is built for the customer in
27:09
somewhat hand created process. That represents each customer's
27:14
Network and they all have unique challenges, but but the model
27:19
types, right? So once you solve A model for a
27:23
pump. Like, I end up picking pump
27:25
here, it's common simplistic as a tear, but Tom, please 200
27:31
types, right? They all have different
27:36
characteristics. Same the model the model that
27:39
you build for a problem. Typically Works across that you
27:42
stayed like different from curves.
27:44
Mean, you need some degree of tuning and optimization, but in
27:48
when it comes to other chemical biological processes, and when
27:51
these assets work together, A combination lock each customer
27:55
and really different, right? They they have different models
28:01
built to represent that is doing So sticking on reproducibility,
28:09
just for one more, not to beat a dead horse, but I know that the
28:14
idea of having the data which is changing and it's constantly in
28:19
flux and then you need to go back and you need to try and
28:23
reproduce that can be difficult. Sometimes, how did you go about
28:27
solving that problem? so, so Intel like
28:40
reproducibility applies to the process of rendering pipeline,
28:47
right? It starts with let's say you
28:50
collected some historical raw data.
28:53
That was in customers email somewhere, the email that was
29:01
put in a drop ball. She started modeling in real
29:05
time. There is also this data being
29:07
collected from directly from sensors, coming through Supreme
29:10
pipeline, somebody who is starting to build machine
29:14
learning models for this digital twin starts working on
29:18
historical data when he or she needs to connect this real-time
29:24
data to the to this historical data to build models.
29:30
That works and works for works as one continuous string.
29:36
Right? And this all works like all of
29:40
this happens, they somehow make it work.
29:43
It's just that at the end of building this, their loss lose
29:47
track of, what, what exactly happened, how we made it work.
29:50
And there is this data, doesn't? These units have changed between
29:59
historical and real-time or the starters?
30:02
They added a new sensor in Between in between these two
30:07
time frames or some other thing has happened and there is like
30:10
discrepancies between this data. So in terms of data, what you
30:13
will need to do is provide like a centralized repository to do
30:20
training like Bo for both real-time and historical data
30:24
use cases and make sure it's one stream of connected data and now
30:30
this is really like an iot style problem ayuda time series
30:34
forecasting, cell phone and not every industry.
30:35
Trees has to face this. But we do but and in in in Time
30:42
series forecasting, this is a fairly common issue.
30:46
Now, the exercise of model building.
30:49
Once the once the data scientist goes and builds this, it's
30:54
important to rebuild it in a separate environment, just to
30:58
make sure it's not like a random stroke of leg one.
31:02
There are like compliance requirements and X rays.
31:05
Spencey like model for understanding requirements
31:10
people asking why is it predicting this this way right
31:14
things like that when but more importantly retraining right so
31:19
when as time passes more and more data becomes available this
31:25
is in by the nature of times it's forecasting and models can
31:29
do better or in other cases where there is like it concept
31:34
of data drift, right? So 2020, for example, was
31:39
completely different here for everybody.
31:42
It basically broke many time, series forecasting, models out
31:45
there that really dealt with real time.
31:48
They attended, as you know, our industry like water rainfall
31:50
patterns change their lot of things change because people
31:52
behaviors change during covid. And essentially, all the models
31:58
built before 2020 became obsolete, right?
32:02
So reap training solves, I like real-time automated retraining
32:08
is also. Now, it's extremely hard to do,
32:11
but reproducible training is like a step, 12 attaining,
32:17
automated retraining. And I want to talk about the
32:22
automated retraining in just a minute because I also find that
32:26
idea fascinating on whether or not we should be doing that and
32:31
how we go about doing it. But I wanted to ask before that
32:37
about the data scientists and their workflow and how I know
32:40
that you have a few data scientist on the team.
32:43
So how do you enable them to collaborate?
32:47
Yeah, so I don't know if others have figured this out, but there
32:51
are signs workflow is and it's messy process.
32:55
It's and it's it needs in the nature at its at needs to be
33:00
that way. Like we need to enable that as
33:02
opposed to drawing a box around that process and saying this is
33:06
how you have to do it. So I have found that the its
33:09
creative process it's a it's it's all over and over.
33:16
I treating our problems like we have Have we had problems models
33:21
that we were unable to build four months and third person was
33:25
looking at it and something clicked, and then the mom got
33:28
fixed, right? So But how do you enable this?
33:34
If you have, we started out we sort of the local jupyter
33:38
notebooks running on people's laptop and that's obviously a
33:42
horrible practice. But that's I think that's how
33:46
many data science project Starts Now.
33:49
How do you enable sharing right? How do you give this to somebody
33:54
and Jupiter lab? And today is not book makes it a
33:58
lot easier but still It comes with, you can just do an entire
34:04
2009, little sense, cause then give it up, give it to a person
34:08
like, in a get to check, get tanned and make sure it click
34:11
run it. Now runs it.
34:13
It feels like all I almost guarantee, right?
34:16
So there are data structures in memory.
34:19
There are things that are saved and stored and configurations
34:22
generated these all needs to be shareable, right?
34:28
So I can't claim to have seen Out that process but we have
34:33
taken. That's right.
34:34
So you say to make a studio enables sharing without
34:41
compromising security, so share then Ali, Khan issued sharing,
34:44
is that 20% team? And sometimes I have we had
34:48
contractors to extend, our work. There is security requirements
34:52
there that you absolutely need to meet the data.
34:55
Needs to stay in certain regions.
34:57
Certain set of people who have already signed the contracts are
34:59
the only ones who have a Access to NDS are the only ones who
35:02
have access to it, right? So if you have a centralized
35:06
jupyter notebook server, everybody has access to
35:09
everything like you can't do that.
35:11
It's so there is permission issues that you need to figure
35:14
out. I'd say like Sage maker Studio
35:19
as an IDE does said reasonably. Good job of solving.
35:24
This the reproducibility of 2009.
35:28
Jupyter notebook that has talked about.
35:30
That's Not exactly solved, it has still has issues to figure
35:36
out, but I think so now we will see that somebody has answers to
35:40
that question. Like do I need to know right now
35:42
but in a few years I think that it's a process that will evolve
35:47
mature and it will change its changing even the last year.
35:49
Last two years, the way data science is done in a notebook as
35:53
a team that process is changing. And that's something that comes
35:58
up in the community a lot. And I know, if anybody here
36:03
wants to chime in, we've talked about it so much.
36:06
It are jupyter notebooks for production.
36:09
Should we just like not even think about trying to bring the
36:13
jupyter notebooks into production.
36:15
So let us know if you have any opinions or thoughts on that.
36:21
There's a great question that just came through that I want to
36:24
highlight and this is one I imagine you've got a few
36:28
considering you've been in tech for so long.
36:31
Ideally one around ml Ops, but you can tell us other War
36:36
Stories if you want. So Vishnu, do you have any cool
36:41
War Stories? You can share with us anything,
36:44
maybe model quality, or model deployment failed and it was a
36:48
complete disaster and what did you learn from it?
36:53
The big kind of the big one it does.
36:56
There are many every day, it's every day, right?
37:01
Well I'm just going to go with one of the top.
37:05
Easiest ones that came to my mind.
37:06
Like we have this South American customer who had this particular
37:11
data sparsity problem. There was a we were basically
37:16
improving the effluent quality and what that means is that
37:19
Water waste water. That gets goes from your home's,
37:22
gets treated and sends back to take a river or a lake and There
37:26
are compliance. The governmental compliance
37:29
requirements on how the quality of Honor should be like there
37:32
are certain chemicals in certain constellations.
37:34
Right now, this is one of those things that gets measured
37:36
manually. And in this particular case it
37:39
is extremely manual extremely manual.
37:45
Once in a day process, that goes somebody goes like dries down to
37:52
a this location collects. This data sends it to a lie.
37:57
Two days later. You get the result, then they
38:00
enters in there, they can Excel sheet somewhere in there and
38:05
then we can set a point two days late.
38:07
Meanwhile, our system is trying to optimize this in real time.
38:12
So this is a problem that your person that but we were very
38:18
earlier like a start-up at that point and it's it doesn't big
38:22
contract but it's Tried to solve the problem, right there.
38:27
It's there are creative ways. Like but like we talked about
38:32
building virtual sensors that can correlate to like other data
38:40
points that I had to have real-time data and then build
38:44
like a virtual way of virtual way of representing this F1
38:48
quality thing but ultimately the lesson I learned was that some
38:54
data science courses are not solvable and it It really
38:57
depends on the data like win this, and that, it seems so
39:02
trivial right now, but it was a very expensive lesson, we
39:07
suspend. Yeah, go ahead, go ahead, finish
39:11
it. We spent man hours and I don't
39:14
even want to talk about the expense in trying to figure out
39:17
the answer. We learned quite a lot from it,
39:20
right. It helped in many of our
39:22
customers that came after that who had the same problem.
39:25
But now, Not as bad, right? But it's the it's a resolution
39:29
of this data that really caused this issue, but the technology
39:35
that the kind of experiment and built for this customer helpful
39:39
in solving, some of the other problems.
39:41
But this particular one simply didn't work well.
39:44
So it was you were saying that you could solve that problem.
39:49
And then at the end of the day, you couldn't.
39:51
Is that what happened? Yeah, exactly.
39:53
What it was so bad. Yeah.
39:55
So some, some Atoms are not, you can build it is yeah, data
39:59
quality no matter how presumptive and arrogant you
40:04
think you are? And yeah, I'm gonna happen now.
40:06
That is latest neural network at it.
40:10
You don't have the data, you can build it.
40:14
That is amazing. That so that's something that it
40:17
reminds me of a talk that I had a few months ago.
40:22
And one of the people that I was talking to was mentioning This
40:26
idea of techno solution is mm I think is the name of it because
40:31
somebody can correct me if I'm wrong.
40:33
But the idea is that every problem can be solved by Tech
40:39
and it's a little bit of an arrogant idea that every
40:45
problem. I mean, this problem potentially
40:47
could have been solved by better Tech if you had more sensors and
40:51
you had all of that. But I when I was talking to
40:54
someone in the I was talking to this person.
40:57
They were talking about, you know, when it comes to bigger
41:02
issues, like mental health should that be solved by Tech.
41:07
And so that's a bit of a tangent.
41:09
I know we just went way into left field for everyone but it
41:13
reminded me of that because it's one of those where you say, oh,
41:16
we got this under control and we can figure out what the problem
41:22
is and we can throw a neural network at it and won't be able
41:25
to Like solve this and then you realize wait a minute if we
41:29
don't have the right data and we if we only have once a day data
41:33
and we're trying to solve this in real time, but our data is
41:36
lagging by three days, that is a problem.
41:40
That is. Yeah.
41:41
It's very difficult to argue that that can be solved.
41:45
No matter what kind of data scientist you can have the best
41:48
data scientist in the world on that, right?
41:50
So that's, that's a great one. There's another question coming
41:54
through in the chat. How would you know?
41:56
If the data has the information that will help you to predict
42:01
what you want or not, do you have pipelines to analyze it
42:06
first, but that's a good question and it's we have a few
42:13
things to do but my answer that question will kind of be largely
42:20
depend on the problem that we are trying to solve a little bit
42:22
different for each customer, but I talked about this.
42:26
Digital twin building process, right?
42:28
And this was something that came up very earlier on like without
42:34
good quality data. This is digital twin is not very
42:37
buildable. So how do you know that without
42:41
actually under taking the exercise of building this?
42:43
How do you know, you have the right amount of data?
42:48
So, several approaches for solving this like step one
42:53
starting from the business, you make sure the sales Sales, get
42:58
the right guidance. Right there is, don't look at
43:03
the sensor placement. Look at the level of maturity,
43:08
you have this customer they if we give them a feedback to ask
43:15
you install, a new sensor sensor will sometimes cost only 200
43:18
dollars to install and that will make a world of difference.
43:21
They willing to do that is their existing data quality.
43:26
T. Are you getting at least one
43:28
data point in 15 minutes? Every 15 minutes like this?
43:31
So, there are like fundamental checks that you can do then by
43:36
like a non technical sales qualification process or like
43:42
right at the before, right? Even before you start thinking
43:45
about model, sad, thinking about even data analytics on this,
43:50
then you only qualify. Like in our case, we only call
43:53
for the customers who we think has like a certain And initial
43:58
quality check mark. And it goes into like domain
44:02
experts, the wine experts alike and these are there in every,
44:07
every designs company. These are non data scientists
44:10
who are coming from the industry made an our case.
44:14
This is bought or quality export space water industry experts.
44:18
They They have been taught a certain degree of data analytics
44:23
and they look at data sets and placement they apply all of this
44:28
their expertise to this. Then they do an additional layer
44:32
of filtering. Now these the first level of
44:35
filtering is the cheapest the second level, it starts getting
44:39
expensive but a data scientist is the most expensive higher you
44:42
can make like but we need to minimize the number of Of bad
44:52
customers reaching data scientist.
44:54
It gets if you can be filtered out earlier, that's better.
44:56
So we equip them with like, this non-programmatic tools, like we
45:02
should we analyze data, things like that.
45:05
That will ask for other critical.
45:09
Less, critical sensors have enough data.
45:12
There are you seeing anomalies in critical sensors, things like
45:17
that, and they have another set of checklist.
45:20
But sometimes they Advance some of the advanced era problems are
45:25
only like figure out after you start building a model.
45:28
You do something's engineering you to initial model and you
45:33
realize that it's just it does not fitting doesn't work.
45:38
Right? So in there what we have tried
45:41
to do and this is we are not fully successful in doing that.
45:43
But as you gain experience over multiple projects there, you can
45:49
Accumulate this knowledge into like a non-terminal framework
45:52
that kind of. So let's say I pick a process of
45:58
flocculation flocculation is like a big tank in which you add
46:02
some chemicals and gets suspended.
46:06
Solids that gets on top of this tank, right?
46:11
So then there are various population mechanism.
46:14
So there are various models that go gets Built For This to
46:17
reproduce this problem. And we kind of build that into
46:21
an ottoman autosomal framework in which the key each model gets
46:24
tried one after the other after the other and certain degree of
46:28
I prefer under, tuning is performed, a certain degree of
46:30
these engineering is performed on all of this and is automated.
46:36
So before it assigned to supplies is saw her intelligence
46:40
to this process is automated, you see, whether like a certain
46:46
degree of qualities? Even correlation is even
46:48
possible. It was Is your name?
46:50
What's your name Norma? See numbers.
46:55
And then, and this is even before this is again before,
46:58
putting in, like, a lot of work from the other scientist, which
47:02
means you can filter out a lot of things when you go through
47:06
this set of steps. But sometimes things just passed
47:10
through and there were instances where we have spent the time and
47:13
money, only to realize that it just doesn't work and in those
47:19
Cases. You have to go back to the
47:21
client and say, hey yes, around the center for us.
47:25
Well no, we won't say sorry until we have given up on it,
47:29
but at that point we'll say, okay, can you get this more
47:33
data? Can we wait a few more months.
47:36
You cannot collect a few more data before we can start
47:40
modeling. Can you do this exactions?
47:42
On this pieces. There are scenarios that we have
47:44
not seen these systems operate in.
47:46
So can you add a little more chemical little less cankles And
47:49
then can we just call it all step testing rights?
47:52
Okay, so then, can we observe the resulting data points so
47:58
that we can capture the Dynamics in the models.
48:00
All kinds of things we do to improve the model building
48:04
process over time but ultimately none of that works.
48:07
Yes to give up. Wow, I find it fascinating.
48:12
This idea of going through three different tiers to get to where
48:18
you want to go and And I really like the way that you put it is
48:21
the data scientist shouldn't be touching it because they're so
48:24
expensive. They need to be focusing on
48:27
stuff that we know actually works.
48:29
And so first you get sales, they go through sales.
48:33
Then you have a subject matter expert that looks at it, that
48:36
knows a bit of data analysis. And then if it passes through
48:39
these filters and then it goes through that auto ml to make
48:43
sure that there's a possibility on that end and then it can go
48:47
to the data scientist. So that It is a really
48:50
interesting way of going about it.
48:52
I hadn't heard that one before so it's it's nice to know now my
48:57
last kind of question for you because I know we're getting
49:01
close to the end when it comes to because you're doing a lot of
49:06
sensor and you're doing a lot of Internet of Things.
49:09
What are some specific ml Ops problems that you feel are
49:15
involved in when you have the internet of things and you're
49:20
trying to to work with that. If there are any That many so,
49:30
yes, they don't have. Things means like having some
49:34
sort of sensors and I still want to classify that further.
49:39
These are all like time, serious problems, right?
49:41
Time series data is what your privately dealing with.
49:46
Not necessarily but in our case we are so.
49:48
But I want to differentiate between like a circle example,
49:51
like companies like Tesla. Who would you think have the
49:54
same issue, right? But in case of Tesla, you own
49:59
their own, the sensors are or they're really sensors, which
50:03
means they get to keep this quality of data that they want
50:12
Vicki to ask this called of data.
50:14
And It's of data that they want but in our case, it's customers
50:19
on the sensors, right? So it contributes has all these
50:24
data quality problems. We talked about which we need to
50:28
really enable data scientists and Engineers working on this
50:32
problem to be like reproducible and essentially.
50:39
Yeah, reproducible. What else comes in?
50:43
Then that's it. Iot time series, forecasting,
50:48
problem, that's somewhat unique to us.
50:51
There is also what we call as, like ASL models or things that
51:01
basically underlying has like a physics equation.
51:04
For example, in this flocculation example told about
51:06
it's there is a certain degree of biology kill process going on
51:10
in there but as soon as you add the boat load of chemicals to
51:14
it, Beyond a regular beyond your regular usage, it to the
51:20
characteristic stage, right? It's like you throw a ball over
51:25
the are typically you expect a particular trajectory, it goes
51:29
and falls in nflp you can Define that curve physics equation but
51:34
as soon as you throw another bulletin hit it it changes its
51:37
course, right? So these control system style
51:42
models and when that Gets combined with iot style problem
51:48
Sokka. Pump for example you can turn on
51:50
and off and it will chemicals. You can add control chemicals to
51:55
this flow Clash narration mechanisms they all has the
51:58
potential to change normally defined characteristic or
52:02
normally defined equation of this process significantly
52:06
enough that typical emel methodologies doesn't work.
52:11
It's a it's a, it's a cutting-edge bleeding edge,
52:14
research, Search field data-driven control model
52:17
predictive control, and we do some good stuff with it, but
52:22
that said largely data science problem again right now.
52:26
If the part of the ml Ops again, the time elapsed comes in its
52:32
facilitating the reproducibility on this as well.
52:39
Yeah, not to me in the last couple minutes about this
52:43
automatically retraining of the models.
52:47
And why you felt that was necessary?
52:51
Because of the number of models that goes through this pipeline.
52:55
It's just it's impossible for somebody to manderley Monitor
52:59
and monitor and we retrain as necessary.
53:05
It's a dangerous territory. Not easy to do.
53:09
It takes a lot of time but once you have deployed five pump
53:12
models, the sixth one, you have certain degree of confidence
53:17
that it's new data and new patterns on you.
53:20
All you need to do is like extra hyper under tuning.
53:23
That point you can automate the retraining but generally, yeah,
53:26
I do not, it's not easy. It's not recommended until you
53:30
reach a certain level of maturity with The Malt with
53:35
until you gain a certain degree of confidence.
53:38
With the models you've deployed in production.
53:41
It works in fintech. It works in attic.
53:46
It was for us but took a lot of time and effort to build it the
53:50
things like fraud detection. For example like they it doesn't
53:55
work. It's hard.
53:57
It's not easy to do. Like you'll have to the patterns
54:01
just kind of drift to that. Those kind of models have our
54:05
two to altering to behavior altering that that's a model
54:11
itself like the neural architecture.
54:13
For example, if they picked one, it doesn't work.
54:15
You need to rethink what consumer Behavior changed to and
54:20
rebuild it sometimes but sometimes it works.
54:23
In our case, most of the times Dynamics remain the same.
54:27
The pump is the pump the pump, sometimes units change,
54:31
sometimes they operate more frequently, it's a small beta.
54:33
So retraining is more applicable to our This case.
54:38
What what about tracks are you looking at when you're
54:42
retraining, like, when you're monitoring these models and it
54:46
triggers a retraining, what is it usually that the metronome
54:50
looking at to call? So that retrain?
54:53
Yeah, I like so there is a human in the loop by the way right now
54:57
or not it's automated enough. In to the point where you can do
55:04
a click in the get retrained, but they models generally emit
55:12
all possible Matrix. It can Pharmacy, may may Mac
55:17
whatever you want to call. It depends on the type series
55:19
model, You're Building other. My other type of models,
55:22
however, there are metrics, but you get to look at them and you
55:25
have a threshold, we set for each of those metrics at the
55:29
point of training and as soon. As the treshold drops based on
55:35
some sort of drifted that the you'll still see the near trust,
55:39
you'll go beyond that are sold. It triggers an alert, it doesn't
55:43
do the retraining, but yet triggers an alert and then you
55:48
can go and look at it and Slick retrain and then industry
55:53
training, right? It's automated but it's not
55:55
automated both at the same time, but getting to this point itself
55:59
is a good. It's a good exercise.
56:02
Yeah, that's so cool because I was going to ask that if there
56:07
was anyone in involved and if you still had the human
56:12
knowledge to be able to look at it, yeah, we need to or maybe
56:17
this is okay, we don't need to re trigger this.
56:20
Yeah, it's minimal. But that is, that is human
56:26
involvement. In every case nugget mean.
56:28
I want to reach a point where if they want to reach a point where
56:30
there is none, but Yep, I'll take whatever is there is the
56:36
idea of you can run into trouble?
56:40
If you start retraining models and you don't have it really
56:46
down solid, like you can retrain models without and then you just
56:51
keep retraining a minute and it causes a bunch of chaos.
56:54
Yeah, absolutely. Yeah.
56:56
I mean that happened happen all the time.
56:59
It said, little Less prevalent for our industry compared to
57:04
like a churn prediction model. Like things like that that they
57:09
change a lot. There are external factors that
57:11
depends on it but it compressor pump.
57:15
Yeah right there is I do think we can reach a point where there
57:22
is where are we training Islam or automated?
57:25
But Someday. Yeah perfect.
57:30
Well this has been immensely insightful for me.
57:35
I really want to thank you Vishnu and everybody that is
57:39
still here listening. Hopefully you enjoyed it as much
57:42
as I did. The yeah, that's about it.
57:46
I think we're going to call it a day here.
57:49
Thanks for listening and we will see you all later.
57:52
Thank you damage on. This is really nice talking to
57:54
you and we will dance. Everyone else.
57:27
Someday. Yeah perfect.
57:30
Well this has been immensely insightful for me.
57:35
I really want to thank you Vishnu and everybody that is
57:39
still here listening. Hopefully you enjoyed it as much
57:42
as I did. The yeah, that's about it.
57:46
I think we're going to call it a day here.
57:49
Thanks for listening and we will see you all later.
57:52
Thank you damage on. This is really nice talking to
57:54
you and we will dance. Everyone else.