Latest / Agentic Conversations (formally mlops.community) / Explainability in the MLOps Cycle // Dattaraj Rao // MLOps Podcast #138

Transcript

Explainability in the MLOps Cycle // Dattaraj Rao // MLOps Podcast #138 · Agentic Conversations (formally mlops.community)

0:00
Hey, hi everyone. My name is dr.
0:02
Ostrow, I'm a chief data scientist at persistent running
0:05
the AI research team here. Working on some cutting-edge
0:08
technologies around ml Ops knowledge platforms
0:12
responsibility. I and next best action, how I
0:16
like my coffee, I am based in Goa India and I love my
0:20
traditional, Indian South Indian style filter coffee, super
0:24
strong and I will take that over any cappuccino.
0:26
What is listeners is another edition of The ml Ops Community
0:30
podcast. I am dimitrios Brinkmann and
0:34
today I am joined by none other than mr.
0:38
Vishnu ratchet conda. Nice nice.
0:42
Very good job for. Not see that Demetrius over the
0:44
course of our friendship, over the course of the two years
0:46
Demetrius and I've been doing this.
0:47
I think now we've just hit the two-year, Mark the divisions.
0:50
Got a lot better at that pronunciation.
0:52
Started off or Honda then it went to ratchet Honda and now
0:57
it's Brad Sherwood shh, man. It's The easiest last name, but
1:01
I can empathize because I do not have the easiest name either.
1:05
So, I feel for you. Anyway, today, we're not here to
1:10
talk about last names. We are here to talk about
1:14
machine learning operations and has always had none other than
1:19
dr. Raj, on the podcast, I'm super
1:23
stoked about this because he's one of those guys that I feel
1:26
like was a I'm just going to Pat myself on the back.
1:30
It was an incredible find on my part.
1:32
Although there's probably a ton of people that know about him
1:35
already because he has written a book and he was, he's written
1:39
one of the o g ml Ops books and we talked about it in the
1:42
podcast, but since then he hasn't written more books and I
1:47
just saw him on. I think it was a medium post or
1:51
I saw the book. I may have been getting into his
1:54
book and I was like, dude we gotta get this guy on the
1:56
podcast. Because as we mentioned, so
1:58
many, Any times in this podcast, the book is incredible because
2:02
the principles that he presents are Everlasting, it's not one of
2:07
those books that you read and then a year later, as technology
2:11
continues it to advance it. No longer is valuable.
2:14
It is a book that you can keep on your shelf and refer back to
2:18
it because of those principles. And you see the way that he
2:21
thinks it's like that. I mean, I don't know about you.
2:24
Yeah. Yeah.
2:25
I mean, that's Raja Rao, just to give you guys a little bit of
2:28
Here he is actually the chief data scientist at persistence
2:32
systems which is a IT services and software engineering
2:36
services provider based out of India.
2:38
And prior to working at persistent, he spent close to 20
2:42
years working in a number of different technology leadership
2:45
roles at GE, including their research division, their power
2:48
division, their transportation Division.
2:50
And I thought that what he brought was such a great
2:54
engineering mindset to this podcast, this Shinto.
2:59
The machine learning problem space, and you can see that and
3:01
how he structures his answers are, we structured the book a
3:04
very structured thinker, which I always appreciate.
3:06
It helps with the communication and it helps with actually
3:09
figuring out what the Insight is that people are offering and I
3:11
give you credit to Beatrice, this was a great, find a great
3:14
podcast. Well, thank you very much, dude.
3:16
One thing that I'm a little bit bummed, we didn't even get to
3:19
get into is he has 11 patents around computer vision and we
3:25
didn't even get to talk to that at all.
3:27
I mean we were so enthralled. All the other cool ml up stuff
3:30
that he's been getting into the like the video analytics work
3:34
that he's doing right now and 11:00 patents that he's got
3:37
going on. It's incredible.
3:39
Yeah, I think my favorite part of the podcast which you guys
3:41
will hear more about are the three top areas that he's super
3:44
interested in when it comes to machine.
3:45
Learning what he sees as up-and-coming what he's
3:48
investing his time and his company's time into.
3:50
I think that was great to talk about and then think a bit
3:52
bigger about. What about you Demetrius?
3:54
What was your favorite part? I like the end when he talks
3:57
about the patterns and when he talked about how he was going
4:00
from rule-based systems and you would deploy rule-based systems
4:03
into a system and then when he made the jump into machine
4:09
learning models that you would deploy into systems and how
4:12
those are effectively the same thing.
4:15
It's just now, this machine learning model is much smarter,
4:17
but you can look at everything that goes around, deploying a
4:21
robe a system or a machine learning model into a system and
4:25
how he just really broke down. On the patterns that are still
4:30
the same and that you can hold on to and you should be thinking
4:33
about whether you're doing rule-based or machine learning.
4:36
I thought that was awesome. And that was probably my
4:39
favorite. Yeah, great conversation.
4:42
Let's get into it. Let's do it before we do.
4:45
I want to mention that this podcast is brought to you by
4:48
none. Other than Super Wise.
4:51
They are a machine-learning monitoring tool, and I have had
4:54
the opportunity to meet the whole supervised team.
4:57
They sponsor The podcast, they are what you need.
5:01
If you are looking for machine learning monitoring on any level
5:05
at any scale, you can check out some of the episodes that we
5:08
have done with. The founders will leave that in
5:11
the show notes if you want to dive deeper and you just so
5:14
happen to be looking for some machine learning monitoring
5:17
tools right now. Check out Super Wise dot AI to
5:23
get all of your monitoring needs met.
5:25
Now let's go into it. That's a Rog.
5:34
It's great to have you on this podcast, man.
5:36
I'm very excited to talk with you.
5:38
Yeah, likewise things Demetrios things, fish room, so you've got
5:42
a pretty incredible background. You've been in the game as we
5:46
could say for like over 25 years I think.
5:51
And you studied mechanical engineering at school and then
5:54
you went on to pursue this software engineering at General
5:58
Electric working on developing monitoring systems for rotating
6:03
Machinery. Can you give us like just that
6:06
thread? What drives you?
6:08
What brought you through the last 25 years?
6:11
It's 24 actually. But yeah, we can round up, man.
6:16
It's all good. But I started off mechanical
6:20
engineer from this where I live now, also I moved back to my
6:25
hometown where I studied and what I mean, engineering always
6:29
fascinated me. I guess I always wanted to be an
6:31
engineer. And to be honest, I didn't in
6:34
the academics. I didn't get through computer
6:35
science. So I was always missing out on
6:38
on that. So that was something that I
6:41
kind of self learn. Even while doing my mechanical
6:43
engineering, I was picking up learning new stuff in computer
6:46
science. And then I think what really
6:48
worked is that discontinuous things to Lots over the 24
6:50
years, 25 years. I think what is really stuck
6:53
with me, what is driving me is the the fact that I can learn
6:57
new things I get scared that one day, I'll wake up and I won't be
7:00
able to learn new things. So I mean that's something that
7:02
scares me every morning. I just whether it's a podcast or
7:05
reading something on pocket or Kindle book, I just kind of pick
7:09
up things and that's what I like to do.
7:11
If you follow me on LinkedIn, I'm like sharing things almost
7:13
every day three or four posts just something.
7:16
Somebody has done some interesting model that has come
7:20
in the market on an LP or something around ml Ops lot of
7:24
great content recently on ML of sucking ruling.
7:26
You're fantastic. Marcus some plug there.
7:30
But yeah, I think that's what really drives me.
7:33
So after my education, in mechanical, engineering.
7:37
I really wanted to do something fun and computers because I, I
7:40
really was not into a classical formula based system.
7:44
So I got into numerical methods. I built like a finite element
7:48
website, so started before trial, and then cause Photon was
7:51
the only language to learn at that time.
7:53
This was like in 98, 97. And then, then I Java was just
7:58
Going up some micro system and released Java, I think 2.0 that
8:01
time. So I just picked up Java because
8:03
the interface was nice. So my my thing on Java was not
8:06
about cross-platform server-side nothing I just wanted to know
8:09
what did my code to look pretty on the screen because Java had
8:13
some fancy charts which Fortran didn't and then that's what I
8:16
did. I built like a website at that
8:18
time there was some people Java applet which was the big next
8:21
big thing. I mean, some of you may not even
8:23
remember what these plate was like the web browser was The
8:28
running applets in the web browser was the thing that time
8:31
in 2000's and that's what I did. I built a website with
8:35
java-based fpm, software for numerical analysis, and
8:39
calculating stress on models. So in that, that was very
8:44
pretty. In fact, that's the thing that
8:45
got me the job in G. So G was trying to do the same
8:47
something similar. They were building design
8:49
software and I got my first job in.
8:51
Bangalore, I moved to Bangalore, from go and work there for four
8:54
years, building design software, and then I within G, I got
8:58
Another opportunity I wanted to move out to explore the world of
9:00
it. So I moved to the US I moved to
9:03
Norfolk Virginia and spend like nine years there and I was
9:07
working with g-power there and we built some remote monitoring
9:10
software for gas turbines. That's the rotating equipment.
9:14
So this this was actually an acquisition by khalji power.
9:17
They're a very nice novel software for capturing any
9:21
anomalies that happened on machines, you could Define rules
9:25
saying that if some thresholds are violated, Flag some alerts
9:29
and you should use Concepts, like, even know Emily yet.
9:32
It was purely rule-based. It was fuzzy logic, lot of cool
9:36
stuff but very deterministic and then slowly we started at
9:40
g-power getting into neural networks neurotic.
9:42
So very big at that time when clean vanilla and also that we
9:46
started getting into that and then back in 2013, I wanted to
9:51
move back to my home town or actually move back to India.
9:55
So I got another job through g, a g transportation.
9:58
In and move back to Bangalore. And then at Bangalore we worked
10:02
so many very interesting work in computer vision.
10:04
So that was my first official for into machine learning where
10:08
we built, really interesting computer vision models, opencv
10:11
based stuff, lot of very new computer vision.
10:13
Stuff at that time, I mean some of this object detection Etc and
10:17
which became very popular. I actually got like around 11
10:19
patents in that domain where we actually had a product, whether
10:24
the camera mounted on a locomotive and from the camera
10:27
view would monitor The trail we track and find anomalies.
10:31
So that became very popular and that was one of the one of I
10:34
will say highlighters for my career and then after that like
10:37
four years back, I'm out I wanted to move back to my
10:39
hometown and got this amazing job at persistent as a chief
10:42
data scientist. It was a very high position and
10:46
I'm actually now at persistent running the AI lab for for them.
10:51
And this is like have some really smart data scientist
10:54
working for me, we have some really initiated as that we work
10:57
on, like Federal Did learning responsibility a lot of work in
11:00
ml, Ops knowledge, graphs, next-gen NLP systems.
11:04
So we get to do a lot of cool work.
11:06
So yeah, that's where that's where I am today and enjoying
11:09
the but still learning. So I get to wake up in the
11:13
morning, done your stuff. So I definitely something I
11:15
enjoy awesome. Yeah, no, thank you for taking
11:18
us through that. That was Illuminating to here
11:20
where you started and all the experiences you gathered along
11:23
the way and where you are now and where I'd love to dive in,
11:26
as they're used to work at persistent, what's Interested in
11:28
right now is taking a look at your LinkedIn.
11:30
You clearly just wrote a book and Cara, sticky kubernetes.
11:34
I see The Thirst to learn and the drive to the Curiosity and
11:39
you and I'm kind of curious from your perch at that she has
11:43
achieved a data scientist at persistent.
11:46
What are the top three areas of machine learning that you're
11:49
most interested in? I've seen you share things about
11:51
large language models, I've seen you share things about
11:54
responsibility. I am a lobs other sorts of
11:57
modeling. Meek's bioinformatics, by the
12:00
all kinds of stuff. Like what are your top three
12:02
areas that you're most interested in?
12:04
And that your Champion championing work on at
12:06
persistent? Yeah, this is a very good
12:09
question which new. So I will say the top three
12:11
areas, definitely will start with the knowledge platforms and
12:14
knowledge graphs. So definitely we do a lot of
12:17
work on large language models but one of the and persistent we
12:22
are like technology services company like a, we had a billion
12:25
dollar company. Now we make the end a lot of our
12:28
Work is for Banking and Financial sector and Healthcare.
12:32
So those are the two most of our customers to measure customers.
12:36
We do some work for industrial also in isvs but gfsi en banc
12:40
and Healthcare are two Prime. And a lot of these areas just
12:44
large language models, don't fit the card.
12:46
We do need to capture the context.
12:49
So capturing contest, extracting entities, capturing the
12:52
relation. So knowledge platform becomes
12:53
very important. You need to be able to explain
12:56
why you're providing certain recommendation or while you are
12:59
giving a search result. So explain the ability of this
13:03
language models using a Knowledge Graph.
13:05
I think that is one area and I wouldn't say we figured but we
13:08
have a some very interesting offerings that tackle this so
13:12
that will be one. Second area is definitely around
13:15
responsible. Something we are doing but we
13:17
tend to get more into interpretability and explain
13:20
ability of ml models again. Banking in healthcare customers
13:24
we are in a very regulatory environment we have to prove why
13:27
a particular Loan was approved or know or why a particular
13:31
diagnosis was provided per patient.
13:34
So interpretability explain, it really becomes very important
13:36
and then one of the things that really a lot of customers.
13:39
I see not enough, customers spend time is to understand
13:43
ecosystem in production, how the data drifts how the concept
13:46
drifts, so that is one area of ml Ops.
13:49
I see a lot of cap and that is another thing that we heavily
13:52
promoted persistent to help our customers to understand the
13:55
Dynamics of the data. And then, the third area, I
13:58
would say would be something around privacy preserving AI
14:00
again, it kind of falls under responsible AI but that is one
14:03
area. We we see is going to grow
14:05
heavily because, like they say, AI is a new electricity and data
14:09
is the new oil, but you're seeing cases where people are
14:12
getting sued for using data without consent.
14:15
So, especially in the healthcare space V, we don't have access to
14:18
data. We have very dummy data in the
14:20
public, repositories, on which we are supposed to build model.
14:23
So we really see something like Federated learning or
14:26
confidential Computing can really help.
14:28
Solve that problem. Keep the data private and use
14:31
only the right insights and only use the radar with right
14:34
governance models. So I think to achieve a long
14:37
answer to your wish definitely knowledge, graphs ml, Ops
14:40
responsibility and privacy-preserving are these
14:42
three areas. I would say are the art of focus
14:44
areas in persistent research and definitely on my top list that
14:49
was extremely Illuminating to hear.
14:51
Thank you for sharing all that. I always like to ask our guests,
14:53
what kinds of things they are interested in, what kind of
14:56
things are animating them and I To dive deeper into one
14:59
particular point, which is you talked a lot about how large
15:03
language models. While they're interesting may
15:05
not necessarily be sufficient for a task because of the
15:07
context, that's required. I'm personally like, I'm very
15:11
interested in diving deeper into that point because we're really
15:13
seeing LMS, have a huge moment right now.
15:15
I mean, the interest is very high and we're seeing a lot of
15:18
discussion around that in our community.
15:20
And so I was wondering if you could give an example of a use
15:23
case, where an llm was not a good application for the
15:27
natural. Which sort of task that you
15:29
might have used. And I understand that certain
15:32
things are confidential, and then you can share the total
15:34
details. But even a toy example, would be
15:36
very helpful to understand that more.
15:38
Yeah. Yeah, this is, this is this a
15:40
good point and, but by no means I'm saying that LMS are not
15:44
effective or powerful. Did they are definitely amazing.
15:47
So the thing that so, typically the some of the traditional use
15:51
cases for LM like question-answering or your
15:54
summarization, so we saw some click one of the Samples.
15:58
We had was that this x research, okay?
16:00
One of the customers was looking at Texas in documents on the
16:04
internet and like text bulletin is published and then you would
16:08
want to query that text text where this is the Virginia state
16:13
tax or some GST. So if there is a text update how
16:17
would you extract that information.
16:19
So we started with the LMS and we could ask some questions, we
16:24
could extract the text, then we started with a predator in model
16:27
fine-tuning. Some existing Corpus that we had
16:30
scraper from the net and then we build a question answering
16:33
system, which was pretty good. It was giving the relevant
16:35
answers, but we could not justify because everything is as
16:38
well. M says, the whole idea is they
16:41
convert your raw text to embeddings and then that
16:44
embeddings is what is used to find semantically matching
16:47
sentence also or a paragraph? So, we could find the right
16:50
answers but we could not justify like how it would be like a text
16:54
rule. So in that use case, what really
16:57
work. It for us would be.
16:58
We wanted to extract specific entities like which county had
17:02
how much person texts and what was that?
17:05
What is the imposition type? So these specific entities, I
17:09
would say related to the text domain.
17:11
So we change the problem to an, ER, problem.
17:14
So, we had specific entities that we extract, we had some any
17:18
relationships that we extract. And then the key thing was
17:21
building a Knowledge Graph. So a knowledge graph would then
17:24
start relating this entities and to each other.
17:26
So if Glock So County was an entity 6.5% was a numerical
17:31
entity and then say good Stax was in position type.
17:35
So saying that Glogster County from this particular date from
17:39
2006, the sales tax would rise to 6 percent that became a
17:43
knowledge that we will store in the knowledge graph.
17:46
So this particular fact, now we could keep it as a sentence in
17:50
as an llm and do a matching. But as you can see our knowledge
17:53
graph because we are showing this as a relation, it gets very
17:56
complex. Now I can do queries like Like
17:58
tell me all counties who have greater than 6 percent sales
18:01
tax, since 2016. So now you get the structure of
18:06
your data, you get a semantics of the data and that becomes
18:09
very important. So, to answer that you can't
18:11
specific question, you could answer to the point questions
18:15
using LM, but using a Knowledge Graph, you can do much more.
18:18
You can explore the space better and make it more explainable.
18:21
That that's what we saw. This was just one example.
18:25
I think that that's that's that's a great example.
18:26
I think it's very it neatly summarizes how it really depends
18:32
on the task, right? It depends on the task at hand.
18:35
What the president based on the problem you design the solution
18:38
and I think that's an obvious statement to make, but sometimes
18:41
particularly as practitioners of a particular art or a particular
18:45
technology, we get excited about the potential of the technology
18:49
without necessarily considering exactly what kinds of problems.
18:52
It's best suited to solve and it seems that what Kind of saying
18:55
is for certain tasks, we're exploring space with particular
19:00
constraints, or with a particular structure is
19:02
important. A tried-and-true approach of
19:05
named entity recognition and sort of mapping the knowledge
19:09
and structuring the knowledge. That way is ostensibly more
19:11
powerful, but a little bit less, clear method of like a large
19:15
language model and this is just one example.
19:17
Of course, there are plenty of cases where lme's are useful for
19:20
for sorting through much larger spaces of data, where you may
19:23
not have as much, A priori understanding of like what kinds
19:27
of entities are important. And what things, you what
19:29
relationships you want to identify sounds like, so, am I
19:32
summarizing is correctly? Yeah.
19:34
Absolutely. It summarizes.
19:35
Well Lieutenant Lee I'm Emmanuel Mason machine learning engineer,
19:42
at stripe and author of building, machine learning,
19:44
powered applications. And if you don't want your
19:47
machine learning models to explode, well, you should
19:50
subscribe to this podcast. So is this something that you
19:57
feel like will change in the future?
20:01
Or is that just an inherent problem with LMS?
20:06
Well, I don't see this as a problem, I see the problem with
20:09
the hype of LMS. Like I think I would people are saying LM is
20:13
like the next best best thing after sliced bread.
20:16
And they can they have throwing LMS is solving everything.
20:19
So, in a real world I think we just need to like like visual
20:22
crack. He said it depends on the
20:24
problem. Like, some problems.
20:26
Like a pure question answering solution even conversational AI.
20:30
I think LM start doing a fantastic job and they can they
20:32
can do really well but if you are go to get into the knowledge
20:35
part, you want to extract knowledge structure, the
20:37
knowledge and answer questions with giving concrete evidence
20:41
and as a knowledge graph is not like a competing technology.
20:44
You use LMS even any are the today, the best?
20:46
Any other techniques are based on LM some in Transformers I
20:49
have beaten most of the benchmarks for.
20:51
Any are so Use lme's to button you to extract knowledge.
20:55
Not just blindly use llm to an solve any problem.
21:00
So, I think that's the main difference.
21:02
I see the llm with a context-based Knowledge, Graph
21:05
growing, as it application area, rather than just pure llm this
21:11
thing, right? I see that cool.
21:12
There is something that I know both of us wanted to talk about
21:16
and you mentioned it for a second.
21:18
When you were given a bit of your background, it's around the
21:21
remote monitoring and Diagnostics and you said
21:25
something before we hit record, that is cool.
21:27
And I wanted to bring back into this conversation right now
21:31
which was around how the ideas didn't really change.
21:38
Like you find yourself doing the same thing with different
21:42
technology in the beginning. You were using very rule-based
21:45
system and then you transition to now a machine learning based
21:48
system but there's still some fundamentals that didn't change
21:52
their And can you give us what exactly it was?
21:56
That didn't change. Yeah, and that example was more
22:00
around like in when I was doing remote monitoring and
22:03
Diagnostics. The idea was you would at that
22:05
time, they would be called domain expert or like
22:08
engineering, experts, who compiled certain rules.
22:11
So rules would be something like if the input temperature exceeds
22:15
so much value at this particular Notch, you may have an alert so
22:19
changed the operation. So as simple, as a simple
22:22
instruction, Checking for threshold.
22:23
So they would write this rule. So this was a domain knowledge
22:26
in the mind of a when expert and you would capture that
22:31
knowledge, as a rule, put it package it as a rule, and send
22:34
it to a system where it will, this rou inference would happen
22:38
and the rule would run and the rule would trigger on the real
22:41
machine. Whenever the temperature cross
22:43
the threshold and you will get an alarm back.
22:46
So that was basically in a nutshell.
22:48
I mean, real reality is much more complex but a simple
22:50
example that would be how a Monitoring and diagnostics
22:53
system would run, you would send rules package them for a
22:57
particular device like this case.
22:58
Device could be a turbine and you would get an alert back
23:02
whenever certain rules or would be violated so that it was
23:05
information for the end user to do something now with ML and
23:09
with ML Ops in a way the way I look at it, you're pretty much
23:12
doing the same thing. You have a development
23:14
environment where you are instead of writing rules, you
23:17
have some training data that you're pulling, you're building
23:20
a model using some gpus or some all Maybe even some low code ml
23:24
tools, like Auto ml, you build a model.
23:26
Once you're happy with the model, you you test it on your
23:29
training and testing validation data.
23:31
If when you're happy with your Precision Recall now that model
23:35
becomes like your likes equivalent to a rule, you send
23:38
it to a to a machine. Now, this could be in the cloud
23:41
or it could be an edge device and now that model will run
23:44
continuously or live data and it will again trigger alerts
23:48
whenever certain thresholds internally.
23:50
Would now, this model is much smarter, now, The rule case,
23:53
the, the domain expert need to figure everything out.
23:56
Like, what is approximately, what should be the temperature
23:58
that should change. But now this ml model has been
24:01
trained on a lot more data. It has seen probably the way I
24:05
would say it is seen the world more because you're given its so
24:08
many examples of real-world data like this.
24:10
So this is a very it's equivalent to that domain person
24:14
writing those rules manually. So all you have changed it.
24:16
Instead of some domain person writing rules.
24:18
You have given a data and made that made it learn, but the Ops
24:21
part Remains the Same Same, you are sending the rule out.
24:24
The inference is happening and you are getting alerts whenever
24:26
the rule change now. So that is awesome.
24:29
Yeah, I understand what you're saying if I can just reiterate
24:33
that back to make sure that I am hearing you clearly it's that
24:37
instead of having to figure out a hundred or two hundred or two
24:41
hundred thousand rules and hard-code them into the system,
24:46
you deploy the model and that is going to figure out the rules
24:50
and it's going to be much smarter than All of these
24:53
hard-coded rules because it's going to be able to understand
24:58
the data and interpret the data. In a way that these hard-coded
25:00
rules wouldn't have been able to.
25:03
Exactly. Yep, that's an excellent
25:04
summarization. So that text and then just to
25:07
add to that. So what remains same is the way
25:10
that you are sending this entire, I will just call it
25:13
intelligence. It could be rule, it could be a
25:15
male model, you're sending this packet of intelligence to an
25:18
edge device, making it to influence and getting some
25:21
insights back as an Alert. Now what remains same is this
25:24
communication mechanism to send it and invoke?
25:27
It also, what remains same is a lot of other things.
25:30
If you think about it, things like bias, like just like animal
25:34
model may be biased on certain factors.
25:37
Now, if I, if I if that guy building those that rules says,
25:40
hey if that gas turbine is in this country is in one of this
25:44
country's, I don't want it to fire Beyond this temperature, so
25:49
that is a bias that that person is intentionally adding to the
25:51
system. My model may be ending this bias
25:54
but if there is a bias which is not supposed to be, right?
25:58
I wouldn't want the human to be adding that as a rule nor would
26:01
I want my ml model to learn that from the data.
26:04
So the fairness checks, the explain ability all becomes
26:08
relevant, it is just that, it's a different time.
26:11
It's a, the model is different, but all the infrastructure
26:14
deployment, all the mL of species, same thing with drift.
26:17
I mean, the data is going to change in production to tomorrow
26:20
that turbine is going to vary over time.
26:22
So, uh, After another one year, there will be lot of degradation
26:25
in the turbine, you will want to change that rule.
26:27
So you would want to check for that drift in this case, the
26:30
physical drift. Similarly, you want to get
26:32
collect new data and retain them and model.
26:34
So, in my experience, I mean, though we didn't call it ml at
26:37
that time, it the infrastructure and the rule deployment, the
26:40
intelligence deployment remains same.
26:42
It's just a lobster is because it is ml.
26:45
So, it's very interesting that there's a lot of similarities to
26:48
and differences to this the way that you summarize.
26:51
How does think about Intel? And systems and how to ship
26:55
those kinds of systems. And what kinds of features are
26:58
the same in different as the kinds of intelligence that we're
27:00
working to embed into systems as change is as awesome and very
27:03
helpful because very to the very much to the heart of what it is
27:07
that we are trying to do with machine learning, which is baked
27:10
some kind of intelligence in a way.
27:12
That's right now, very difficult but we hope someday through our
27:15
work and the industry's work will be very easy into the
27:19
operation of products and I want to use that fundamental Set of
27:23
observations that you offered as a way for us to start talking
27:26
about your book. You released a book recently
27:27
with O'Reilly, called Kara's to kubernetes.
27:30
And it's I love the title sticking with the caves.
27:33
And I want to start first by asking you, what is it that
27:36
compelled you to write this book and what sin and can you give
27:39
our listeners a little bit of the story of how you got to it?
27:43
Yeah. And first of all, thanks for
27:45
asking over the book. I somehow missed that in mind, I
27:48
often tend to do that. We have a lot to talk about.
27:50
So yeah, I'd love to hear about it.
27:52
There's some. So to be honest writing a book,
27:55
Something that was always on my bucket list.
27:57
I really like sharing my things. So, even a lot of my colleagues
28:00
who also wash for it, I love to share my experience.
28:03
If there is a new technology, I learned, I would love to share
28:05
that even LinkedIn. I tried tend to put a lot of
28:08
these kind of posts to share information.
28:10
So so I really wanted to compile and, and not being from a
28:13
computer science, background from my mechanical engineering
28:17
background, and then picking up right from the basic software
28:22
languages and then getting into machine learning, I felt I
28:26
brought a different perspective to machine learning.
28:28
It's not like a typical somebody who has done the degree in
28:33
computer science and gone through it.
28:35
I kind of brought more practicality.
28:37
So I really wanted to share my unique perspective and that is
28:40
the one of the main reasons. And then when I thought about
28:42
it, I mean, it had been in back of my mind, but especially with
28:46
work at GE Transportation. The, with the work on the
28:49
computer vision project, we really spent a lot of time
28:52
trying to master a system to deploy Intelligence to the field
28:58
and then that was something I really saw that.
29:00
Especially, at that time. Things like Cuban artists was
29:02
new, you flowers. I think Dave and Chuck had just
29:05
released his first YouTube video on kubernetes.
29:08
It was that time and so we didn't really have any tools.
29:10
So in to be honest I have after I got your invite, I went back
29:14
to the book and check, I have not mentioned ml office in the
29:16
book, though, the book is called carrots to given.
29:18
It is Journey of 4ml model to production.
29:20
I don't call it ml obstacles. Envelopes was not a word at that
29:23
time. They think that is What I wanted
29:25
to share that taking a model, taking this intelligence,
29:28
putting it in production, what are some of the loops to go
29:30
through and like Hirsch no. You said, I think I found at
29:33
that time, the two top Technologies for kerosene
29:35
kubernetes. Everybody was building a male
29:37
models and carers and everybody was trying to deploy models in
29:40
kubernetes. So I try to do a Best of Both
29:43
Worlds. The half of the book talks about
29:45
Karis a lot of computer vision kind of models and then and to
29:49
building some ml models, even some text analytics models and
29:52
then second half is more around. And the last couple of chapters
29:56
which are my favorite arm, or my thoughts on how this this
30:00
scenario, would change how the amount deployment will change.
30:03
And at that time, none of these mL of standards were published.
30:07
And I'm very happy to say even if I read the book now.
30:09
I mean, just yesterday I glanced over the last two chapters, most
30:12
of the things are very relevant, even with the state-of-the-art
30:15
ml, Ops things that are happening today.
30:17
I think a lot of the basic still remain the same, so, the book
30:20
released May 20, 19. And I have to say that a lot of
30:23
the, the fundamental concepts that are Are in the book,
30:26
talking about like, well, what is it about machine learning
30:29
that we should understand important Concepts, like
30:30
bias-variance, tradeoff sand, how to develop models.
30:33
How do you think about maturing your infrastructure?
30:35
Going from just a model to a container to then using
30:38
containerization? It has that incremental
30:41
component very well laid out that's embedded in the title,
30:44
right? Carriers to kubernetes what I'm
30:46
curious about is to say, okay it's it's November 20 22.
30:51
O'Reilly comes to you and says, hey, that's a Rog.
30:54
We love the first version of your book.
30:55
We want version to we wanted to go out in May 20, 23, What's the
31:01
title of that book? That has the same message of how
31:04
to go from problem 2 ml system and what kinds of new content
31:10
are you including that's a definition of, it's a great
31:13
question. But actually I have thought
31:16
about that. In fact, we I did have a
31:18
conversation but and I didn't want to do the Second edition,
31:22
because unfortunately, though the title was catchy, caressed
31:24
equivalent is, it was very cool based.
31:26
So I mean I didn't want to make it too.
31:28
So one of the changes I would do is not make it so tooling base
31:32
because at that time when I wrote it, everybody was talking
31:35
about Evan. Kubernetes was, I mean, they
31:36
were like, you people, were they talking about kubernetes as the
31:40
technology? So, I would rather focus on the
31:43
patterns and that's what actually even at persistent.
31:47
When we talk to customers, we try to look at the common
31:50
patterns. That are there patterns like
31:52
drift monitoring, explain ability into patterns around
31:55
packaging models of validation error analysis.
31:59
I mean, definitely the book died out.
32:02
Try to focus around ml patterns and Vector chip beat me to it.
32:05
Her her new book ml system patterns of MLS is a which which
32:09
I love I think she just released their book and that that's an
32:12
amazing book but that is the thing that I had in mind like if
32:16
I was going to write a new book out, focus more on the patterns
32:19
and just like, I explained with that ml, the Remote monitoring
32:22
example, I mean the patterns have not changed back from the
32:26
2000s, the you're still packaging intelligence.
32:29
Deploying intelligence monitoring for drift, checking
32:31
for bias. So All That Remains the Same.
32:33
So I would write something around the genetic patterns and
32:37
then you can use carers by to watch whatever acknowledge you
32:39
want to use, or some low CO2, like a show, pie carrot, or Sage
32:43
maker to build some of these things.
32:45
So even today, when at persistent, when customers come
32:47
to us for ML, Ops we we try to focus more around the space.
32:51
A turn say that. Okay, what is the problem you're
32:52
trying to solve? What is the best pattern for
32:54
you? And then, then that can be
32:56
realized? I mean, if you want it to be
32:58
done in Sage maker vertex AI, we can build it or we can customize
33:02
that particular tool to do it. But I would say to answer you
33:04
question, my next Focus would be more around writing about the
33:08
patterns and what are the best tools to realize this patterns
33:11
or or even use those tools didn't exist.
33:14
What I would visualize as the tools to realize some of these,
33:18
hey Laszlo here. If you are serious about it,
33:21
Philips, if it subscribe right now.
33:28
You are abstracting away from the tooling so that it is
33:32
something that ages well and you can see in your book.
33:36
As you were saying it's Karis and kubernetes you didn't really
33:40
mention ml. Ops you didn't really go into it
33:43
so much and and a lot of what is there?
33:45
The essence of what is there that you're talking about it?
33:48
Lives on Beyond Karis kubernetes it even if both of those End up
33:54
not being useful in 20 years, the ideas that you proposed in
33:58
the book are still useful. So now when it comes to those
34:01
patterns that you're talking about, what are the things that
34:04
you see over and over and you say, all right, this is the
34:07
pattern that you're going through.
34:08
So, in this case, these are things that you need to keep in
34:11
mind. Some examples would be something
34:13
that I would like to start with our data catalog.
34:16
So, you would want to not have your, I think everybody has
34:20
moved out from the CSV. But even if you have a data
34:23
warehouse, Like snowflake, you would want to have some
34:26
cataloging mechanism, maybe DVC or you have some version queries
34:31
or snowflake. Has this time travel feature
34:34
that you can go back into like for a period of time and pull
34:37
data. So some way to catalogue your
34:39
data and and publish it a lot of people catalog it but it is the
34:44
catalog just remains with the coder.
34:46
So you would want to have some sister even if though it looks
34:50
mundane but you would want to have a way to have your model
34:53
data. A diversions published and your
34:56
data data, scientists have easy access to them.
34:59
So that's one thing I think is very valuable and on the similar
35:02
lines, when the model is ready, a model registry.
35:06
Now today model richest is becoming very popular and of
35:08
course, like today features towards our girls are getting
35:11
popular. And now, there are like vector
35:12
databases which to similarities. And those are also getting
35:16
popular especially with like Facebook as is fast and Spotify
35:19
is annoying. Especially when you do
35:21
recommendations, they really help you but I think model
35:23
register Something I would, I don't I see less.
35:26
I would, I would really love to see, because modern registry
35:29
becomes something like a modern literature catalog.
35:32
Becomes like your basic a foundation to your ml system,
35:36
like, many customers, I see. And we at persistent, we we have
35:39
customers in different life cycle, right?
35:41
From somebody who are very new to ml two people building
35:44
hundred 100 models in production.
35:47
So we see a lot of them not paying enough attention to
35:51
having a very managed model registry.
35:54
In modern governance. So I think, because what that
35:57
does is now you have a very organized catalog for your
36:00
models and the next step would be to look at patterns like
36:03
deploying models of micro services, or what is getting
36:06
popular today, is ml as a service.
36:09
So another, another pattern we also want to look at so when we
36:12
look at MLS a service, many customers are looking at
36:15
monetizing those models because because typically you look at ml
36:19
model is okay, you have a pickle file and some developer will
36:23
come and pull it. It put it in an application and
36:25
consume it, but I, but debtor doesn't, especially these are
36:28
very valuable models in the industry that I come from
36:31
Banking and Healthcare. These models are the ones making
36:34
money for this customers. So they really want to monetize
36:38
them. I mean, these are the main major
36:40
assets. They are trained from their data
36:42
and they can make much more money by cataloging this better
36:46
and even making them available as a Marketplace.
36:48
So today, some of the things that are some customers actually
36:51
monetize in them. They have a token-based System
36:55
to actually invoke this models. Like, Amazon has like your
36:59
Amazon Marketplace, Marketplace offering where they can charge
37:02
customers based on usage of this models or even internally.
37:05
They would want to have a better control of how this models are
37:09
you so they can manage the lifecycle of this.
37:10
So again, at one long, answer to your question.
37:14
But I would say some things like a model calculate model,
37:17
registry data cataloging these pack, the mighty managing
37:20
multi-tenancy many times customers have been there.
37:23
If you are multiple tumors for the same model, how do you do
37:25
multi-tenancy? These are not some things that
37:27
our data scientist would think of.
37:29
But it is something that an ml engineer or a ml Ops person
37:33
should think of and should we should have.
37:35
I mean, these are, these are solved problems.
37:37
We should have like, a standard way to reduce this solutions.
37:40
And then, like the earlier ones, I talked about bias and
37:43
fairness, I mean, that is something you, you have
37:46
libraries, which can easily calculate that.
37:48
But the key thing is, you need to consider the many, many
37:50
customers. I've seen are like they're
37:52
hesitant to include them. They see bias as a like a
37:55
negative thing or they will get audited, but that is something
37:58
you need to start incorporating at least you need to have some
38:00
audit reports. Some model card of your model.
38:04
It's like we can't be biased if we don't audit it.
38:10
That's the man. That is so interesting.
38:13
What you are talking about with the motto?
38:15
Registries, or just like having that model Marketplace because I
38:18
know that was something that algorithm you was doing back in
38:22
the day and they They yeah they ended up getting bought and so I
38:27
don't know if there are other companies or tools that have
38:31
come and gone to Market with that same approach, but it is a
38:35
fascinating one to think about and to see how how others are
38:40
tackling that problem. I also really like this idea
38:43
around the multi-tenancy and managing multi-tenancy because
38:46
it is so difficult when it comes to security, right?
38:49
And like what you need to get past the devil, That cops team,
38:55
what you need to make sure everything checks out and if
38:58
you're a data scientist trying to do that, it's a lot harder.
39:02
Yeah, exactly. In fact, to interesting you
39:05
mention about are going to me. I spoke to Diego when we were
39:08
building this platform back before he went to dinner or W.
39:12
Oh yeah, yeah. So he actually went really
39:14
interesting discussion over email, but he was sharing some
39:18
of his experiences. And the good thing is he
39:20
actually made a lot of those papers public come in the way
39:24
behind the scenes, how they were doing algorithm here.
39:26
So, I think, yes, that that is one of the very popular.
39:29
I mean, that's a prime example of a very popular part of
39:31
pattern. Yeah, so cool.
39:34
They were ahead of their time. Yep.
39:36
So that the Rog, man, this has been excellent.
39:40
I cannot thank you enough. Super happy to have you on here
39:44
and learn from you. You have so much wisdom, you
39:47
have so much experience, and if anyone wants to get your book
39:52
yet, they can go to, oh, Riley, I think it is or Amazon.
39:56
Yeah, they'll have it there and they can pick it up.
40:00
The it has aged well, There's some books that do not age well,
40:04
because they talk about how to do things and the step-by-step
40:08
guide and then as soon as the new package comes out or the new
40:12
hot thing on the market comes out, then you recognize that all
40:16
that you've learned from that book is no longer available or
40:19
useful to you. This is not one of those books.
40:21
This is a book that you will learn a ton from even though it
40:25
was written three years ago over three years ago.
40:28
So this man it's incredible. You did a great job and I am so
40:35
thankful to be able to speak with you and talk with you about
40:38
all this cool stuff and thank us.
40:39
Thank you so much guys. Skyler I lead machine learning
40:50
and health rhythms if you want to stay on top of everything
40:54
happening in ml. Ops, subscribe to this podcast.
40:57
Now. Now, now, now, now now Now.