Latest / Agentic Conversations (formally mlops.community) / MLOps - Design Thinking to Build ML Infra for ML and LLM Use Cases // Amritha Arun Babu & Abhik Choudhury // #221

Transcript

MLOps - Design Thinking to Build ML Infra for ML and LLM Use Cases // Amritha Arun Babu & Abhik Choudhury // #221 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Amrita Arun Babu Maiser.
• 0:04
  I am currently mill platform leader at Flavio and I like my
• 0:10
  coffee mainly strong, like an espresso shots.
• 0:14
  That's what I would. Hi, my name is Opik Choudhury.
• 0:18
  I work as a Senior Managing Consultant in IBM and I love my
• 0:23
  coffee as French within Vanilla from Wawa.
• 0:26
  Wawa is big here in Pennsylvania.
• 0:31
  Hello and welcome back to another MILFS community podcast.
• 0:34
  I am your host Dimitrios and today we got to talking about
• 0:39
  the MLP's maturity levels within businesses going from zero to
• 0:45
  two and how you can tie that back to business value.
• 0:50
  We haven't had a conversation like this in a while.
• 0:52
  I really appreciate that. Both of these folks for coming
• 0:56
  on and talking to us and Aretha and Abbie gave us a bit of a 1-2
• 1:00
  punch. Abek was able to go deep into
• 1:03
  the technical side of things. I really enjoyed the tangent
• 1:07
  that we went on on how you can approach Milos from the
• 1:12
  different personas, whether you're coming from IT DeVos
• 1:15
  persona or a Data Engineering persona and the OR the data
• 1:19
  Scientist persona. What you need to know what
• 1:22
  specific traits and skills you need to acquire.
• 1:28
  If you would like to start using machine learning in your company
• 1:33
  or start thinking a little bit deeper about ML OPS from these
• 1:39
  different angles, O if you're coming at it from data science,
• 1:42
  OK, you need to start thinking about how can I really be using
• 1:48
  source control all the time. We have the joke that data
• 1:51
  scientists don't necessarily like to use Git, and I know it's
• 1:56
  a stereotype, but sometimes stereotypes are true, so that is
• 2:02
  all I'll say. Hopefully you are not that data
• 2:04
  scientist because you're listening to this and you
• 2:06
  recognise the value of using git.
• 2:08
  But I digress. The idea here is that Abik was
• 2:13
  able to give us this technical angle and Amritha gave us the
• 2:19
  view from a product manager perspective and trying to sync
• 2:24
  the needs of the management or leadership with the needs of
• 2:31
  someone who is an engineer or working on the platform.
• 2:35
  What kind of metrics do you need to look at to know that your MLT
• 2:39
  is successful? Let's dive into it before we do
• 2:42
  it. You know, it always helps if you
• 2:45
  leave a review or just go ahead and give us one of those stars
• 2:51
  on Spotify, drop in a comment if you're on YouTube.
• 2:54
  It means the world to me. I will see you on the other side
• 2:58
  and of course, don't tell anybody, but we'll be doing a
• 3:03
  MMORPG conference June 25th in San Francisco.
• 3:08
  It's super secret right now, but soon enough you will be hearing
• 3:12
  about it. All right, let's go.
• 3:22
  Who put their learning how to cross country ski?
• 3:26
  Was that you and Rita? I know.
• 3:31
  Is that you a big? I'm not sure, no, I didn't know
• 3:36
  as well. Where did I get that from?
• 3:39
  Cause we asked for fun facts about what you all are.
• 3:43
  Like I did. I'm sorry.
• 3:45
  Yes, I did. I wanted to learn.
• 3:47
  Cross country ski. Yes, that was me.
• 3:50
  Yeah. I guess you haven't gotten too
• 3:53
  far if you don't even remember that you put that in the three
• 3:56
  months ago. Because we did not have enough
• 3:59
  snow. And you know, and I I don't know
• 4:02
  if I mentioned I live in Boston, it's supposed to snow a lot.
• 4:06
  And this year was climate change.
• 4:08
  We hardly had one snowstorm. So like we didn't use anything
• 4:12
  any of our could. Yeah, you just get the cold, not
• 4:16
  the snow that. Comes along with it, yeah.
• 4:18
  Ohh, wow. Wow.
• 4:20
  So, all right, well, I'm excited to chat with both of you.
• 4:23
  I know that this has been something that we've wanted to
• 4:26
  get into for a while. We want to talk today all about
• 4:30
  tying the business objectives to the technical needs when it
• 4:35
  comes to MLRO's and AI initiatives.
• 4:38
  Abik, I know you can go deep on the technical side.
• 4:41
  Amritha, you're going to bring that PM perspective.
• 4:45
  Let's start with this. Abby, can you give us a bit of
• 4:48
  background about yourself and how you came into tech?
• 4:51
  Yeah, sure. So I'm a senior managing
• 4:54
  consultant in IBM with a over like 11 years of experience.
• 4:58
  I am right now at the intersection of data engineering
• 5:02
  and machine learning and of course using continuous delivery
• 5:07
  of and serving the model. That's my current role,
• 5:10
  overseeing a team of data engineers and data scientists
• 5:13
  bringing value for multiple clients for IBM.
• 5:18
  I started my IT journey basically as a Cobalt developer
• 5:21
  and soon I was transitioning into the data size, first as a
• 5:25
  data analyst, then into engineering and more recently as
• 5:28
  a machine learning engineer. So you got to see the full
• 5:33
  pipeline. You've had a lot of different
• 5:34
  experiences along the way. Yeah.
• 5:37
  And so, Aretha, give us a breakdown of what you're doing.
• 5:42
  I know you're currently at Clavio, you were at a WS, and I
• 5:46
  would love to hear just the TLDR of your journey.
• 5:49
  I'm currently leading the ML platform at Clearview and I've
• 5:56
  over about a decade of product management experience.
• 5:59
  I started off working on supply chain and ecommerce platforms,
• 6:04
  building integrations with sellers and as well as contract
• 6:08
  management and those kind of initiatives.
• 6:11
  Over the years, I had an opportunity to transition slowly
• 6:15
  as I built like BI instrumentation and data
• 6:17
  pipelines into machine learning. That was one of the ways I
• 6:21
  transitioned and currently last three years have been focusing
• 6:25
  on machine learning initiatives. Excellent.
• 6:28
  So there is something about that you said before we hit record
• 6:33
  and I want to get right into it, which is talking about how Milos
• 6:37
  is still very new in general. It was a buzzword.
• 6:41
  I think it got dethroned as the buzzword of the tech year when
• 6:47
  LM's came out that became the new buzzword and then LM OPS.
• 6:50
  So yeah, I think we're we're over the hump of the MLS or the
• 6:54
  LM OPS craze right now. But they're still is this
• 6:59
  feeling that when it comes to DevOps, that's a very mature
• 7:02
  field and actually in fact I'm at Cube Con right now.
• 7:06
  There's like 11-12 thousand people here.
• 7:09
  It's mayhem and you can see that this has had very this has had
• 7:15
  many years of development under its belt.
• 7:18
  ML OBS is not like that at all. We've got maybe one conference
• 7:24
  and that might be the conference that we're about to put on that
• 7:28
  has to do with this stuff. But can you break down what
• 7:32
  you're seeing and what what like made you come up with this idea
• 7:35
  of, hey, it's still new and we still need to figure some things
• 7:39
  out? Where are the loose ends in your
• 7:40
  mind? Yeah, sure.
• 7:43
  So as you rightly said a few years back, it was a buzzword
• 7:47
  DevOps. Everyone knew about DevOps, but
• 7:50
  not many people knew that. It's a rendition of DevOps into
• 7:54
  machine learning. That's what we call MLPS and
• 7:58
  people did know that there is something existing has existed
• 8:02
  as DevOps and Machine learning but not many people are or other
• 8:05
  organisations knew how to and get the best out of this thing.
• 8:10
  So now more and more organisation including the
• 8:13
  clients that I work with Dane and Dale, they have understood
• 8:16
  the value of value that Milos brings to the table as far as
• 8:20
  continuous integration and continuous delivery is concerned
• 8:23
  and they have started reaping the benefits.
• 8:25
  They have started understanding that the whole lifecycle process
• 8:29
  of machine learning I would say pipeline or the old machine
• 8:33
  learning lifecycle is long and to make it purposeful for day be
• 8:39
  business activities they need to shorten it and streamline it.
• 8:42
  And the best way to streamline is is to use Milos.
• 8:45
  So yeah, people have started realising the true value of it.
• 8:49
  And how do you see the lifecycle being?
• 8:53
  Has it? Because I I remember one thing
• 8:56
  that we often would look at were the different maturity levels
• 9:00
  that you could say, alright, here's a example diagram of a
• 9:04
  mature way of doing continuous training or continuous
• 9:08
  integration and continuous development in machine learning.
• 9:12
  Do you feel like that has now become more standardised?
• 9:16
  Because sometimes it's like, all right, cool.
• 9:18
  We have, we have this notion that the more automation, the
• 9:24
  more mature. But that isn't necessarily
• 9:27
  always better, especially when it comes to ML, which is I think
• 9:30
  a little bit where it starts to veer off the beaten path of
• 9:34
  DevOps, right? And with DevOps you would, if
• 9:37
  you can just not touch it ever, that's great.
• 9:40
  With ML, there's a lot of things that you should probably be
• 9:44
  looking at, and so can you break down the way that you see
• 9:50
  different maturity levels and Oregon life cycles.
• 9:54
  Yeah. So to your question, so a
• 9:56
  majority level can be broadly classified into 30 being the
• 10:00
  maturity level 0. It's that they just conceived
• 10:03
  it. They are in the process, the
• 10:05
  first stepping stone of it, but they haven't done much into the
• 10:08
  continuous integration and continuous delivery part of it.
• 10:11
  Part 2 would be the kind of automated, the continuous
• 10:18
  delivery part of it and not much into the continuous integration
• 10:23
  part of it and I'll breakdown these into further smaller
• 10:26
  pieces when I liberate. And in a nutshell the part three
• 10:31
  would be absolute Automation, continuous Integration,
• 10:34
  continuous delivery, everything done.
• 10:36
  But Level 3 is all the organisation tried to aspire to
• 10:42
  over years of experience, but not many have reached even till
• 10:46
  now. Even many mature IT companies
• 10:50
  with very mature IT infrastructure, they haven't
• 10:53
  been able to reach to Level 3. They mostly are somewhere
• 10:56
  between Level one, Level 0 and Level 1.
• 10:59
  And that's been my experience for multiple customers that I've
• 11:03
  worked with. So the first customer that I
• 11:06
  started working with back in 2018 when ML also still look
• 11:10
  kind of a buzzword, they started with 00 and now after like
• 11:15
  several years there are somewhere in between level one
• 11:18
  and level 2 and there are various constraints which
• 11:21
  prohibit them from going to full level 2.
• 11:26
  Yeah, it is not easy at all. And you mentioned you can break
• 11:30
  down a little bit more of what is in each one of these.
• 11:33
  Can you go into that? Yeah, sure.
• 11:35
  So the first thing is, so I'll break the various stages of the
• 11:42
  entire Milos thing. First is the analysis of the
• 11:45
  data. You get the data and then you
• 11:47
  orchestrate an experiment. So in orchestration of the
• 11:51
  experiment you come up with the machine learning model that you
• 11:55
  are trying to work on. So first is the data
• 11:58
  preparation, data model training, then model evaluation
• 12:04
  and then model validation. So this is a typically a machine
• 12:09
  learning developer or an AI developer would do in his test
• 12:14
  environment, and that's called the experimental experimentation
• 12:18
  and the development phase. Once he's done with that, he
• 12:21
  will package it into a source code and then he will put it
• 12:26
  into a code repository for versioning, not in level 0.
• 12:32
  You have orchestrated the experiment and you deploy it
• 12:37
  manually. There is not much of A scope of
• 12:41
  versioning. They are trying to version it
• 12:43
  but they are somewhere in between doing it manually and
• 12:46
  versioning and then deployment is also done manually.
• 12:51
  So this is a level 0. So in deployment they manually
• 12:57
  deploy it, while if you mature to the next level in level one,
• 13:04
  this whole portion is automated. So let's say the entire
• 13:07
  orchestration of the experiment, data re modelling, training,
• 13:11
  evaluation, everything, it's all standardised.
• 13:14
  So let's say I as a machine learning, machine learning
• 13:19
  developer, one to use as a sort of standardised template for my
• 13:24
  development. I want to ensure that the data
• 13:27
  engineering team is also on the same page.
• 13:29
  They are also using the same forward.
• 13:31
  There is a different amount of sync between them.
• 13:34
  So that's when we are trying to orchestrate the pipeline, we are
• 13:37
  all in the same level and when we are kind of deploying it in a
• 13:42
  package, it all helps in a better way.
• 13:46
  So in a C, ICD, I mean in DevOps, we build codes, I mean
• 13:52
  we deploy codes here, we deploy the entire pipeline.
• 13:56
  That's what we call about packaging.
• 13:59
  So that's the crucial difference between stage at all and stage
• 14:05
  1. Right.
• 14:07
  I'm afraid I would want to sell into any part of it, or if you
• 14:10
  have any questions now. And I agree with you.
• 14:14
  Like when you said the template, some of the things that I'm able
• 14:17
  to relate to is like providing you know like a Docker image or
• 14:21
  to say that here is the image that you're going to probably
• 14:25
  convert for your experimentation pipeline and this is the image
• 14:28
  that we are going to deploy. Where this helps with respect to
• 14:33
  tying to the business value, which I think we will deep dive
• 14:36
  eventually is to say that this reduces time for the developer
• 14:41
  significantly. This makes sure there is less
• 14:45
  back and forth, increases collaboration.
• 14:48
  So that's the key piece that we need to focus here.
• 14:50
  Actually, yeah. Amrita, at the risk of totally
• 14:53
  derailing what Avik was saying, I want to get back to like the
• 14:57
  the key differences between level one and level 2.
• 15:00
  But I think it would probably be a great point here if you can
• 15:04
  mention what some of the key metrics are that you're looking
• 15:07
  at as you're going through these different maturity stages
• 15:10
  because it can't just be tied to we have more automation, right?
• 15:15
  Or we have the ability to do XY. I've heard the DoorDash team
• 15:21
  talk about how one thing that they're looking at is velocity
• 15:25
  to deployment. And so how fast can you get
• 15:28
  something out is one key metric that they look at.
• 15:32
  What are some metrics that you've found really help paint
• 15:37
  the picture of if your team is getting better?
• 15:41
  So what a couple of things, agree with Birthday DoorDash
• 15:44
  team and what you mentioned because at least the North Star
• 15:47
  vision for an ML platform is to make sure that you as a platform
• 15:53
  can enable scientists to get their idea from ideation phase
• 15:57
  to production as short as time as possible.
• 16:02
  That's that's the North Star metric.
• 16:04
  Underneath that there are various ways to think about
• 16:06
  this, especially in the case of Abhik mentioning 02/1 stage.
• 16:10
  We can think of like it's not that we are automating
• 16:14
  everything, right? We are.
• 16:15
  Let's say our goal post is to automate only bring
• 16:19
  standardisation. We are not still focusing on
• 16:23
  continuous integration, right, Bring standardisation and make
• 16:28
  seamless deployment with that as the goal.
• 16:32
  Bringing standardisation would mean that.
• 16:34
  OK, how am I streamlining and reducing the time in data
• 16:37
  exploration? Straight how am I providing
• 16:40
  standardised libraries or sticks in the training phase?
• 16:45
  You know, how will that reduce the time for scientists?
• 16:49
  Or are we providing like any data quality insights so that
• 16:52
  scientists have an insight of, you know, which is the highest
• 16:55
  data set quality, how can they reduce time.
• 16:57
  So these are some of the ways that we should think about how
• 17:01
  to reduce time and how does this tie to our North Star metrics?
• 17:05
  Yeah. And to add to that, I want to
• 17:09
  cite another example that I have from my experience.
• 17:12
  So let's say one of the customers who are big into
• 17:17
  wholesale distribution, they wanted to do customer
• 17:21
  segmentation. So first they wanted to end
• 17:23
  their machine learning pipeline. They introduced this the entire
• 17:27
  machine learning pipeline from all the way from data
• 17:29
  acquisition to serving the model.
• 17:33
  When they were in Level 0, they were not much into a whole lot
• 17:38
  of automation. They saw that the entire cycle
• 17:40
  is like several months, Ohh of duration.
• 17:44
  Now one thing they figured out, I mean they were just cool with
• 17:47
  that. But then one fine quarter they
• 17:50
  realised that the model that we are serving, it's outdated
• 17:54
  because the behaviour has changed and also the data
• 17:59
  underneath has several factors. Several biases have come up and
• 18:03
  which need to be incorporated into the module.
• 18:05
  So they want to make sure that they reduce the cycle, they
• 18:08
  increase automation that one thing that would prompt them who
• 18:13
  are in adopt to a second level of automation.
• 18:18
  Yeah, that is a great point that you can't get that ground truth
• 18:23
  always though, right? Because if you're looking at
• 18:26
  like a fraud detection model or if you're looking at a lone
• 18:31
  scoring model, better yet and you give someone a loan and then
• 18:35
  they default on the loan two years later, that is very hard
• 18:39
  to get that ground truth back into the model and be having it
• 18:44
  continuously updating in that regard.
• 18:46
  So it's it's a little bit harder for some of these use cases.
• 18:49
  And as you're going through this, it almost feels like
• 18:54
  depending on your use case, you're going to be looking at
• 18:59
  these different levels in certain ways and you're going to
• 19:05
  be trying to track it pretty obviously there's going to be
• 19:08
  different metrics that are going to be more important to you and
• 19:11
  more valuable to you. But I don't want to dive down
• 19:16
  that rabbit hole yet because I'll be, I know you have from
• 19:19
  1:00 to 2. And so talk us through what the
• 19:22
  differences are from semi automated to basically fully
• 19:26
  automated I think. Yeah.
• 19:30
  So the fully automated, but look like where not only the
• 19:35
  integration but the serving also is fully automated.
• 19:39
  So I'll try to tie all those pieces together with an example.
• 19:43
  So we have, let's say a pipeline in the development phase, which
• 19:47
  includes the data preparation, model training, evaluation and
• 19:52
  validation everything into a single pipeline as a single
• 19:57
  package and as Amrita previously told.
• 20:00
  We have orchestration images using a docker or Kubernetes
• 20:05
  that help deploy. So when we have orchestration
• 20:09
  automated we are into the second level of maturity.
• 20:12
  I mean first level of maturity there and once we have the
• 20:16
  package deployed and we register it into a model registry.
• 20:21
  So there are two advantages to this.
• 20:25
  So first is it helps all the teams In Sync and the 2nd thing
• 20:31
  is it helps versioning of the model.
• 20:33
  So let's say the Today the model is working correctly, tomorrow
• 20:36
  it may not. I just the business requirement
• 20:38
  is that I have to go back to the first model, it's just going to
• 20:42
  a different version and deploying that.
• 20:44
  So it's that easy. It helps the flexibility of
• 20:48
  changing the business requirement needs.
• 20:50
  And then you train the model and you serve the model and once the
• 20:57
  prediction comes and it goes to the end users and this is the
• 21:02
  true level of maturity that comes in.
• 21:04
  The level second is that you can use continuous monitoring and so
• 21:08
  now the models are in autumn pilot mode and they are serving
• 21:12
  the model. Now there are two ways you can
• 21:16
  prompt it. So first is again like I am.
• 21:21
  I'm pretty sure that I want to update the model every 3-4 days
• 21:26
  with new hyperparameters, new updates of the models and
• 21:29
  everything before even the the performance of the model itself
• 21:34
  degrades. So I can set up an automation
• 21:36
  that I will change these parameters right in the pipeline
• 21:40
  itself so that new models are served continuously so that the
• 21:44
  business gets very accurate prediction.
• 21:47
  So that's one scenario and the second scenario is the moment
• 21:51
  the data is, the data is served in the in the model.
• 21:57
  And I see that there may be different triggers like the
• 22:02
  skewness of the data changes or various discrepancies come
• 22:07
  within the data. So I want to make these changes
• 22:09
  immediately to my model based on some triggers from the data.
• 22:13
  Are you able to incorporate those things as well?
• 22:17
  So these sort of continuous monitoring of the data and the
• 22:21
  model and various KPIs and metrics and then taking
• 22:25
  appropriate action as an iterative loop.
• 22:28
  So this is something, this is the ability that a complete
• 22:32
  automation model gives us and everybody aspires to do that.
• 22:37
  So this is something in a nutshell.
• 22:39
  It is something, it's like that kind of pie in the sky.
• 22:43
  Like, yeah, wouldn't it be great when we can get all of that
• 22:47
  going and it's just automatically retraining and
• 22:50
  automatically updating? It does feel like what you were
• 22:52
  just talking about is very much something that harps on if we're
• 22:57
  doing what some people would call traditional machine
• 23:01
  learning, classical machine learning, right.
• 23:03
  How does this change when we start adding LM's to the mix?
• 23:08
  Because some of the stuff that you're mentioning like are back
• 23:12
  or the OR to be able to just rollback your model that
• 23:19
  probably. Works.
• 23:21
  More or less the same when it comes to a fine-tuned LLM.
• 23:25
  If you grab like Mistral or Lama to off the shelf and you fine
• 23:29
  tune it, but then the new fine tune isn't quite as good as the
• 23:32
  last one, so you can just roll back.
• 23:34
  But I imagine there are some things that you need to be aware
• 23:39
  of when you're looking at the traditional or classical machine
• 23:42
  learning versus the new generative AI.
• 23:45
  Yeah, at least in my opinion, right?
• 23:48
  For the Elms, at least what I have seen in a couple of places
• 23:52
  where we built LMS. LMS require massive datasets.
• 23:56
  So again, deploying elements through a 0 maturity mill OPS is
• 24:04
  different from what it looks like or mature two.
• 24:06
  I haven't seen in mature 2. So I wouldn't be able to speak
• 24:10
  more onto that. But what I have at least seen in
• 24:13
  zero and one stage is that being able to have access to this
• 24:18
  massive data, whether internal data or external data.
• 24:21
  Having those data pipelines are set up.
• 24:24
  Having streamlining those access initiatives like like having a
• 24:29
  data catalogue that can basically provide you insights
• 24:32
  into what are the various sources that exist, whether
• 24:35
  internal an external. For your particular use case,
• 24:38
  how easy is it to get what is the lineage and what is the
• 24:44
  schema and all the good stuff. And being able to sample this
• 24:47
  particular data and being able to use in a continuous basis is
• 24:51
  what differentiates from level 0 to level one.
• 24:54
  In level 0 you would just be able to get one batch of data.
• 24:57
  In level one, I would expect my scientists to be able to get all
• 25:01
  these data in a continuous, refreshed manner.
• 25:04
  And to add to that, another thing that I have observed over
• 25:08
  the last few months, people trying to incorporate LLM's into
• 25:15
  an entire mill off scenario is that they are wary about the GPU
• 25:21
  costs. Olms are.
• 25:23
  So for instance I'm talking about the GPT 4 that our client
• 25:28
  has been using oscillate and they have been using it for
• 25:31
  various experts experimentations but not as a full scale ML OPS
• 25:36
  deployment. And the GPU costs can snowball
• 25:41
  at times. So whenever the usage is pretty
• 25:43
  high, so they are really scary about.
• 25:45
  I don't want to write be very cautious about going to complete
• 25:50
  automation. So that's another thing which
• 25:53
  I've experienced. It's fascinating to me because
• 25:55
  it does feel like for a lot of the ways that Mill is out there
• 26:04
  in companies and making companies money or saving
• 26:07
  companies money, it really depends on what use case you're
• 26:13
  using. And a lot of the times, the most
• 26:16
  mature ways that companies are using machine learning is
• 26:21
  traditional mill and it's very proven because it is a little
• 26:25
  bit older, it's a little bit more mature.
• 26:27
  You can prove out that hey, these this is what we're doing,
• 26:31
  this is how much money we're saving.
• 26:33
  We have these metrics. We know that it's working like
• 26:35
  this with LM's. It's a little bit more
• 26:37
  exploratory. And we're still trying to figure
• 26:40
  out like what does a mature system look like in this regard
• 26:45
  and what are the use cases that we can incorporate it either
• 26:49
  into our product or into our company.
• 26:52
  Because maybe it's just like a classic like chatbot rag that
• 26:57
  you have different people in the company being able to quickly
• 27:00
  get up to speed on projects, whatever it may be.
• 27:03
  And so the requirements there are drastically different than
• 27:08
  if you're trying to figure out, again, going back to the fraud
• 27:11
  detection model or the loan scoring model where you're not
• 27:14
  going to use an LLM for that ever.
• 27:16
  And you do need this very mature way of updating the model and
• 27:22
  continuously retraining it and being able to rollback as easily
• 27:26
  as possible. And you want to be super mature
• 27:28
  there. And with the LM, maybe it's just
• 27:31
  like, hey, we make a few GPT calls, we've got some prompts we
• 27:34
  like, and that's kind of good enough for where we're at right
• 27:39
  now. We see it helping and we're
• 27:41
  trying to operationalize it, but we don't necessarily need to do
• 27:44
  much more on it. So I think The Dirty little
• 27:48
  secret is, and I would love for people that are listening also
• 27:51
  to hit me up and tell me potentially The Dirty little
• 27:55
  secret that I've been seeing around, and I imagine you all
• 27:58
  have seen it too, is that a lot of the money that is being made
• 28:03
  with machine learning and AI right now is still like
• 28:06
  traditional AI or traditional mill, like these classical
• 28:10
  models that people call them. That's where a tonne of cash is
• 28:14
  either being saved or being made.
• 28:16
  And LM's have taken the hype. And so everybody's talking about
• 28:21
  AI and you've got like just the random people on the street
• 28:26
  saying how it's going to take over the world.
• 28:27
  But if you look at it and you look at like statistical machine
• 28:31
  learning is making the cache still.
• 28:33
  And that's The Dirty little secret that I I think you all
• 28:36
  have seen too. Changing gears though, I do want
• 28:41
  to talk a little bit about how when you are putting together
• 28:45
  these systems and you're thinking about the metrics and
• 28:50
  Aretha like design thinking is very popular in the product
• 28:54
  management space. How can we bring that type of
• 29:02
  ideology into the development of our pipelines, our maturity, our
• 29:09
  customizable and modular mill O stacks?
• 29:13
  Yeah, uh, one of the couple of ways that I have seen this work
• 29:19
  can be successful is always working backwards from customer
• 29:24
  for mill OPS or any any product that you take in working
• 29:29
  identifying who is your primary customer, who is your secondary
• 29:32
  customer. Sort of like the actors in the
• 29:34
  ecosystem to identify them and work backwards from them,
• 29:38
  understand their use cases, understand their pain points.
• 29:41
  What does this mean? Let me take an example like when
• 29:45
  building an MLB platform in the last over the last three years,
• 29:49
  there are a couple of my my primary customer is data
• 29:53
  scientist, There is a secondary customer who is an ML engineer.
• 29:57
  There's a third customer who is a BI engineer or any of these
• 30:01
  analytics roles, right. So these are my 3 customers.
• 30:05
  And then once I identify this, usually I work back from
• 30:10
  understanding their pain points. What does an scientist going to
• 30:13
  do like? To your point, OK, they're
• 30:14
  building a fraud detection model.
• 30:17
  Great. They're also building a
• 30:19
  recommendation in an app with recommendation Nuggets to the
• 30:25
  customers on their UI. Or, you know, as you converse
• 30:28
  with a particular web service that might be, they're giving
• 30:32
  you recommendations. Alright, real quick, let's talk
• 30:34
  for a minute about our sponsors of this episode, Making It All
• 30:38
  Happen. Lattice flow AI.
• 30:40
  Are you grappling with stagnant model performance?
• 30:43
  Gartner reveals a staggering statistic that 85% of models
• 30:47
  never make it into production. Why?
• 30:49
  Well, reasons can include poor data quality, labelling issues,
• 30:53
  overfitting, underfitting and more.
• 30:56
  But the real challenge lies in uncovering blind spots that lurk
• 30:59
  around until models hit production.
• 31:02
  Even with an impressive aggregate performance of 90%,
• 31:05
  models can plateau. Sadly, many companies optimise
• 31:10
  for prioritising model performance for perfect
• 31:14
  scenarios while leaving safety as an afterthought.
• 31:17
  Introducing Lattice Flow AI The pioneer in delivering robust and
• 31:22
  reliable AI models at scale. They are here to help you
• 31:26
  mitigate these risks head on during the AI development stage,
• 31:30
  preventing any unwanted surprises in the real world.
• 31:33
  Their platform empowers your data scientists and ML engineers
• 31:36
  to systematically pinpoint and rectify data and model errors,
• 31:41
  enhancing predictive performance at scale.
• 31:45
  With Lattice Flow AI, you can accelerate time to production
• 31:47
  with a reliable and trustworthy models that scale.
• 31:51
  Don't let your model stall. Visit Lattice Flow dot AI and
• 31:56
  book a call with the folks over there right now.
• 31:59
  Let them know you heard about it from the MO Obs Community
• 32:02
  Podcast. Let's get back into the show.
• 32:04
  So in order to build these what what, what are the pain points
• 32:09
  that they encounter across the entire lifecycle that Abbie just
• 32:12
  walked you through, right. So in the data exploration in
• 32:15
  the training and you know fine tuning or evaluating and then
• 32:21
  releasing the model and how do they and what are the pain
• 32:24
  points to the encounter And how can I tie this to saying that,
• 32:28
  OK, let's say you encounter data exploration plane point or let's
• 32:31
  say you encounter, it's very hard for you to retrain the
• 32:36
  model. Then with all these aspects you
• 32:41
  are always looking into like hey, how can I make it easy for
• 32:44
  you? Like what does it mean for you
• 32:46
  to retry? Like let's say you today you're
• 32:47
  not able to retrain, what is, how is that impacting our
• 32:52
  ability to be AI leader? Let's say because of because
• 32:55
  you're not able to retrain, we are giving slightly different
• 33:00
  recommendations than what is what can be very accurate that
• 33:03
  leads to your models not being accurate, not models not being
• 33:07
  precise. So those are some of the ways
• 33:10
  that we need to apply design thinking.
• 33:12
  To just summarise, work pack for from customers, identify all the
• 33:17
  touch points across various phases of your life cycle and
• 33:21
  map it to your business objective to say what is the
• 33:24
  impact if this issue persists. And when you map it back to
• 33:32
  these metrics and the ways to be able to say what the impact is,
• 33:37
  do you have clear ways of being able to take it?
• 33:42
  Like OK, this is not in Python And the data scientist needs it
• 33:46
  to be in Python. So them having to learn YAML or
• 33:51
  something is going to cost us X amount of money.
• 33:54
  Like I can only imagine that the best way of putting it into a
• 34:01
  language that everyone in the company is going to understand
• 34:03
  is always going to be how much money is this making us costing
• 34:06
  us that type of thing. So how can you translate, not
• 34:11
  knowing Python or not having it be in Python into like actual
• 34:14
  dollar amounts? Yeah, that's a very good
• 34:17
  question. One of the situations that I
• 34:19
  face this is scientists often are very good at working as a
• 34:24
  community. They're very good comfortable
• 34:26
  with Python, right. There are scenarios where when
• 34:29
  they are working with like let's say petabyte scale of data or
• 34:31
  larger scale of data and distributed compute, they have
• 34:34
  to work on Spark. But they didn't and manage
• 34:37
  services right. They do not want to work on
• 34:39
  Spark because it has steep learning curve and it comes with
• 34:41
  its own thing. So and then that takes away
• 34:45
  their attention span from innovating like focusing on
• 34:48
  building the model versus they need to learn a new technology
• 34:51
  right. So here the way you can
• 34:53
  translate this pain point or time lost into saying that you
• 34:59
  have X skills, that the data scientists are developed and
• 35:03
  they are comfortable in you and they need to build wise skill
• 35:06
  and it takes. Let's say if you have 10 data
• 35:08
  scientists that need to build wise skill and it takes 20 hours
• 35:11
  for them or you know X hours over the weekend, how can you
• 35:15
  translate it? Let's see if your organisation
• 35:17
  has like 3000 scientists. If this is how the managed
• 35:20
  service for training and model build that you're providing,
• 35:24
  then that only just makes it even bigger a problem.
• 35:30
  So yeah. So translating it into hours and
• 35:33
  then being able to extrapolate out like, well, we're paying
• 35:37
  these people X amount a year and so every hour is worth this
• 35:41
  much. That means that if they need to
• 35:43
  spend 20 hours learning at least the basics of Spark, and this is
• 35:47
  not, they're not going to be professionals with Spark in 20
• 35:51
  hours, it's gonna take them a lot longer than 20 hours.
• 35:54
  And so this is just to get to the basic level.
• 35:57
  And then there's those unknowns that later on you have a data
• 36:03
  scientist who thinks they understand Spark and they end up
• 36:07
  doing something that costs the company a lot of money.
• 36:10
  And so that is a little bit of a another piece of the puzzle that
• 36:14
  I'm sure it's like a risk that you could map out.
• 36:18
  Yes, yeah, changing priorities, organisation, priorities, there
• 36:21
  are various factors that adds to it.
• 36:24
  Because all said and done, if all variables are constant and
• 36:27
  you only have to learn spark, that's one way to look at the
• 36:30
  problem. But real life and organisations
• 36:33
  are not like that A. Little more messy, yeah.
• 36:35
  Yeah, I could see that. So one thing that I wanted to
• 36:39
  get into also was around the different ways that people come
• 36:45
  in to Milos and Abik. I know that we had mentioned
• 36:49
  this before. We hit record, which is very
• 36:51
  much about the user journey of MLM's requires a certain amount
• 37:00
  of understanding. We broke down the maturity
• 37:02
  levels. We broke down the pipelines or
• 37:04
  the life cycles more, Better said.
• 37:07
  And what you have is that a lot of people come into the field
• 37:14
  out of necessity like a data scientist who now needs to
• 37:17
  deploy their model and they need or they need to start thinking
• 37:21
  about how to move up the ladder and the maturity field.
• 37:26
  Or you have someone from the DeVos field that is saying, OK,
• 37:30
  now I need to make sure that our mill platform is reliable.
• 37:33
  How much mill do I need to know versus the base of DevOps that I
• 37:38
  already have? So it's almost like, can you
• 37:42
  maybe give us a a summary of if you're coming from these
• 37:46
  different areas, what are some things that you might want to
• 37:50
  focus on? And I almost realise now after
• 37:55
  talking to you too, is that it's not just I'm coming from DevOps
• 38:00
  and I want to learn Milos because that is very broad and
• 38:03
  so that's probably not going to be that helpful.
• 38:06
  It's like I'm coming from DevOps and I want to learn recommender
• 38:10
  systems and I want to learn loan scoring systems or I want to I
• 38:15
  want to help my team who is working on prediction, churn,
• 38:20
  prediction models. And so there's the use cases
• 38:24
  that you have to look at. Or I'm coming from coming from
• 38:28
  data science and I now am going to be working with the ML
• 38:34
  platform team. What do I need to know?
• 38:36
  How much DevOps should I be thinking about All of that or
• 38:39
  I'm a data engineer? I understand pipelines really
• 38:42
  well, but now I need to be an ML engineer.
• 38:45
  I need to work with the mill engineers and so I know that's
• 38:48
  like there's a big surface area there, but I get the feeling you
• 38:53
  have some insights into the journeys of each one of these
• 38:56
  personas. Yeah, sure.
• 38:58
  So broadly there would be you rightly covered.
• 39:01
  Broadly there would be 3 personas.
• 39:03
  One with the data engineering flavour, one with the data
• 39:07
  science or the machine learning frameworks, and one is the
• 39:10
  deployment or the CICI or the DevOps labour.
• 39:14
  People with different backgrounds join this entire
• 39:18
  effort all the way from data acquisition all the way to model
• 39:22
  serving and the entire iteration in between and depends on the
• 39:25
  organisation. How we want these three silos I
• 39:31
  would say to be demarcated from each other or be rather
• 39:35
  intermingled with each other. So again depends on the
• 39:38
  organisation and the culture. But broadly I have seen that the
• 39:42
  person who is coming from data engineering background, he is
• 39:46
  mainly in the beginning of it where he is good in acquisition
• 39:51
  of the data from multiple sources.
• 39:53
  The entire transformation part of it, the where he uses his
• 39:57
  engineering skills for the exploratory data analysis part
• 40:01
  of it and the transformation and he needs to have certain degree.
• 40:06
  So what he what he needs to pick up in this new ecosystem is some
• 40:11
  degree of understanding of various models, not too deep,
• 40:14
  but some degree of analysis, understanding of in what format
• 40:17
  the data scientists would need it.
• 40:21
  So that understanding is required.
• 40:23
  As far as data scientists concerned, his most involvement
• 40:27
  would be during the. Experimentation in the
• 40:29
  development phase, where once he has deployed the model, he also
• 40:35
  needs to make sure that he has some amount of visibility into
• 40:39
  the deployment part of it from the experimentation phase.
• 40:44
  So for instance, what are the parameters, how I need to
• 40:47
  parameterize, how do I need to work with the code repository,
• 40:51
  how I need to work with the versioning?
• 40:53
  And we talked about YAML files. You need to work with YAML
• 40:56
  files, not just the actual the Jupiter notebook.
• 41:00
  So these are the new things that he needs to pick up as far as
• 41:06
  someone from DevOps of background is concerned when he
• 41:12
  is into the ML OPS part, he again one thing is that it is
• 41:17
  very iterative process particularly for an MLM's
• 41:21
  professional as a machine learning engineering
• 41:24
  professional. So somebody who is not in too
• 41:28
  much into data science and knows only DevOps concepts, he needs
• 41:33
  to pick up some basics of hyperparameter tunings.
• 41:36
  Like what are the things certain things I need to take care of?
• 41:40
  So let's say these are the things that change in the change
• 41:43
  in the data that I see. What are the things that I need
• 41:46
  to change that will manipulate my model to work in the way I
• 41:51
  want? So he needs to be aware of the
• 41:53
  hyper parameter tunings and these concepts as a DevOps
• 41:57
  professional. So these three silos, they have
• 42:02
  their work cut out but still need to know something extra to
• 42:05
  work with each other. Yeah, that's great.
• 42:08
  And Amrita, from the product manager perspective, what are
• 42:13
  some things that someone coming from a more generalised product
• 42:17
  management background and then jumping into the more technical
• 42:23
  side of mill and mill products or just Milos in general or mill
• 42:28
  latform even, what are things that that person would need to
• 42:32
  know? Yeah, I've been thinking about
• 42:35
  this as I keep getting asked this question.
• 42:38
  So if I could give you like what are the things that where I had
• 42:42
  steep learning curves where what helped me was I had steep
• 42:48
  learning curves in as to understand the data science
• 42:51
  concepts because I came from a world of you know building all
• 42:57
  the UI pieces and designing the UI.
• 43:00
  So with respect when I transitioned to this field
• 43:04
  understanding the data science concepts, what are those
• 43:07
  terminologies, what do they mean in different areas?
• 43:10
  What does that translate into like let's say if some when
• 43:13
  someone tells you that they are building a continuously
• 43:16
  refreshed pipeline for let's say real time data versus a batch,
• 43:19
  what does that mean? What does that translate in
• 43:21
  terms of code like? I come from an engineering
• 43:23
  background even for me it was like how does this look in the
• 43:26
  code like, is it different? Whereas the difference coming
• 43:29
  from you know, so those sort of things were something that it
• 43:34
  took. That is where I took a lot of
• 43:37
  lot of time and then there was another piece of it And when I
• 43:41
  worked on building models like I worked on building PII models.
• 43:44
  So there understanding what kind of model architecture does come
• 43:49
  in when someone tells that they are.
• 43:51
  Let's say you're not training a model with a public data set.
• 43:56
  Oh, we don't need ground truthing dataset.
• 43:58
  Why? Like what?
• 43:59
  Why does this model does not need a ground truthing data?
• 44:02
  Understanding those different aspects helped me build better
• 44:07
  relationships with scientists and ask better questions to
• 44:10
  scientists. Yeah, yeah.
• 44:12
  So trying to figure out what those constraints are and really
• 44:16
  why there are these constraints or there aren't these
• 44:19
  constraints. I don't look at it as
• 44:22
  constraints, rather I look at it as what you have been focusing
• 44:25
  on previously drastically different from what you're
• 44:28
  focusing now. At least this was the situation
• 44:30
  back like a couple of years ago when I transitioned into it.
• 44:34
  Today everyone's talking about AI today, everyone's talking
• 44:37
  about Emil. This is a trend riding the wave
• 44:41
  in the trend. You know, you're quite aware of
• 44:44
  these concepts, you're quite aware of these terminologies and
• 44:47
  the buzzwords right, which you can self teach yourself.
• 44:52
  But again, these are the concepts that you need to self
• 44:55
  teach yourself if you want to wanted to learn.
• 44:57
  That's how I would put it, yeah. Yeah.
• 44:59
  And another piece to this that I think is fascinating too, is to
• 45:03
  be able to distinguish what are the unique factors of data
• 45:09
  pipelines that you need to be thinking about, and what happens
• 45:13
  with data that doesn't necessarily happen with code.
• 45:16
  And then what are the unique factors of coding and the coding
• 45:22
  pipelines or like the continuous development, CI, CD pipelines
• 45:28
  basically is what I'm trying to say in too many words.
• 45:31
  But knowing those two and how they look and what is different
• 45:35
  and what is different when it comes to data versus code, that
• 45:40
  is quite useful. Yeah.
• 45:43
  So the other piece of this, which I think is worth us
• 45:48
  checking out and maybe talking about for a moment is around
• 45:53
  someone that is coming from a front end or full stack
• 46:00
  development background. And now they've caught the bug
• 46:05
  because they are like, ohh, you know what?
• 46:08
  There's this new term that is now very popular called the AI
• 46:11
  Engineer. I can prompt.
• 46:14
  I've been able to stand up a website.
• 46:16
  I can see that people enjoy what I have created.
• 46:21
  Now I need to make my product a little bit more bulletproof, and
• 46:26
  so I am rising as this AI engineer.
• 46:30
  I need to learn about prompting prompt templates, fine tuning
• 46:36
  rags, and the ways that rags can fall over the different tools
• 46:40
  that I'm going to be using in the LLM ecosystem.
• 46:44
  But what else should that person be thinking about when they were
• 46:48
  to start using AI for their products and really start going
• 46:54
  from that? Not from zero to 1, because I
• 46:57
  think that's where you really can see the new alms shine, but
• 47:02
  more from that one to N. Is your question like let's say
• 47:06
  if you have a product or if you have a.
• 47:09
  Let's say you're an ecommerce company, you're selling products
• 47:13
  and you are easier question like as an engineer or as a product
• 47:18
  itself, how do you scale from using to foundational features?
• 47:23
  You know, using L&M to foundational features to NOR,
• 47:27
  yeah. What are you thinking
• 47:28
  specifically? I was thinking more about from
• 47:30
  the engineers perspective, but I do also like that idea because I
• 47:34
  have. We have had a few people on here
• 47:36
  come on and they said look, we were building traditional
• 47:38
  models, we were doing classical NLP and then we were able to
• 47:45
  just hit open AIS API and all that work we did for the last
• 47:50
  six months or year, it all went out the window in one API call
• 47:55
  because it was way better at some sentiment analysis or some
• 48:00
  keyword. It was just infinitely better
• 48:02
  than anything that we had made over time.
• 48:06
  But now we're in that phase where we're like, OK, cool, we
• 48:09
  are using the API and maybe that's what we want to do.
• 48:12
  We have the MVP or we have something that's that's actually
• 48:17
  working, and then scaling it from that one to end.
• 48:21
  But let's start with the first part and then we can go maybe a
• 48:25
  little bit down that rabbit hole of when people are using it and
• 48:30
  scaling their product features with LLM O.
• 48:33
  As an engineer, what do I need to know about and how do I need
• 48:38
  to get into the space of like being more of an AI engineer and
• 48:43
  less of a full stack dev. So as far as.
• 48:47
  So far getting into an A, I engineer, so some of the things
• 48:51
  that you need to be wary about us and I have seen that these
• 48:55
  are the two things that fall apart very easily.
• 48:58
  One is the scalability. So you need to be really vary
• 49:04
  about the scalability because data multiplies and particularly
• 49:07
  if there is a lot of seasonality and trend it can easily
• 49:11
  overshoot what you are thinking and your whole pipelines are
• 49:14
  capable of handling. So from a data engineering, from
• 49:20
  engineering perspective, you need to have those things sorted
• 49:24
  first even before you start optimising your pipelines.
• 49:28
  So that would be the first piece is how to scale your data.
• 49:31
  So if you're in a cloud environment, you have to make
• 49:34
  sure your load balances and all those things are in place.
• 49:37
  You are able to hold those data influence upfront and correctly.
• 49:42
  Second is in your model whenever you are scaling it, you are able
• 49:48
  to understand that you have considered all the biases
• 49:51
  underneath and if you have the outliers that may come into
• 49:56
  picture how you're going to deal with that and from an mill
• 50:00
  engineer perspective how you are going to serve the model.
• 50:05
  Serving the model is a very important aspect because in a
• 50:08
  development environment you will not if you're serving and if the
• 50:12
  data increases and your performance changes, you need to
• 50:16
  be very careful about if the serving meets the criteria of
• 50:20
  the business. So these three are the aspects
• 50:23
  then UTB vary about and foremost.
• 50:27
  Fourth and foremost is the compliance aspect of IT.
• 50:30
  When in the development perspective, we quite often
• 50:33
  overlook the compliance and the privacy part of it with our own
• 50:39
  model or we do make sure that we have various compliance and
• 50:44
  cheques that we are not using these fields for these sort of
• 50:47
  predictions. However, if we are using a third
• 50:51
  party model or maybe an ALM just with the API call, we not maybe
• 50:57
  we are not very sure that how they are handling the compliance
• 51:00
  and if they are in accordance with our company policy.
• 51:03
  So we need to do the due diligence of compliance with
• 51:07
  even the third party models. Oh my God, I didn't even think
• 51:10
  about. That that is such a great point.
• 51:13
  And a lot of times it's not so easy to figure that out.
• 51:17
  It's maybe it's hidden behind the terms of service that's 200
• 51:22
  pages long or it just isn't around and you can Yes like good
• 51:26
  luck trying to figure that out and if it matches U with your
• 51:29
  companies your yeah that's yeah that's a fascinating 1 that
• 51:35
  there's like a whole. Sort of a problem space that
• 51:38
  exists out there to determine this compliance piece of it,
• 51:43
  translating into what is the particular region, country,
• 51:47
  their government policies, their compliance, how are they
• 51:50
  catching up? How are these companies adapting
• 51:53
  and how does it all translate to you as a final adopter of
• 51:57
  companies? You know, sort of like a
• 51:59
  blockchain, like how does it all?
• 52:01
  Has we figured it out? Because everyone's at a
• 52:04
  different pace here, right? Yeah, a perfect.
• 52:07
  Example of that is something that I I refer to this story
• 52:12
  quite often in the Milos community and this was before
• 52:15
  the Milos community started. I was on a call with one of the
• 52:21
  now Milfs community members, Jeremy, and he was saying how he
• 52:25
  was working at a health tech startup and he had data in
• 52:30
  Canada and that data could not leave Canada.
• 52:34
  There was no way that data was going anywhere outside of Canada
• 52:38
  because the Canadian laws do not allow that.
• 52:41
  So even though the company was global and they had a lot of
• 52:44
  data scientists in the UK or in the US or in India, Australia,
• 52:50
  you name it, nobody outside of Canada could touch that or add
• 52:54
  that to their models. And you think about such a
• 52:58
  simple example of I want to build a rag chat bot that
• 53:02
  anybody can query and know about the data and know what's going
• 53:06
  on with our data. How can you ensure that that
• 53:10
  chat bot is not going to touch the Canadian data?
• 53:14
  Unless you're inside of Canada, right?
• 53:16
  Or maybe it's you have to build 2 separate chat bots or you have
• 53:19
  to do that. But unless you're thinking about
• 53:22
  that problem, you probably wouldn't even add it.
• 53:24
  You would just say like, ohh cool, I've got my rag, we're
• 53:26
  good. Like I just added a tonne of
• 53:28
  value to the company. Where's my raise type thing?
• 53:31
  And then you realise, no, I may have broken some laws.
• 53:35
  Let's hope nobody finds out about that, yeah?
• 53:38
  That that's a very good point. Because as as long as you are in
• 53:43
  the experimentation phase, you as a lone wolf, lone data
• 53:47
  scientist, you can do whatever you want.
• 53:49
  But as soon as you put it into MLPS, you have to come up with
• 53:53
  your Dick chief data architect or someone who has the bigger
• 53:57
  picture. And you have to make sure that
• 53:59
  in your entire ML OPS landscape you address these things even
• 54:04
  before going to level 0, let's say.
• 54:06
  So these are the things that need to be taken care of some of
• 54:10
  the ways that. I think just to add to that is
• 54:12
  some of the ways you can operationalize this and make
• 54:15
  sure even whether you're a small company or a big mid size or a
• 54:20
  big company is having these security reviews to make sure
• 54:23
  that how compliant is your product given now everything is
• 54:29
  quite global, it's no more just us and plus cons everyone, every
• 54:33
  country and their laws are constantly changing.
• 54:36
  So having your legal team and having your security team
• 54:39
  aligned and now you know overseeing this particular
• 54:42
  releases is quite helpful. Yeah, your legal team.
• 54:45
  Needs a rag on the different compliance laws that are
• 54:50
  changing constantly. So you need like 50 different
• 54:53
  rags in the company to help 50 different teams stay up to date
• 54:57
  with all of that. Ohh, so this is great.
• 55:01
  Well folks, I appreciate you coming on here.
• 55:04
  I know that we are hitting up on time.
• 55:07
  Is there anything else you want to mention before we jump off?
• 55:10
  Is there any questions that I did not get to ask that you
• 55:13
  would have loved to have dove into 1?
• 55:17
  Particular thing that I can think of is monitoring piece and
• 55:22
  evaluation. How does that change when you
• 55:26
  are in zero stage one and two? That's something like at least
• 55:31
  from the product point of view monitoring model.
• 55:34
  Monitoring can be quite basic when you are in 0 to 1 where you
• 55:39
  do not have a pipelines for getting you know ground truth
• 55:43
  data. So you cannot compute a lot of
• 55:45
  metrics, but whereas you know and you'd also don't have two
• 55:49
  links to look into like consistently monitor a
• 55:53
  particular model, but that's where you get very scrappy.
• 55:57
  You say that, OK, I'm going to take a sample size of the model
• 56:01
  outputs and ground truth myself and then I'm going to measure
• 56:06
  the accuracy and understand the model performance at one.
• 56:10
  How does it change? I would like to hear from ABI
• 56:13
  what has he seen from 1:00 to 2:00 and how does model
• 56:16
  monitoring change? Yeah, yeah, as far as.
• 56:20
  My experience is concerned. So for level 0, the monitoring
• 56:23
  is like pretty rudimentary because the life cycle is long.
• 56:26
  So you just like see you analyse the data once in a while or the
• 56:31
  business users, they tell you that OK, it's not performing as
• 56:34
  per requirement. Then you go ahead and see that
• 56:38
  OK, maybe the data is not meeting my expectation or there
• 56:41
  is bad data or the model, the purpose of the model has
• 56:44
  changed, the business requirement has changed.
• 56:46
  So the iteration is like a bigger iteration while in in
• 56:50
  two, I mean the totally automated.
• 56:53
  So you need to be very, very serious about your monitoring.
• 56:56
  You have to not only have to do the quality of the data, but
• 57:01
  various KPIs that come out of your model.
• 57:03
  You have to be on top of that and you will have to eventually
• 57:07
  implement A dedicated modelling tool and which are pretty common
• 57:10
  across multiple cloud platforms. You have to deal with them,
• 57:15
  leverage their properties to ensure that your APIs are up to
• 57:20
  mark, and if any of them are changed, you work with your
• 57:23
  model. So that's how the difference is.
• 57:26
  Yeah. You know something that's
• 57:28
  fascinating. To me is how when it comes to
• 57:32
  classical or traditional mill, we have almost like this golden
• 57:37
  metric that we can look at, which is accuracy and then when
• 57:42
  you switch over to BLMS, accuracy falls apart.
• 57:46
  There is no real kind of depending on the use case.
• 57:49
  You can try and if you can then that's great, like do it.
• 57:53
  But how are you going to measure the accuracy of a summarization?
• 57:58
  It's more that's why there's so many evaluation tools that are
• 58:02
  coming out right now, right and trying to evaluate how well it's
• 58:06
  doing. But that doesn't really bring
• 58:11
  into account what you're talking about, how at the end of the
• 58:14
  day, it's really what the business wants.
• 58:16
  And if the business needs change or if things start going into a
• 58:20
  different direction, that's what you want to know.
• 58:23
  And whether that is being able to evaluate the output, or
• 58:27
  letting the users tell you that this isn't actually useful with
• 58:31
  a thumbs up, thumbs down, or whatever.
• 58:32
  It may be in the evaluation. Or seeing that it's not useful
• 58:37
  because people aren't taking the predictions, or they just aren't
• 58:41
  using it and your accuracy or your accuracy score just tanks.
• 58:46
  All of these are different metrics that you need to be
• 58:48
  thinking about when you're looking through the lens of.
• 58:52
  Like how valuable to the company is my AI effort?
• 59:00
  Yep. Yep.
• 59:01
  Agree. 100% agree. Yeah.
• 59:03
  Yes. Ah.
• 59:05
  It all comes back to, yeah, how can we tie this to a number that
• 59:09
  the company cares about? And how can we make sure that we
• 59:12
  can show that we're doing stuff? Because I will say this again, I
• 59:19
  haven't, I've heard it so many times on different calls of
• 59:23
  people saying like the boss thinks that the data science
• 59:28
  team is just a cost centre. They're not a profit centre at
• 59:32
  all. It's like, what are we paying
• 59:33
  this guy for or this woman for? What is going on here?
• 59:37
  Because I'm not seeing any results when in reality they may
• 59:42
  be producing a tonne of results. And so it's like it's up to us
• 59:46
  to clearly define how we are impact in the business.
• 59:50
  And I appreciate the both of you breaking this down for me and
• 59:55
  really going deep into different ways that we can explain it and
• 59:59
  we can show leadership the value of the ML and AI efforts.
• 1:00:06
  Perfect. Thank you so much, Dimitris.
• 1:00:08
  Thank. You thank you for having us.