Latest / Agentic Conversations (formally mlops.community) / MLOps for GenAI Applications // Harcharan Kabbay // #256

Transcript

MLOps for GenAI Applications // Harcharan Kabbay // #256 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm Hassam Kabe, go by Harry, work with worldwide technology
• 0:04
  as a lead ML engineer. I'm really more like a tea fan,
• 0:10
  but whenever I take coffee I usually go with the black coffee
• 0:14
  and like whatever regular brand is there, not not having a
• 0:19
  particular affinity to anything. What is up ML OPS community?
• 0:23
  We are back with another episode of the ML OPS Community Podcast
• 0:28
  and as usual, I'm your host, Dimitri OS.
• 0:32
  We're going to get right into it with Harry because this is an
• 0:35
  episode all about reliability. If you are trying to do anything
• 0:40
  in production, you know how important it is.
• 0:43
  He breaks down all the ways that things can go a wire and how he
• 0:48
  goes about stopping them. I will give you a little hint,
• 0:52
  he talks a lot, a lot, a lot in this conversation about
• 0:56
  templatizing things and also processes.
• 1:01
  Let's get right into it. But before we do, I want to
• 1:03
  share with you one song that I have found.
• 1:07
  In fact, it's a whole album. But since we do not have time to
• 1:10
  listen to the whole album, I'm just going to share this
• 1:13
  beautiful first song off of the album.
• 1:18
  And if you know anything about me, you know that I adore
• 1:22
  Gregorian Chance and jazz music. Specifically jazz music with
• 1:28
  Chet Baker and Bill Evans and a little bit of Stan Getz and Tom
• 1:34
  Jabeen. But the reason I want to play
• 1:37
  this song with you is because it merges my 2 loves of Gregorian
• 1:45
  Chance or this very traditional, almost like cathedral music with
• 1:54
  the new cool hip jazz. And it has some saxophones in
• 1:59
  there. The song or the album is
• 2:02
  officium. The song that we're listening to
• 2:06
  here is called Perezi Miji Dominil.
• 2:12
  Anybody that speaks Italian can correct me on that.
• 2:16
  What a beautiful song. Let's just bring this up a
• 2:19
  little bit, bring it into focus. So much emotion, so good.
• 2:57
  Let's get into the conversation we had with Harry.
• 3:01
  Wishing you all such a beautiful and wonderful day.
• 3:06
  I just can't help myself listen to this type of music.
• 3:19
  Dude I got to start with this. What do you have against local
• 3:23
  LLMS? It's it's not like, you know,
• 3:26
  hard stance. It's more like the value out of
• 3:29
  that, like, you know, local LMS, good for experimentation.
• 3:32
  But The thing is people are more excited about, you know, they
• 3:36
  can run those things like on Mac, like you can, you can spin
• 3:39
  it up like this and that I'm thinking about the bigger
• 3:42
  picture. That is how do you get these
• 3:44
  things operationalize? How do you get value out of
• 3:46
  those? Right?
• 3:47
  And I have experience working on Kubernetes like a more hardcore
• 3:54
  guy, part of the team who actually took it all the way up
• 3:59
  to writing our own operators. So I understand how these things
• 4:02
  work. And when I think from that
• 4:04
  angle, I feel like, OK, local testing is good up to a point.
• 4:08
  But even then, I don't want to make it like, you know, that
• 4:11
  everyone is doing on their own. You have no control and little
• 4:15
  bit of control or at least the manner how you are accessing
• 4:19
  makes a difference. If we can deploy it in a way
• 4:23
  like a like AAPA kind of thing right from the beginning, it
• 4:26
  makes more sense to me. So it's almost like you're
• 4:31
  afraid that people are creating bad habits by getting used to
• 4:36
  these local LLMS, Almost like we did with Jupiter notebooks where
• 4:41
  you do things. All of a sudden it becomes
• 4:44
  standard practise to do everything in your Jupiter
• 4:46
  notebook and then you throw it over the fence and you say OK
• 4:49
  now figure this out because I got some cool working on my
• 4:53
  local drive and now it's an OPS problem.
• 4:56
  Yeah, you said it more accurately.
• 4:59
  Well, one thing that I, I think you've got a lot of insight into
• 5:03
  is really operationalizing this from the Cicds perspective.
• 5:09
  And so let's take the classic 2024 use case of rag.
• 5:15
  I don't care if it's naive rag, graph rag, advance rag, I'm
• 5:19
  going to make a rag rap song and put all of these different types
• 5:23
  of rags into it. But you want to be able to make
• 5:29
  sure that it is battle tested when you are doing it.
• 5:34
  And CICD and testing integration tests, all the different kinds
• 5:39
  of tests that you can have is a great way of doing that.
• 5:42
  I know you've got some cool things to say about that, so I'd
• 5:46
  love to hear your perspective. Sure.
• 5:49
  So already lot of stuff has been talked about lag, I'm not going
• 5:52
  to go into that. What it is people already
• 5:55
  understand. But one thing that has changed
• 5:57
  with the the coming of the other lenses like no, people start
• 6:02
  thinking like AP is that there's some APA model that you can hit
• 6:05
  and get the results back, right. So when it comes to RAG or
• 6:09
  operationalizing RAG, you have to you have to think about like
• 6:13
  in a microservices way. You have your embeddings or
• 6:18
  vector stores somewhere stored could be on premises or in the
• 6:22
  cloud. Then you have the other land.
• 6:26
  Most of the companies, I think people are using the
• 6:30
  Lamborghinis out there like TPT 4 or other things.
• 6:33
  I mean, they're having some latest a lot of claims around
• 6:37
  that. But how do you run those things
• 6:38
  on process is still not questionable.
• 6:41
  But The thing is you most, most of the data scientists are the
• 6:45
  MLA engineers. They used to call those rules,
• 6:47
  but now it's more like EA engineers or AP ensemblers say
• 6:50
  like that orchestration engine, right?
• 6:56
  So basically now you are making call, you get a query from a
• 6:59
  user, you make a call, you do search, do retrieval, right, get
• 7:03
  some results back, send it to LLM after with their
• 7:08
  augmentation in the prompt and you get the result back, right.
• 7:12
  You have different hops there. So before we jump on to CACDI,
• 7:17
  really put some emphasis on resilient architecture like all
• 7:21
  of these could be single point of failure.
• 7:22
  So you have to think about how would you not fail on any of
• 7:27
  these as a single point, right? Like LLMS, or you can have a
• 7:31
  pool of LLMS and you overcome the 420 nines or the rate limit
• 7:35
  errors. Same thing for the databases,
• 7:38
  like if an instance is unavailable, set those in a
• 7:41
  cluster format, right? And then you start thinking of
• 7:45
  these like as deployments and then your CICD come into
• 7:49
  picture. Right now the orchestration
• 7:52
  engine that we talked about, which is responsible for that
• 7:56
  blue thing, like you're talking different pieces.
• 8:00
  How I have seen I've been working upon is more like a
• 8:03
  Kubernetes setup. And in Kubernetes you have the
• 8:08
  different concepts like deployments or Statefulset.
• 8:11
  These are more like stateless apps.
• 8:14
  So a simple deployment could work, but it's it's multiple
• 8:18
  replicas in there. But then how do you bind
• 8:22
  everything like, you know, people are developing that or
• 8:24
  the ML engineers are doing the development work.
• 8:26
  How does that get get to the Kubernetes or some CID around
• 8:31
  that right CACD around that Argo CD is one of the ways you can do
• 8:35
  that. There are other options as well,
• 8:37
  but I like it super easy to consume.
• 8:39
  You can set it up like, you know, keep listening to this
• 8:43
  GitHub. Whenever you see updates, you
• 8:45
  can perform a sync on to the other side.
• 8:48
  So that also provide you the option to rollback if you want
• 8:52
  to. But if you take one step back,
• 8:55
  like when you do the core development itself, right, how
• 8:58
  can you make sure that you are following some kind of standard
• 9:02
  there? Like you containerize your code,
• 9:06
  right? So that way you keep track of
• 9:10
  all your libraries. You have the image version now
• 9:13
  that image version or creating the image, you'll not be running
• 9:16
  Docker Compose. If you are operationalizing
• 9:18
  these things, you'll have some automation like in set up some
• 9:21
  Jenkins kind of job that you know listens to your GitHub like
• 9:25
  activities there. And whenever you have a merge,
• 9:28
  it's going to run or create an image for you.
• 9:30
  And then you'll see you'll need some images stored like you know
• 9:34
  where you're keeping your artefacts.
• 9:36
  We don't keep it in GitHub, right?
• 9:38
  GitHub is not meant for those things.
• 9:40
  So the bits and pieces of automations are here and there,
• 9:44
  but the bigger picture is once you have the Argo CD set up, so
• 9:49
  container is just the code. Now you have to also think about
• 9:53
  that you'll not be running it just directly in production.
• 9:56
  You'll have those dev test environments or maybe stage or
• 9:59
  pre prod depending upon how the companies are using.
• 10:02
  But then you'll not be creating separate images either, right?
• 10:06
  So with image, you'll have two pieces of configuration, one
• 10:10
  that is non confidential, right? Like your URLs that you're
• 10:14
  hitting or some parameters that you're tuning, right?
• 10:18
  But then you have the secrets as well, like APA keys, you'll not
• 10:21
  be keeping it in the code. So those kind of things in
• 10:26
  Kubernetes, you have the concept of secrets.
• 10:28
  You can use secrets, right? But then you can announce it
• 10:31
  further, like going to think about external secrets stored in
• 10:35
  Hashi Corp or you know, Azure keyboard.
• 10:38
  You can set up those operator that keep on syncing those
• 10:42
  things for you and make sure that you have the latest secret.
• 10:45
  That way you can implement like more segregation of duties as
• 10:50
  well. Like you know I have the
• 10:51
  engineers who are just responsible for rotating the
• 10:55
  keys. They do that in keyboard.
• 10:57
  It automatically gets synced in there and applications are
• 11:00
  consuming those. The other configuration we
• 11:03
  talked about is like non confidential, like, you know,
• 11:07
  API endpoints or I want to do tweak a little bit of
• 11:11
  temperature here and there. I don't want to put it into the
• 11:14
  code because that way I'll be recreating my images, right?
• 11:17
  I don't want to do that. So those kind of things you can
• 11:20
  manage via config maps and Kubernetes.
• 11:23
  So that way when you have these things super easy, you upgrade
• 11:26
  your code. Most of the things when you're
• 11:29
  testing, you may not be touching your images or regenerating the
• 11:32
  images. You are just changing your
• 11:35
  config maps, which is all your configurations, how you're
• 11:37
  tweaking it, right? For example in your retriever
• 11:41
  you have been say considering like 10 top 10 results.
• 11:45
  Now someone says OK let us try it with 15 results, right?
• 11:49
  If you have that saved in your config maps, all you need to do
• 11:52
  is just update there, sync it up, restart your port, and
• 11:55
  you're done. And hopefully people are going
• 11:58
  from 4:50, not ten to 15 because.
• 12:02
  That's. Going to just kill your output.
• 12:06
  But I guess every use case is different.
• 12:09
  No, I think about Kubernetes being such a beast and you're
• 12:17
  able so it's on one hand it's a beast, but you want that, you
• 12:23
  want that capability and you want that power.
• 12:26
  And so from my side, I'm thinking about, OK, how have you
• 12:34
  been seeing differences from the LLM side and just grabbing these
• 12:40
  commoditized models and having a different rag architecture
• 12:46
  versus I'm doing some kind of recommender system.
• 12:51
  I have the model that I created. Am I baking it into the Docker
• 12:55
  image? Am I like because you don't need
• 12:58
  to have that config map necessarily or it's not as
• 13:02
  important. So there's some, it's almost
• 13:04
  like there's these, these differences, but at the end of
• 13:08
  the day, it's not so different. Yep.
• 13:12
  I mean, even if you go to the primitive MN models, right, Once
• 13:17
  you start thinking like, no, I want to sell these as like
• 13:21
  API's, not just, you know, passive and fresh that it is
• 13:25
  running like a Chrome job. I tried something with the key
• 13:28
  server key native serving. It used to be called like that,
• 13:31
  right? It's more like a sort of this
• 13:33
  kind of thing is super cool. It has as a like a preprocessing
• 13:38
  then your main thing and then you can do post processing.
• 13:41
  It's super cool. And you see, similar to key
• 13:44
  server, there's a racer about there that I'm going to play
• 13:47
  around a little bit with. But yeah, those kind of models
• 13:51
  you don't you, you select accurately there.
• 13:54
  You don't need to bacon into a Docker image.
• 13:57
  You can create those apps like using some inference APA setup
• 14:02
  there. That can also reduce some work
• 14:04
  because you don't want to, you know, create DNS entry for each
• 14:08
  and every app you deploy. So I think the knowledge of
• 14:12
  Kubernetes like like you said this, it's a beast like the lot
• 14:16
  of possibilities to know what you need makes a big difference.
• 14:21
  You don't want to apply the same rule of pump everywhere, right?
• 14:25
  You'll be creating a lot of work for yourself.
• 14:28
  Yeah, that's such a great point. And I find it fascinating how
• 14:37
  people with different backgrounds or coming into the
• 14:40
  ML space reach for the tools that they know, and they are
• 14:47
  trying to figure out how to make this tool that they're very
• 14:50
  familiar with work with this new paradigm that they're now
• 14:54
  getting themselves into. So if you talk to a data
• 14:58
  engineer, it's almost like, yeah, well, Airflow's what I go
• 15:01
  with, and it's not necessarily the best.
• 15:04
  It's kind of a pain in the ass sometimes.
• 15:06
  But then maybe if they're progressive, they'll use some
• 15:11
  more data engineering new age type of tool, like a mage or a
• 15:16
  prefect or something like that. But then if you talk to the dev
• 15:20
  OPS folks, it's like, yeah, Kubernetes all the way.
• 15:24
  We're using Argo workflows or Argo CD also, or maybe they're
• 15:29
  saying we'll use a bit of cube flow if they somehow got conned
• 15:36
  into having to use cube flow, which pour out a little bit of
• 15:42
  drink for everyone out there that is using cube float and
• 15:47
  having to suffer through it. But that is, it's always
• 15:51
  interesting to me on how you approach these different sets of
• 15:56
  problems by knowing what your background is and having the
• 16:01
  comfort of saying, I'm really skilled in Kubernetes, so I'm
• 16:06
  going to make Argo CD work with it.
• 16:08
  And you also understand inherently what is important
• 16:14
  and, and for you, it's probably really easy to just reach for
• 16:17
  some kind of infrastructure as code.
• 16:19
  I imagine you're going for Terraform right off the bat.
• 16:22
  And whereas if you tell a data scientist that it's almost like
• 16:26
  infrastructure as what? Where are we going with this?
• 16:29
  You're right, you're right. I think everyone has a piece to
• 16:33
  play and it's a teamwork. It's not no one man
• 16:36
  responsibility to operationalize.
• 16:38
  We need that testing happen, but should not be so carried away by
• 16:43
  running it locally that you think about the biggest, the
• 16:46
  bigger picture. You, you forget about the bigger
• 16:48
  picture, right. Should be keeping those things
• 16:50
  in mind, right? Yeah, yeah.
• 16:53
  We'll, we'll touch upon that form.
• 16:56
  So the the other cool part that you were saying and specifically
• 17:00
  on that to your point, you having the dev OPS background
• 17:08
  and chops, of course you're going to be looking at something
• 17:13
  from the bigger picture and saying this is great, but how do
• 17:18
  we get it into production? How do we make sure that when
• 17:21
  it's in production, it is checking all of the boxes?
• 17:24
  Because it's not just like we can have it in production and
• 17:27
  you can cover the output side of things.
• 17:31
  You also have to have that security and you have to make
• 17:33
  sure that you have no vulnerabilities.
• 17:35
  And you have to. There's so many different pieces
• 17:38
  that you're seeing from your vantage point that potentially
• 17:43
  others who don't have this type of background aren't looking at
• 17:48
  or aren't even thinking about. Yeah.
• 17:50
  You're right, I I think that that's related to what I was.
• 17:54
  Oh nice. Thinking about at that point,
• 17:56
  right. So you mentioned Q flow.
• 18:01
  I, I had my to operationalize Q flow and you know, having coming
• 18:07
  from that Kubernetes background, it helped me a lot.
• 18:10
  But then we have to think about the security and other aspects,
• 18:12
  right? I, for example, like a tiny
• 18:16
  little things here and there. You don't want to share the
• 18:20
  database passwords or other things with everyone, right?
• 18:23
  Are there ways that you can make it available if someone needs
• 18:26
  that, right? But then you have to we, you can
• 18:30
  implement some of those things right from the beginning, like
• 18:33
  in queue flow when you create the notebook, you can create
• 18:36
  configs. So like pod defaults.
• 18:40
  I, I wrote something about that a couple of weeks back.
• 18:46
  So you can, you can set those up.
• 18:47
  It basically matching the label of your pod.
• 18:51
  It can make certain things available to it, right?
• 18:54
  For example, you say credentials tell you didn't find your own
• 18:58
  tag. It'll make those credentials
• 19:00
  available because you defined it that way in the port defaults.
• 19:03
  It's the power of port defaults to get it done in Kubernetes you
• 19:07
  have to be thinking about mutating webhook.
• 19:10
  You'll be setting up your own server.
• 19:12
  But The thing is, you may not need to overengineer if there
• 19:16
  are some tools available. I'll rather save my time and use
• 19:20
  those tools other than, you know, showing off my knowledge
• 19:23
  there because it also creates a lot of LCM for me.
• 19:26
  If something something gets updated in future, right?
• 19:30
  It's unpleasant. I I don't want to take that all
• 19:32
  work, right? Unnecessary work.
• 19:34
  I'll be rather spending my time on something newer or cooler.
• 19:38
  Yeah, yeah, totally. You're you should start trying
• 19:41
  to give yourself a job exactly. Or give yourself more work.
• 19:46
  Really the whole idea is, hey, let's automate as much as
• 19:50
  possible so I don't have as much work where I can have less work.
• 19:55
  So you mentioned about teleform earlier.
• 19:58
  If you want we can dive a little bit into that one would love to
• 20:02
  perfect. That's the topic that is very
• 20:05
  close to my heart. Even after all the shit went
• 20:08
  down with Hashicorp and the. For those who don't know, can
• 20:14
  you school us a little bit? I think what happened was
• 20:16
  Hashicorp, who is the the lead maintainer or basically the
• 20:23
  company behind Terraform, they changed the open source licence
• 20:29
  for Terraform and people who were using it after a certain
• 20:32
  version had to start paying a licence or an enterprise fee or
• 20:37
  something along those lines. And it just made the community
• 20:40
  that was built around Terraform revolt and people were pissed.
• 20:44
  So pissed that they did a hard fork.
• 20:46
  Anyways, I digress. Talk to me about terraforms.
• 20:49
  Yeah, I mean Terraform or, or you know, NC world or any other
• 20:54
  thing that the the point is that no one in your eyes creating
• 20:59
  these things manually. Like, you know, when something
• 21:02
  got released on Azure, like Azure open AI models site,
• 21:06
  everyone was super excited and you know, jump on to that and
• 21:10
  start doing that. So one of my friends and mentor
• 21:13
  from my current org, he told me like, you know, five years back,
• 21:18
  I, I was learning how to use and see what at that point he
• 21:23
  mentioned that, you know, while logging into servers, if you
• 21:26
  don't need to, if you have the configuration in there, right?
• 21:29
  So when you start thinking about those things, those are very
• 21:32
  helpful when you are operationalizing, you'll think
• 21:34
  about all, OK, if I need to go there and create a storage
• 21:39
  account like, you know, now a new member is onboarded, I
• 21:44
  assign that work to him. I'll be passing on several
• 21:48
  things. One, this is the naming
• 21:50
  convention. This is how you should create a
• 21:52
  resource group. You know, these are the tags you
• 21:54
  need to put on and these are the network firewall rules you need
• 21:59
  to enable, right? Or any additional configuration
• 22:01
  like you can think about 11:50 points that you need specific
• 22:05
  about that. And that is an example of one
• 22:07
  resource. Now think about open AI or any
• 22:11
  search or you know database instance there.
• 22:14
  So when all these configurations is multiplied it gets out of
• 22:19
  control. So that is where like Terraform
• 22:25
  is super handy. I feel like anything you are
• 22:27
  doing like provisioning in cloud super useful.
• 22:32
  You just set it in a correct way and you can have a good version
• 22:37
  control that way. Everyone is responsible.
• 22:41
  If like I published a post sometime back about how I
• 22:46
  created a modular approach like having each of those resources
• 22:49
  in its own module and then create inventory for each of
• 22:54
  those resources. For example, resource groups
• 22:57
  create a CSV so it is going to loop through that and read and
• 23:01
  you can have the parameters. However you want to apply like
• 23:03
  set of configures that you want to apply.
• 23:06
  Now if I need to create those resources like I want to go from
• 23:10
  10 to 20 resource group now, right, I was on 10 so I can ask
• 23:15
  anyone to just add those to that CSV create a PR someone gets
• 23:19
  reviewed. They are not changing the
• 23:20
  configuration. They are just adding to the
• 23:22
  inventory file. The configuration is going to
• 23:25
  loop through that create those for me every everything is
• 23:29
  defined in set of rules that I want to apply on top of those so
• 23:32
  breezy. And the other thing I learned
• 23:35
  over the years is like when you, when you go through GitHub
• 23:38
  route, you create a PR, you need to educate a little bit.
• 23:41
  Like, you know, people should not be approving it just for
• 23:44
  sake of approving. I get it.
• 23:45
  And I do that like little things make big difference.
• 23:51
  If you put some like a template for APR, right?
• 23:54
  It gives you a number of options.
• 23:56
  Then asking people, OK, just review that, you know, making
• 24:00
  sure I haven't added anything confidential.
• 24:03
  Do that. I did my due diligence to test
• 24:05
  certain things. I think everyone feels
• 24:08
  responsible and they make sure you know the code that we are
• 24:12
  pushing. It is reviewed.
• 24:15
  It's definitely better than how what a like if you assign it to
• 24:19
  a person would do right? And doing your QA on top of
• 24:22
  that. Wait, say that again.
• 24:24
  So basically just giving almost like a check box, check boxes
• 24:29
  that people can look at and say, did it go through all these
• 24:32
  things? And not like a reminder list
• 24:36
  like, you know, you can create your own template.
• 24:38
  I'm big fan of templatizing. Yeah.
• 24:41
  I actually listed on something that we should talk a little bit
• 24:44
  on that You can templatize the different things like in
• 24:47
  operationalizing or helping the teams out there.
• 24:51
  I tried to templatize my dating with my wife.
• 24:55
  I didn't go over that well. I think that's taking it a
• 24:57
  little too far. Man, you have two separate
• 25:01
  things. So anyways, yeah, so you
• 25:05
  templatize the PR in a way that you're getting like, basically
• 25:15
  you're getting everyone to look at it, not just one person to
• 25:20
  own it. That, that that's the value of
• 25:23
  PRS, right? If you and I are collaborating
• 25:25
  on a project, you developed something, you said, OK, you
• 25:28
  know, go ahead and make these changes.
• 25:31
  I do that, I create a PR, but I don't want to create too much
• 25:36
  work for you. Like there's certain things that
• 25:38
  I should be double checking on my behalf, right?
• 25:41
  So that is the PR template. As far As for my so when I
• 25:45
  create a PR, it's going to prompt me with all those things.
• 25:48
  Make sure your code is following, you know, the
• 25:51
  standard camel keys or other things, naming convention,
• 25:55
  whatever you have, make sure you don't add any confidential
• 25:58
  information. And they're right, I'm going to
• 26:00
  do that. Or maybe, you know, make sure
• 26:01
  that you tested these things locally, then I submit that PR
• 26:05
  and then you're going to review. When you review this, it's not
• 26:08
  just, you know, checking that, OK, I, I got a PR, I just
• 26:11
  clicked that you're going to verify the things that you are
• 26:15
  supposed to verify. So that way it's not just I
• 26:18
  don't the code, but it's a shared responsibility when we
• 26:21
  push those changes to the next level.
• 26:24
  Yeah, I like the idea of the shared responsibility and
• 26:28
  getting everyone on board, but I my mind instantly goes to what
• 26:35
  do you do in a retro or what do you do when shit hits the fan.
• 26:38
  Everybody probably does the Spider man meme and just points
• 26:41
  it somebody else. It depends from org to org, The
• 26:47
  organisation that I'm currently part of a really super cool,
• 26:52
  very high test score. Yeah.
• 26:55
  And, you know, there are honest mistakes that happen.
• 26:58
  So we should not be like no going to the blame game.
• 27:02
  It's more like, OK, what are the lessons learned out of that?
• 27:06
  First thing, it's not going to happen overnight, right?
• 27:09
  You have to shift your like a design patterns from people
• 27:16
  oriented design to more process oriented that you have things
• 27:21
  properly documented. Wherever you need to automate,
• 27:24
  you automate. But it is it should not be like
• 27:26
  that. I automated my now no one knows
• 27:28
  because I don't have a read me right.
• 27:31
  It's all in my mind. So there's a, there's a very
• 27:35
  thin gap when it comes to a subject metrics part and a
• 27:39
  blocker because you know, once you start working as a subject
• 27:42
  metrics part, you cross into a line where you're blocking
• 27:45
  everyone else now because you created a dependency.
• 27:48
  So I'm more towards advocating open it up, you know, get the
• 27:54
  things out there properly documented, have knowledge
• 27:58
  sharing sessions, involve others, mentor others.
• 28:02
  One youth, you should be learning you, there should be
• 28:05
  people in your team or your company that you should be
• 28:07
  learning from. Then there should be that who
• 28:12
  are you are mentoring? You should be spreading that
• 28:14
  knowledge out. And there should be someone who
• 28:17
  should be double checking your facts, right?
• 28:20
  Like I have something on LLM. Then I can ask you, OK, what do
• 28:24
  you think about that? And you say, OK, that doesn't
• 28:26
  make sense at all, dude. Like we need those things if it
• 28:30
  is just all mean there is no learning.
• 28:37
  So this idea of making sure that you don't become the subject
• 28:41
  matter expert, you're talking principally in the code base.
• 28:47
  It's not necessarily how we might think about it in the ML
• 28:51
  world of, oh, you have the subject matter expert that is
• 28:54
  one of the stakeholders in this use case and you want to get
• 28:58
  them to label some data or do something tell you about the use
• 29:01
  case. It's more you saying, if I am
• 29:05
  the only one that knows how this application runs, I am the
• 29:09
  subject matter expert on this application and all of the
• 29:13
  questions are going to have to come through me and that's going
• 29:15
  to no doubt slow things down. Yeah.
• 29:21
  Yeah, you are right. So it's not see you need subject
• 29:25
  metrics parts, but it's more about how you put those things
• 29:28
  into operations. You should not be blocking other
• 29:31
  that people don't have any clue how this thing is running,
• 29:33
  right. I mean, and the way you
• 29:37
  implement the thing you how do you write code with proper
• 29:40
  commands or read means that is very necessary.
• 29:42
  So you need to set up those things more process oriented,
• 29:46
  not people oriented that that. That was the point I was trying
• 29:48
  to make. Yeah.
• 29:50
  Well, it's funny you say that because I've been on this kick
• 29:53
  recently on how dev OPS specifically, right?
• 29:56
  It's talks about people, process and technology.
• 30:00
  And I make the point that a lot of people forget in that group
• 30:08
  of three, technology comes last. And so, but people tend to put
• 30:14
  technology first, especially when it comes to ML OPS or the
• 30:19
  LLM phase craze that's happening.
• 30:23
  It's very much about, here's the tools that you can use to solve
• 30:27
  your problems, not necessarily about here are the types of
• 30:34
  people you need and what your team should look like, what kind
• 30:37
  of experience they should have. Here's the processes that you
• 30:40
  should be thinking about. And then last but not least, you
• 30:44
  can go and dip into the technology section.
• 30:48
  What you're saying? Yeah, people shouldn't be first.
• 30:51
  There should be processes first. I think people have a role.
• 30:55
  I'm not saying that anything with the process you have, you
• 30:59
  need people, right? But I'm saying is you should not
• 31:01
  be stuck with, OK, there is someone who knows about it, but
• 31:05
  we don't know how that thing works.
• 31:07
  There's no knowledge around that, right?
• 31:09
  So when you implement why the designs are poor in my opinion
• 31:12
  is people keep most of the things in their mind and you
• 31:18
  know, they implement thing or they get the things working, but
• 31:22
  you have to think about, you know, you don't want to be stuck
• 31:24
  with that. Like I'll be supporting all the
• 31:26
  tickets around that thing. I have to open it up so that I
• 31:30
  can get on to the newer things and let someone you know, learn
• 31:33
  and help with the the day-to-day task or the support generated by
• 31:38
  those things. All right, real quick, I want to
• 31:40
  tell you about our virtual conference that's coming up on
• 31:42
  September 12th. This time we are going against
• 31:45
  the grain and we are doing it all about data engineering for
• 31:49
  ML and AI. You're not going to hear rag
• 31:53
  talks, but you are going to hear very valuable talks.
• 31:56
  We've got some incredible guests and speakers lined up.
• 32:02
  You know how we do it for these virtual conferences.
• 32:04
  It's going to be a blast. Check it out right now.
• 32:07
  You can go to home dot ML OPS dot community and register.
• 32:14
  Let's get back into the show. There's a fascinating piece to
• 32:18
  me changing gears now when it comes to the idea of monitoring
• 32:22
  and how you would approach monitoring being more of a dev
• 32:26
  OPS person. But then in this LLM slash ML
• 32:31
  world, you need to be monitoring so many different things than
• 32:34
  just the system, right? And the logs and traces are
• 32:37
  great, but what other things are you monitoring?
• 32:41
  And especially with ML, you're going to be monitoring the
• 32:46
  predictions and how they're doing and the business metrics
• 32:51
  that they're going against, but you are potentially going to be
• 32:55
  monitoring the data and the data flows.
• 32:58
  And so having somebody with a dev OPS background, I imagine
• 33:03
  when you're starting to think about data monitoring, you're
• 33:07
  trying to fit it into this box of, all right, I'm coming from
• 33:11
  dev OPS, I'm looking at logs and traces.
• 33:14
  Now I should, are there logs and traces for like the data
• 33:18
  monitoring? Talk to me about that and how
• 33:21
  you how you approach that problem or tackle those problems
• 33:24
  too. Sure, we have a really strong
• 33:26
  team of reliability engineering here at the liberty.
• 33:32
  So when I when I think about logging and monitoring, I would
• 33:36
  say the, you know, the the next version or how we Quan these
• 33:40
  this term these days is observability.
• 33:43
  You don't want to, you know, digging into logs or doing
• 33:46
  monitoring, looking at the things you should be aware of
• 33:48
  the the accurate things coming out of that.
• 33:51
  Right. I'll, I'll talk a little bit
• 33:54
  like a before rather than think, right, the permit of ML models
• 33:59
  like you run pipelines maybe in in any ML OPS platform, right.
• 34:06
  But if you think about you are actually replacing the business
• 34:09
  processes. Someone was you doing their work
• 34:11
  manually, they were maintaining things in Excel and here you
• 34:14
  come with the knowledge of EIML data science.
• 34:18
  You said, OK, I understand your business problem.
• 34:21
  Let me create a model for you. OK, so I replaced whatever they
• 34:25
  were doing, but at that time when we replaced, we took a sign
• 34:29
  of that. OK, I commit that my model will
• 34:33
  be delivering this much of accuracy.
• 34:36
  Now there was someone had to knowledge that you've replaced
• 34:39
  with your model over the years. People are going to rely on your
• 34:43
  model more and more if you do not have observability, like,
• 34:47
  you know, this is the the limit I committed to, but you're not
• 34:50
  measuring yourself. You're marching towards a
• 34:53
  failure because you're not keeping an eye on the data
• 34:57
  drift. If the underlying data is
• 34:59
  changing, there's no way that you're monitoring someone is
• 35:02
  going to point out at some point and that'll be unpleasant
• 35:06
  because you know, when these things are monitor or auditor,
• 35:11
  you have no time to fix those things back, right?
• 35:14
  Yeah, it's too late. So I played a little bit around
• 35:18
  in queue flow, like how you can emit those extra metrics.
• 35:24
  The thing is, if there's a packaged model, there are a lot
• 35:28
  of things already out there. But if you have understanding of
• 35:31
  Prometheus exporters or you're creating your own model, it
• 35:34
  could be a simple linear regression model.
• 35:37
  You have your training pipeline, you can think about, OK, let me
• 35:39
  emit these metrics when I'm running, when I whenever I'm
• 35:43
  running the training, right, I want to emit these metrics like,
• 35:47
  you know, for example, the runtime, this is my environment,
• 35:51
  the number of rows I processed and these are the metrics coming
• 35:54
  out of that, right? And you can easily though these
• 35:57
  things to graph on our dashboards or a similar kind of
• 36:00
  a like, you know, UI set up alerts on top of that.
• 36:06
  You can see things beforehand when it actually, you know, dips
• 36:09
  down. All the accuracy is not what you
• 36:11
  promised, but the same time you already touched upon all the
• 36:15
  important parts you like you talked about the data pipelines,
• 36:18
  right? So if you have something based
• 36:22
  out of the spread of the data already, right?
• 36:25
  So rather than relying on just the model, because model is the
• 36:30
  last stage where you're running those things, but the data
• 36:33
  change has already happened or it is happening over week,
• 36:35
  right? So if you keep an eye on that in
• 36:38
  a way like, you know, visually getting things out of that,
• 36:41
  creating some cool visualisation on top of that or maybe some
• 36:45
  logic that you are saying, OK, it is going beyond the standard
• 36:49
  deviation, you know, the range that is permissible.
• 36:53
  I should be later, I should start working on that.
• 36:56
  So that way it is, it is super important.
• 36:58
  Now when it comes to LLMS, it is, it's like, you know the, I
• 37:06
  would say it's a free range of metrics that you need to now
• 37:09
  think about you, you cannot say, OK, I have this number, I'm
• 37:12
  good. Now you talk about RAG, right?
• 37:14
  You take a step back, you are doing basically three things
• 37:19
  there. You, you are getting the results
• 37:21
  back and then you are having augmented import, right?
• 37:25
  That is you use your prompt, you go to your retrieve results in
• 37:29
  there and you send it to LLM for final response.
• 37:33
  So if you say I'll just get the final response evaluated, that
• 37:38
  is half picture because if you retrieved results are not up to
• 37:42
  that quality, you'll not get the final response right.
• 37:46
  So now that evaluation has to happen at that step as well.
• 37:50
  And then you have to think about, OK, how do I do that?
• 37:53
  Like you provided me a list of questions.
• 37:55
  I run it, I get the results from my vector store, but how do I
• 38:00
  evaluate if it is actually accurate or getting the things
• 38:03
  back? So you can think about, you
• 38:06
  know, different approaches. LLMS can be used to donate
• 38:10
  question answers as well. People have been trying that.
• 38:12
  You provided a, a document, you say give me a list of question
• 38:16
  and answers out of that. You can use those credits to
• 38:19
  then do a search and see if your document pieces show up there,
• 38:24
  right. And that is the piece on the
• 38:26
  retrieval side. Then you can see the order where
• 38:28
  they're coming up. Do you need a good read anchor
• 38:30
  over there? But you need to measure those
• 38:34
  things at separate points and see how they differ.
• 38:38
  So that goes back to our point where we were talking about the
• 38:42
  CACD as well, or how you deploy these, you know, separating
• 38:45
  things in config maps and the secrets out there.
• 38:49
  If you don't have it set up properly, then you'll be doing a
• 38:52
  lot of work regenerating those images.
• 38:54
  So think about, you know, I, I might be playing around with
• 38:57
  these parameters. I need these APA endpoints like
• 39:01
  instead of getting just the final response back my image or
• 39:05
  the application that I'm designing, it'll have another
• 39:08
  endpoint that will just give me the search results.
• 39:10
  So that way it'll be easy for me to do the evaluation of those
• 39:15
  retrieved results as well as the final results if I want to do
• 39:18
  that. And the the other piece that
• 39:23
  comes to my mind is the inference latency.
• 39:26
  So when you use LLMS or any APIs, right, you have to think
• 39:31
  about is it a batch processing or is it a real time.
• 39:36
  So in the batch you can think about pops up hub model, you
• 39:39
  have something like a queue mechanism.
• 39:41
  You know your applications are publishing to that and then you
• 39:45
  know slow and steady you are picking from that.
• 39:48
  You keep your not huge compute footprint, but you keep it up to
• 39:53
  a level where you keep on, you know, reading from that queue
• 39:56
  without creating too much of backlog.
• 39:59
  But when it comes to real time, like most of these chat bots,
• 40:03
  whatever we are doing with the LLMS, these are all real time.
• 40:06
  You don't want your users to be waiting in minutes to get a
• 40:09
  response back. It doesn't make any sense,
• 40:10
  right? So in those you should be
• 40:12
  capturing or measuring, not capturing the measuring the the
• 40:16
  end press time lag there. How much time does it take?
• 40:20
  And that is where the things we discussed like, you know,
• 40:24
  resilient architecture. If your client is doing a retry,
• 40:28
  like I hit a LLM gave me a rate limit, what do I do next?
• 40:33
  Right? How can I reduce that time?
• 40:36
  Like if I know my limits already, I can beep up the rate
• 40:39
  limits for the LLMS, but it is not always possible.
• 40:42
  You have to think about. I'll create a pool so there are
• 40:45
  different products available there.
• 40:47
  One of the things that I explored a little bit is Azure
• 40:51
  API Management. I'm going to write a something
• 40:54
  on that very soon like proxy. Yeah.
• 40:58
  The thing is you can capture all those metrics out there and
• 41:04
  without measuring I think you cannot improve, right.
• 41:07
  I mean you need to measure 1st and then you are in good shape
• 41:10
  that you can improve upon. You can see how it makes a
• 41:12
  difference. You on ordered a new model if it
• 41:14
  is taking much time, more time or less time, right.
• 41:18
  And then in observable decent, you know, DevOps background, you
• 41:21
  mentioned already resource utilisation is also very
• 41:26
  important because you don't want to, you know, throw all the
• 41:30
  resources available. I, I remember one of my data
• 41:34
  scientists asked me, can you just provide me, you know, like
• 41:37
  24 CPU's and 48 GB of RAM to run this thing.
• 41:41
  I said, man, why do you need that?
• 41:43
  They don't. So they had a hard time to
• 41:45
  understand request versus limits when it comes to Kubernetes,
• 41:49
  right? So they said OK, I want
• 41:52
  everything as a request like from the beginning.
• 41:57
  So they just wanted it from, even though they weren't going
• 42:00
  to use it all the time, they're like this potentially could need
• 42:05
  this amount. So we might as well request it
• 42:08
  and make sure that we have that capability.
• 42:11
  Yeah, I think so, Yeah. Yeah.
• 42:15
  What'd you tell them? What's what did you school them
• 42:17
  on? Were you like?
• 42:18
  Maybe we could do something? Called serverless.
• 42:21
  Yeah, I told him, you know, the request is more like whatever
• 42:25
  bare minimum you need to get it started and then it can go up to
• 42:29
  all the way to the limits. And you know, this may be good
• 42:33
  enough for you to run like up to maybe 8GB of RAM is good, right?
• 42:38
  We'll find more as we do that. But The thing is, when you do
• 42:41
  the development, you're not going to do it with all the data
• 42:45
  available, right? You'll be doing it with the
• 42:46
  lesser data because it's a development.
• 42:49
  Now we logically have the next step.
• 42:51
  That is your test environment of the UAT where you'll be
• 42:54
  potentially running some kind of regression testing or load
• 42:56
  testing there, right? So that is where if you have the
• 43:00
  proper observability set up, you keep an eye.
• 43:02
  OK, I see with these many users per minute, I'm getting up to
• 43:06
  this level. OK, what do I need to do?
• 43:09
  I see a couple of crashes. I need to bump up if, if it is
• 43:12
  hitting, you know, OMS out of memory errors, I need to bump up
• 43:15
  my memory if I see it's crashing due to CPU limits, I need to
• 43:19
  increase that. And I personally have found
• 43:22
  like, you know, not every app is similar.
• 43:25
  Like if you are just going with a Jaguar, you might need like
• 43:29
  more CPU's as compared to if you are doing some summarization,
• 43:32
  you might need more memory. So it's not one like you know,
• 43:39
  the rule of thumb. I would say it's more like clear
• 43:42
  observability. If you have things in place, go
• 43:47
  through the set of processes like I'm supposed to do a UAT
• 43:50
  before I go to production. I realise, OK, this has a little
• 43:53
  bit different need I need to do that.
• 43:57
  So we talked about logging earlier, right?
• 44:00
  So you could, you could be having a lot of logging
• 44:03
  solutions already where your standard out or standard, right?
• 44:07
  Like no data dogs plunk or anything else, right.
• 44:11
  But the problem with the I decide what I have observed so
• 44:14
  far. Like I worked with six or seven
• 44:16
  different teams doing my position from a database
• 44:20
  administrator on to this side. We keep on dumping a lot of
• 44:24
  information, but do we actually look at that information?
• 44:28
  Never. That's the so classic story,
• 44:33
  Yeah, we may need it later, so we might as well grab it.
• 44:38
  I, I prefer more quality of logging.
• 44:41
  Like whenever you are designing right, you have different
• 44:44
  modules like in the modular approach, you start thinking
• 44:47
  about, OK, what do I need from this?
• 44:50
  Like when this module runs right, what do I like need as a,
• 44:55
  as a mandatory thing or a nice to have thing?
• 44:58
  You can keep those things enabled or disabled.
• 45:00
  Again, going back to the configuration thing, you can
• 45:03
  have a flag there where I say, you know, different logging
• 45:06
  levels, like minimal or, you know, standard or extended
• 45:11
  logging, whatever. And then all of my things get
• 45:13
  enabled or disabled based on that.
• 45:16
  But when you create those metrics, think about in the
• 45:19
  logging, not just to dump that data, maybe create an like a
• 45:23
  nice shape that you can easily create dashboard like, you know,
• 45:25
  Jason format, super easy to read later on.
• 45:29
  But once you roll these things out and, you know, after a
• 45:33
  couple of years, someone says, OK, how do we get this
• 45:36
  information, man? You're writing it in here now.
• 45:39
  It's a big thing. So it's like a lot of LCM, it's
• 45:42
  generated. I have to go back and look at
• 45:45
  the code, how I wrote, maybe some of the things I do not
• 45:48
  recall at all, right? It's more like these little
• 45:51
  things make a big difference if you bring it into your process
• 45:54
  like, you know, like these eight considerations when you write a
• 45:57
  new app. So I'll mention just one thing,
• 46:00
  then I'll wait for your imports here.
• 46:04
  I talked about temperatizing. So what I experiment a little
• 46:08
  bit like, you know, these applications have a broader
• 46:11
  design. We touched on the K native
• 46:12
  serving or K server. You could be running with Django
• 46:16
  server or you could be running with fast API, right?
• 46:19
  You can actually create those repository templates as well,
• 46:24
  like create a skeleton with all those things.
• 46:27
  I usually do that. So I ask people, you know, if
• 46:30
  you have any suggestion, upgrade that template.
• 46:32
  If they have to create a new repository, just create a clone
• 46:36
  so that way they get the bare minimum things or even the
• 46:38
  instruction and the read me that make sure these things are
• 46:41
  covered. That has helped me a little bit
• 46:44
  on you know, making sure like no like 55 mandatory things are
• 46:48
  already there. And then I get imports that
• 46:50
  people feel like OK, now it is easy for me.
• 46:52
  I realised while coding that OK, this thing is missing, we should
• 46:55
  make it available. They create a pull request
• 46:58
  against that template. Yeah.
• 47:00
  So then five years later you don't go back and say how did I
• 47:04
  structure this? What exactly was going on here?
• 47:08
  What was I thinking? And you save yourself a lot of
• 47:12
  time, which makes a tonne of sense.
• 47:14
  I mean you said a lot there, man.
• 47:15
  There is so many threads that I want to pull on.
• 47:19
  And one just going back to the beginning of what you were
• 47:24
  saying on being sure that if you set certain SL AS or SL OS
• 47:32
  around models, you better damn well keep them.
• 47:35
  And the way to keep them is especially the more products
• 47:40
  that are being built with that service or that model, then it's
• 47:45
  almost like you're getting more and more dependencies on top of
• 47:49
  your service. And so it becomes more critical
• 47:54
  now if you are not having the visibility into when the model
• 48:00
  isn't working, you're setting yourself up for a bit of pain
• 48:06
  and hurt. And so I just wanted to say that
• 48:10
  because it's like you gave your word or you said this model is
• 48:14
  going to have this type of accuracy or it's going to have
• 48:17
  whatever it is that you agree on.
• 48:20
  You have to make sure that it does that.
• 48:23
  And like you said, if you're not tracking it or if you don't have
• 48:26
  metrics around it and you're not tracking it and observing it,
• 48:29
  then you don't actually know. And that's where problems can
• 48:33
  arise. And and the other thing you have
• 48:36
  to think about you, you know, you lost the your stakeholders
• 48:40
  confidence if they report some issue, not are you aware of
• 48:45
  that? Like things can happen, models
• 48:47
  may not perform, data has changed, but you should at least
• 48:50
  be aware 1st and you know, it should be other way.
• 48:53
  OK, I'm setting up this meeting. I realise this is changing a
• 48:56
  little bit. We need to, you know, work on
• 48:58
  that. Yeah, get out in front of it.
• 49:01
  Because if they're coming to you and saying why isn't this
• 49:04
  working, it may be OK if they do it once, but if it's two times
• 49:10
  or three times, then it's like, I'm not going to build with that
• 49:13
  service. It's not reliable.
• 49:16
  Yeah. And and then what you do all
• 49:19
  that work for to get that model out into production.
• 49:23
  Why did you even do that if people then later aren't going
• 49:26
  to want to build with that service?
• 49:28
  So, so that was one thing that I, I found fascinating, just
• 49:32
  pulling on another thread that you were mentioning in your last
• 49:37
  answer, really fascinating when you were talking to this data
• 49:42
  scientist who wanted the Max allocation right away.
• 49:47
  And your approach to it, if I understood correctly, is saying,
• 49:53
  let's try and give you the most basic allocation possible.
• 49:58
  And then when we start to progress it through the
• 50:02
  different stages into production, we're going to be
• 50:07
  increasingly trying to break it or trying to stress test it.
• 50:12
  And then we're going to see what comes up and whatever comes up,
• 50:15
  we're going to deal with that when it comes up.
• 50:17
  Is that how you look at it on that road to production?
• 50:21
  Almost. So the example that I gave you
• 50:24
  was more around a Jupiter notebook, like where people are
• 50:28
  running and you know, in queue flow you can create your own
• 50:30
  notebook servers and you can also request your resources.
• 50:35
  So the point that I was trying to get there is, you know, if
• 50:39
  you're coming from a purely data science background, right, you
• 50:44
  don't want to run with lower resources or you say you don't
• 50:48
  understand the difference between request and limits,
• 50:51
  right? So it, it was that kind of
• 50:53
  scenario that I need all the servers.
• 50:56
  So differentiation of story responsibility, like when you
• 51:00
  are developing, right, you are not supposed to or you are not
• 51:04
  required to get all the production data and run
• 51:07
  everything. Like, you know, in the models,
• 51:08
  you can take a sample size, right?
• 51:10
  There are different ways you can do that.
• 51:13
  So it's more like education that you don't need to do that here
• 51:16
  in this environment. It is more for development.
• 51:19
  But then when it comes to request versus limit is more
• 51:22
  like limit is your hard limit. Request is a minimum thing you
• 51:25
  can start with, right? You don't need it because if you
• 51:28
  go with matching the request with limit, you are reserving
• 51:30
  all your resources and then you don't even need those right.
• 51:35
  But with the request and limit, then you can check it from your
• 51:38
  Grafana dashboards or whatever availability you have.
• 51:41
  How much are you reaching like no, are you actually touching
• 51:44
  the limits or not at all? Yes, in terms of moving it to
• 51:49
  production when we are developing, we are not doing a
• 51:53
  load testing at that point. Doesn't make much sense to me
• 51:56
  honestly to do a load test in a Jupiter notebook, right?
• 51:59
  I mean, you should be running an app, man, that that's what the
• 52:02
  need of the app there, right? So it's more like education and
• 52:07
  segregating. OK, we do it at this stage.
• 52:09
  And at that I would expect your, it, your notebook is shaped like
• 52:14
  an application at that point. It has all those, you know, 8 or
• 52:17
  10 things that we discussed standardising like proper
• 52:20
  logging or, you know, API endpoints, proper naming
• 52:23
  convention there. Then give it all the data and
• 52:27
  the resources. And when we do that testing, we
• 52:30
  have a better understanding how much resources we need, where it
• 52:33
  is hitting the limit. But at that point as well, we
• 52:37
  don't need the blind testing. You should have some something
• 52:40
  in my in your mind, right, that I want to reach like these
• 52:43
  number of user transactions per minute.
• 52:45
  I have to make it working for that.
• 52:47
  You can break any app if you want to write, but what value do
• 52:50
  you get out of that? Like if you say, OK, give me an
• 52:54
  app and I'll, you know, break it.
• 52:56
  But what do you get out of that? Your business goal is to make it
• 52:59
  happen for this many users. And I'll provide you the
• 53:02
  resources for that and for like day-to-day monitoring, getting
• 53:08
  patterns, anomaly detection, then you have proper
• 53:10
  observability. You'll be probably leaving that
• 53:12
  every week or every month. And you say, OK, I see a spike
• 53:16
  in traffic, we need to revamp our resources, right?
• 53:19
  So everything goes logical. And are you, when you're having
• 53:24
  that transition from whatever it was, whether it's LLM locally
• 53:31
  being run to now, it needs to be converted into an app or a
• 53:36
  Jupiter notebook, then now it needs to be more in the shape of
• 53:39
  an app. Have you found success trying to
• 53:43
  put that process onto the data scientists or what does it look
• 53:49
  like? What is the translation process
• 53:52
  look like? Yeah, so the beginning, it was a
• 53:55
  little bit of a struggle, but I I touched upon something like
• 53:59
  temperatizing sometime, you know, people try to push back
• 54:02
  because there's a too much unknown.
• 54:05
  If you provide them a scaffolding, OK, this is a
• 54:07
  skeleton. These are the files.
• 54:10
  So slowly and surely with combination of knowledge sharing
• 54:13
  session, people start feeling comfortable on then peering and
• 54:17
  peering like, you know, you hook up a person with someone who
• 54:19
  already know that. So you slowly spread that
• 54:22
  knowledge out. So it's not purely a technical
• 54:26
  solution. It's more like a timing and then
• 54:29
  the way we work. When you start on the project in
• 54:33
  Greenfield, it is easy because something new you are
• 54:36
  implementing. But if it is something already
• 54:39
  implemented and you want them to make changes, you may have more
• 54:43
  challenges there. Yeah, It comes back to that
• 54:48
  people, process and technology idea.
• 54:52
  Here's straight people and processes.
• 54:55
  No amount of technology is really going to help you on
• 54:58
  this, No. And so going back to the, the
• 55:00
  question I was going to ask you before I remembered what I
• 55:04
  wanted to talk about was really around the different areas that
• 55:10
  things can fail. And specifically we were talking
• 55:14
  about this RAG development and you were saying, do you have
• 55:17
  replicas of your vector databases?
• 55:20
  Do you have the LLM proxies? Do you have all the like?
• 55:26
  Make sure your secrets aren't on GitHub.
• 55:29
  What other ways have you seen or do you think about?
• 55:33
  What other vectors are you thinking about to make sure that
• 55:37
  your architecture is as robust and reliable as possible?
• 55:43
  That's a really great question. And especially with the LLMS and
• 55:47
  more and more cloud vendors over there providing all these models
• 55:51
  as APIs, right? Like now you're not limiting
• 55:56
  your traffic within your own centre.
• 55:57
  You're you're making a lot of hops and here and there, right?
• 56:01
  So it's again, you know, your personal opinion based on the
• 56:05
  knowledge that I have with different things.
• 56:08
  I feel like no security is one of the common things which gets
• 56:12
  overlooked. If you're creating new resources
• 56:17
  and your process is more like people oriented, you still rely
• 56:20
  on manual work. There are chances there could be
• 56:24
  human error, there would be some misses here and there, right?
• 56:28
  And that can lead to some big problem.
• 56:31
  But when it comes to the things they itself like what you have,
• 56:35
  for example, there's a newer model that come out there and
• 56:38
  you want to start using that, right?
• 56:43
  I think you need to virtually or logically segregate the task.
• 56:46
  Like experimentation is one thing, but when you take it out
• 56:50
  to like a next step, like from dev, test, prod or whatever
• 56:53
  you're moving, you need to have proper evaluation.
• 56:55
  If you're not doing that, you're bound to again or the risk of
• 56:58
  failure is high because you don't do the, the due diligence
• 57:02
  there because it's a big change in itself.
• 57:06
  And all the pieces in this workflow are important.
• 57:09
  I know we took example of RAG, we have the retrieval engine.
• 57:13
  Suppose, you know, tomorrow you want to switch to a new vector
• 57:16
  database. How do you test the the quality
• 57:20
  of that search results that you're getting One thing good
• 57:25
  and bad about the LLMS or anything that is coming across
• 57:29
  is people feel like OK, all I need to do is reinstall, import
• 57:33
  this and start using right. But behind the scene.
• 57:37
  These are some APA endpoints or maybe not so huge, but still
• 57:45
  another thing in your landscape, like a module that is providing
• 57:49
  some functionality. And if you're not doing AQA for
• 57:55
  each of those components and you start looking at just the final
• 57:59
  response, I think the risk of failure is very high there.
• 58:04
  Well, even if you go back to the classic ML example we took, like
• 58:07
  not keeping or keeping an eye on your data that you are so
• 58:11
  relaxed now, OK, my models are running fine, but that, you
• 58:15
  know, there's the data drift. You're not aware of that it's
• 58:19
  without proper observability. It'll take a lot of time for you
• 58:22
  because what happens over time, people start focusing more on
• 58:27
  the final outcome. I think it makes sense for the
• 58:31
  business, but if you are the one who is implementing or who are,
• 58:34
  who is researching, right, you should be aware of each and
• 58:38
  every step and and know how to do QA of each of that hop.
• 58:43
  Otherwise, you know, you're bound to get into an unpleasant
• 58:47
  situation where you say, OK, now I don't know, man.
• 58:49
  I have to go to square one and start analysing each of that
• 58:54
  thing. And you should have an incident
• 58:56
  response plan, like if something fails, right?
• 58:59
  How are you going to respond to that?
• 59:01
  It is not that I implemented something like generator by AI
• 59:04
  is relatively new. I know a lot of people have been
• 59:07
  experimenting, but if you roll out this app right, you may be
• 59:12
  seeing a different kind of situation as compared to a
• 59:15
  regular app or a regular model, right?
• 59:17
  They don't see the responses or your services unavailable.
• 59:20
  Your LLM starts behaving like no differently like it starts
• 59:26
  giving you the information that it is not supposed to or some
• 59:29
  error message. People start asking questions
• 59:32
  that you never thought they were going to ask.
• 59:34
  There's so many different ways they can not do it.
• 59:37
  That's a great point actually. One thing upon that right?
• 59:40
  Like use LLMS for asking those questions.
• 59:44
  You may consider having some kind of topic modelling on those
• 59:47
  questions as well. Like you know what people are
• 59:49
  talking about so that you are aware. 100% that's one of the
• 59:54
  most important pieces when it. Comes to just one more thing
• 59:57
  Dimitro's before it skips my mind that is about you know we
• 1:00:02
  talked about the libraries like people are pulling and using it
• 1:00:05
  but if you see the one liberty around that right we you may
• 1:00:10
  already remember something that happened around.
• 1:00:15
  Some zero day. I know, I know what you're
• 1:00:21
  talking about, but I can't remember exactly what you're
• 1:00:23
  talking about. But yeah, there was a
• 1:00:24
  vulnerability. Yeah.
• 1:00:25
  So I mean, if you don't have a proper vulnerability scanning
• 1:00:29
  for these libraries, right, you don't know what you're
• 1:00:31
  implementing, right? No one remembers over time.
• 1:00:34
  So I think right from the beginning you need to have all
• 1:00:37
  the wells and whistles when it comes to getting things to
• 1:00:40
  production. For experimentation, it might be
• 1:00:42
  OK, but I've seen situations where people start playing
• 1:00:46
  around and they later realise, OK, this was a licence thing
• 1:00:49
  you're not supposed to do commercially.
• 1:00:51
  Right now. It's an ugly situation because
• 1:00:54
  the developer is not aware and he created something that, you
• 1:00:57
  know, an organisation uses questionable.
• 1:01:00
  So those kind of things someone has to watch if a process is
• 1:01:04
  there. I know I've been talking
• 1:01:05
  process. I measured like 20 times but
• 1:01:09
  can't stress enough. But it's another vector to be
• 1:01:12
  thinking about and I appreciate that because it's not just,
• 1:01:15
  again, it, it doesn't just take into account how many database
• 1:01:23
  replicas do I have. That's a whole different type of
• 1:01:26
  vulnerability that you could potentially get yourself into
• 1:01:28
  some hot water for because you didn't check the licencing on
• 1:01:33
  that open source tool that you were using.
• 1:01:36
  And so being aware of all these different ways that things can
• 1:01:40
  go wrong is quite useful. And I wonder about just since
• 1:01:47
  we, you mentioned how you want to have an incident response
• 1:01:52
  plan and you also were talking about how, yeah, to bring down a
• 1:01:58
  service is always going to be possible if you throw enough
• 1:02:03
  crap at it. And what did you accomplish
• 1:02:07
  there by bringing it down? It's almost like, So what good
• 1:02:10
  job can we get back to work now? I, I think about stuff like we
• 1:02:17
  had on here someone who is into chaos engineering.
• 1:02:22
  And I imagine you've coming from DevOps, you've probably heard of
• 1:02:27
  the term. And for those who haven't, it's
• 1:02:31
  very much thinking about how can we try to really stress test or
• 1:02:37
  how can we have a hypothesis about if something hits the fan,
• 1:02:41
  what will happen from here? And then can we test our
• 1:02:45
  hypothesis and see if our system is reliable or not?
• 1:02:49
  If that thing that we like, we blow up something or we start
• 1:02:55
  sending 10X the amount of traffic or we start doing
• 1:02:58
  something like that. Have you thought about that in
• 1:03:01
  the machine learning world, and if so, how?
• 1:03:06
  It's a great point. It is is totally applicable
• 1:03:08
  because when you're talking about the resilient
• 1:03:10
  architecture, right, it cannot be compared without talking
• 1:03:13
  about the, the kilos engineering and but in in terms of the
• 1:03:19
  permitted models like before the LLMS word, we have been relying
• 1:03:23
  more on the regular regression testing because most of those
• 1:03:27
  things were either in a passive in France like like a chrono
• 1:03:31
  fashion. Is it and?
• 1:03:33
  Start opening those slowly and shortly to more and more APIs
• 1:03:38
  like creating the key serving etcetera.
• 1:03:42
  But it is important because as you are bringing in lot and lot
• 1:03:46
  of AAML to all these technologies or the existing
• 1:03:49
  solutions that you have to start thinking around those lines.
• 1:03:53
  I personally have not spent too much time but that is on my
• 1:03:56
  radar. Maybe could be a topic that we
• 1:03:58
  will discuss in some future broadcast.
• 1:04:01
  I would like that, yeah, 'cause I often think about it as how
• 1:04:06
  can you do this type of chaos engineering with the data as
• 1:04:09
  opposed to the service. So what happens?
• 1:04:13
  And it's, it's the same theory of I have a hypothesis and I'm
• 1:04:17
  going to see if my system is robust enough to be able to
• 1:04:21
  handle this. And so instead of saying what
• 1:04:24
  happens if we get 10X to traffic, what happens if all of
• 1:04:28
  a sudden this database goes down or if I all of a sudden start
• 1:04:33
  getting 10X the data going through?
• 1:04:35
  Are we able to handle those types of situations?
• 1:04:39
  And so it's bringing that same mentality, but doing it a little
• 1:04:42
  bit more. I guess it's a little bit more
• 1:04:43
  on the data engineering side of things that you want to just
• 1:04:49
  stress test your system and make sure that everything is is doing
• 1:04:54
  what it says it's going to do. But one thing I, I do like to
• 1:04:58
  mention here is, you know, availability of any resources in
• 1:05:03
  that whole landscape, right? Like a common point of failure
• 1:05:06
  we talked about. So when we talk about
• 1:05:08
  observability, right, we really rely on I have a logging and
• 1:05:12
  monitoring in place, right? I have a logging solution, but
• 1:05:15
  there are ways that you can set up some health cheques around
• 1:05:18
  that. Like even if you create a fast
• 1:05:20
  API app, right? You can create a health
• 1:05:21
  checkpoint on top of that or through that, you can hit your
• 1:05:25
  databases and get a health check.
• 1:05:27
  So if you, I have seen some of those like official document
• 1:05:32
  images, they always come with the health check, right?
• 1:05:36
  But when you create one, most of the time those things are
• 1:05:39
  overlooked. So I'm going to say that
• 1:05:43
  template thing again. So where with me create
• 1:05:46
  something that has, you know that make sure you have these
• 1:05:49
  health check and an example Docker file which has those
• 1:05:52
  things incorporated. But the point here is you need
• 1:05:57
  all those things right from the beginning so that you can reduce
• 1:06:01
  the risk. The risk is still going to be
• 1:06:04
  there. Like, you know, we are all from
• 1:06:06
  some data science and machine learning background.
• 1:06:09
  You can minimise the probability of something, but you cannot
• 1:06:12
  reduce it, right? It, it, it can still happen.
• 1:06:14
  It's more like if you are aware, OK, how I can be the first one
• 1:06:18
  to know because my team implemented, if I have proper
• 1:06:21
  thing in place, I might be able to have the cheques in there.
• 1:06:24
  Like, you know, if my database, like before making a database
• 1:06:27
  call or I, I know my UIAPA app that has cheques and balances in
• 1:06:33
  there. If my underlying APA is not
• 1:06:36
  performing, it'll say, OK, we are experiencing some issues,
• 1:06:40
  right? I'm not, I'm not going to get
• 1:06:41
  through the traffic, like I'm not going to get you to the next
• 1:06:45
  hop, right? Those are kind of a common
• 1:06:47
  development patterns. But The thing is if you don't
• 1:06:49
  have the proper health check monitoring in place like then
• 1:06:53
  you'll be doing some over engineering or getting into the
• 1:06:56
  issues. We are spending too much time on
• 1:06:59
  troubleshooting. Dude, this bitter, awesome
• 1:07:01
  conversation. I really appreciate you coming
• 1:07:03
  on here. Thank you.