Latest / Agentic Conversations (formally mlops.community) / Designing for Forward Compatibility in Gen AI // Rohit Agarwal // #189

Transcript

Designing for Forward Compatibility in Gen AI // Rohit Agarwal // #189 · Agentic Conversations (formally mlops.community)

• 0:00
  So I'm Rohit Agarwal and the founder of Port Kira I, which is
• 0:04
  an LLM OPS company and not too much of a coffee aficionado, but
• 0:09
  I think I prefer my South Indian philtre coffee.
• 0:12
  That's my drug. Welcome back everyone.
• 0:16
  This is Dimitrios, your host, and we are here for another ML
• 0:20
  OPS community podcast. Today we're talking with none
• 0:23
  other than my man Rohit, who has helped me so much along the way.
• 0:26
  As you will hear throughout this episode, he's currently working
• 0:30
  on Port Key. Before that he was doing stuff
• 0:33
  in the LM space, in the DevOps space.
• 0:36
  You can hear when he talks how fundamentally different he looks
• 0:41
  at working with LMS as a software engineer.
• 0:45
  He's coming with that. Background.
• 0:46
  He's coming with that, hey, things need to be production
• 0:49
  grade if we want to serve them to the public and how can we get
• 0:54
  there with LLM's? How can we make sure that we?
• 0:58
  Think about all of the different error scenarios or the worst
• 1:03
  case scenarios. Those edge cases you could say.
• 1:06
  I had a blast talking with him. He is.
• 1:10
  He gave me two ideas that I'm chewing on now, and I think it's
• 1:15
  fascinating. It's funny because he's like,
• 1:16
  you've probably thought about this, but I'm putting a name to
• 1:19
  it. One is the API Gateway, and I'm
• 1:23
  excited to hear what everyone else thinks.
• 1:25
  When it comes to the AI Gateway, it's almost like this.
• 1:29
  Since it gateway, for lack of a better word, I think they did a
• 1:32
  great job naming it where everything goes through that,
• 1:36
  whether it is a an LLM call or whether it's no.
• 1:42
  Basically all of your LLM calls are going through this gateway
• 1:45
  to make sure that it's being routed to the right model, but
• 1:49
  also that it's getting the right prompt.
• 1:51
  So he goes through the four pillars that an API Gateway
• 1:54
  should have. Took me a minute to wrap my mind
• 1:57
  around all four pillars, but I really liked.
• 2:00
  What he's thinking about when he thinks about the AI gateway.
• 2:05
  So let me know what you think about that.
• 2:08
  The other piece that I want to mention and call out it was
• 2:11
  towards the end. He talks about forward
• 2:13
  compatibility and. This idea that.
• 2:19
  People are a little bit scared to work on something because
• 2:23
  they don't know if in a month you're gonna get a new open
• 2:27
  source model that comes out that renders everything that you're
• 2:29
  working on obsolete or potentially open AI is going to
• 2:32
  come out with a fine tuning and then ah, shoot, like I don't
• 2:35
  need to be working on fine tuning llama anymore, whatever
• 2:38
  it may be, but that. Kind of FOMO of the future and
• 2:43
  what's going to happen can lead to analysis paralysis.
• 2:47
  And you get a little bit paralysed because you don't know
• 2:51
  what's going to happen and then you're not able to take
• 2:54
  advantage of what you have right now at your fingertips.
• 2:58
  So his idea of, hey, we need to make sure that when we're
• 3:01
  building our systems, we are thinking about the potential
• 3:05
  futures. And I really like how he talked
• 3:08
  about decoupling. Our code from these API calls
• 3:14
  and what actually gets out. And so this goes back to the AI
• 3:18
  gateway. Let me know what you think.
• 3:20
  As always, love hearing from you and if you would like to help us
• 3:26
  out. It would mean the world to me if
• 3:27
  you share this with just one friend and let them know that
• 3:32
  you think they would enjoy this episode.
• 3:35
  That's all for me. See you later. good to have you
• 3:45
  here, man. The same here Dimitri was.
• 3:48
  I think we've been chatting on and off for almost, what, four
• 3:51
  or five months. So you sort of followed our
• 3:54
  entire journey. So it's been fun.
• 3:57
  Yeah. And one thing about the chats
• 3:59
  that we have that I always appreciate from you is, hey,
• 4:02
  you're quick to help anything that I need in the community.
• 4:06
  You're always giving me a hand whether that was the first LM
• 4:10
  and production report that we did where I was fully blocked
• 4:13
  and I got on a call with you and you.
• 4:16
  Absolutely. Unblocked me.
• 4:17
  I cannot say how much of A heaven sent you were because you
• 4:21
  went through that data man and it was obvious that you've been
• 4:24
  playing around with data for ages.
• 4:27
  You were able to bucket everything.
• 4:29
  And a lot of the insights that I put in the report came from the
• 4:34
  Excel sheets that you shared with me.
• 4:36
  So I appreciate that a tonne. And then also just the stuff
• 4:40
  that you're working on is in the hot seat right now, right?
• 4:44
  Like where you sit in the stack, It's in this new.
• 4:47
  Piece of the AI world where I don't think we were necessarily
• 4:53
  dealing with it back in the day when it was the predictive ML as
• 4:59
  some people like to call it or traditional ML.
• 5:02
  And now with the LM's, we do have this whole new area that
• 5:08
  you're exploring and I love getting insights from you when
• 5:11
  we talk on how you're looking at this area.
• 5:15
  And so I think that's a good segue into.
• 5:19
  What have you been working on for the past?
• 5:23
  X amount of time. I know it's been like a year or
• 5:26
  a little under a year. How long you been working on
• 5:28
  this inside the, I've been working on this almost six
• 5:31
  months now, but been in this space, the generative space the
• 5:35
  last three years. So I think we started off
• 5:38
  building. So I was at Fresh Works.
• 5:41
  We're building a customer support product and we were
• 5:43
  trying to incorporate AI across multiple areas there.
• 5:49
  That's when I think I've worked on GPD 2A little bit with our
• 5:52
  data science and machine learning team.
• 5:54
  Then saw the entire transition to the earliest closed source,
• 5:59
  the closed model of GPD 3 which was Davinci.
• 6:02
  And then saw the next two years building a AI writing assistant
• 6:06
  with pepper type dot AI. And then in those two years, I
• 6:10
  think the maximum learning was that the technology is truly
• 6:14
  magical which we're seeing right now.
• 6:16
  There's just so many use cases across different verticals that
• 6:18
  are possible. But then if you were really to
• 6:22
  launch a production grade tool using the APIs that Open AI or
• 6:26
  anthropic or others provided, then you needed this entire
• 6:30
  tooling layer in place. And I think that becomes
• 6:33
  extremely, extremely critical at the moment.
• 6:37
  And what is the tooling layer, I mean what are they and what are
• 6:40
  you seeing in the space and especially like where you're
• 6:43
  playing and and you've created port key and like what does the
• 6:47
  landscape look like? Absolutely.
• 6:50
  I think if I were to almost and from the very beginning we've
• 6:54
  looked at it as let's draw Adls from the traditional engineering
• 6:58
  world. So you used to have servers,
• 7:01
  used to have databases, and then on top of these servers you
• 7:04
  would have some sort of a server management layer, a database
• 7:09
  management layer. On top of that you had this
• 7:11
  entire DevOps layer. So to manage your entire DevOps,
• 7:16
  there were lots of tools in different brackets and these
• 7:19
  kept on getting more and more detailed, more and more niche as
• 7:24
  developers went deeper into applications.
• 7:27
  Can I stop you right there? So you're saying like a?
• 7:32
  Something like you would have the cloud, you have your A WS,
• 7:34
  but then you also have your Kubernetes exactly sits on top
• 7:38
  of it. And then all of a sudden, on top
• 7:40
  of Kubernetes, you have Ansible that is for networking and
• 7:45
  you've got all this fun stuff that goes with Kubernetes.
• 7:48
  Yes, yes, exactly right. And then you have on top of
• 7:51
  Ansible, you had these other plugins, you had open source
• 7:55
  libraries. To just even do a blue-green
• 7:59
  deployment, companies would use something like Spinnaker.
• 8:02
  So this is a bunch of these tools that existed for you to
• 8:05
  just manage your existing systems and then you would build
• 8:08
  your application on top of it. And I think the main thing that
• 8:13
  enabled was people could build complex applications.
• 8:16
  So think about it, right, The pieces of the puzzle probably
• 8:20
  existed even in 2011, 2012 or even before that.
• 8:24
  But then all of these complex SAS applications coming out
• 8:26
  after that because a strong foundational DevOps tooling
• 8:30
  layer sort of evolved. Yeah, you didn't need to spend
• 8:35
  as much time monitoring servers on managing servers.
• 8:37
  You could rely on, you know, an AWS or GCP to do that.
• 8:41
  And then when you really thought about saying you I need to scale
• 8:44
  my application, do I need to do it manually?
• 8:47
  Kubernetes came in and helped out and then Kubernetes became a
• 8:50
  pain. And then you build Ansible on
• 8:52
  top. It's basically standing on the
• 8:53
  shoulders of giants. And then enabling these really
• 8:56
  complex use cases to be built on top of what essentially is
• 9:00
  still, you know, at the very basic edge, it's still a compute
• 9:06
  unit. But then how do you exploit that
• 9:09
  compute unit? Became this entire SAS BI would
• 9:12
  say. Uh-huh.
• 9:13
  Yeah. And so you're seeing that like
• 9:15
  you're drawing a parallel to right now, and the compute unit
• 9:19
  is no longer a cloud resource. It's a foundational model,
• 9:24
  exactly, yes. So you would almost imagine a
• 9:26
  compute so in place. And this is very loosely drawn.
• 9:29
  I'm sure the purists will vulture me on this, but let's
• 9:34
  take a compute unit as a foundational model.
• 9:36
  You want a quick way to look up these outputs of this compute
• 9:41
  unit, which is an embedding, which is where you need our
• 9:44
  vector database. So those two become your two
• 9:47
  core components. You have a model, and these
• 9:49
  could be multiple types of models, and then you have a
• 9:52
  vector database on top of this. You will need something to
• 9:56
  really manage the deployment, etc of your own model.
• 10:00
  If you're doing or if you're just using an open air
• 10:03
  anthropic, then you don't need it, you just need to use the
• 10:06
  APIs, right? So then there is this one single
• 10:09
  layer of management deployment, tuning your own specific model,
• 10:15
  which is optional obviously. Now on top of this stack which
• 10:19
  is almost readily available today to all developers since
• 10:22
  this new layer which is evolving, right.
• 10:25
  So in the traditional mill world and I hit to call it traditional
• 10:28
  because it's probably just three years, four years old, but then
• 10:31
  existed the ML OPS layer which were all of these amazing
• 10:34
  companies like Wits and Biases and Arias and Mosaic etcetera,
• 10:38
  Databricks. Now there's a new layer evolving
• 10:41
  for generative applications which is more the and you know I
• 10:44
  think the jury is still out on is it LLM OPS or FMO OPS.
• 10:48
  But yes, there is this entire layer that needs to evolve that
• 10:52
  enables companies to build beyond chat with your PDF kind
• 10:56
  of application. So I think not dissing on chat
• 10:58
  with your PDF, but then there's so much more you can do with
• 11:01
  foundational models that hasn't been explored till now.
• 11:05
  So you have this entire OPS layer that needs to come up.
• 11:09
  OK, so. Yes.
• 11:13
  Exactly that the idea of. Drawing the parallel between.
• 11:20
  What we were doing in. Essentially the DevOps world or
• 11:24
  going from cloud to Kubernetes to a much more sophisticated
• 11:28
  Kubernetes ecosystem. And then that allows people to
• 11:35
  not spend time fiddling with the resources and spend more time
• 11:39
  building the products that are bulletproof and they're not
• 11:42
  going to go down every other day because of scaling issues or
• 11:46
  whatnot. This is exactly what you're
• 11:49
  saying. Right now we've got GPT and
• 11:52
  we've got these wrappers that are coming out that you can have
• 11:56
  your chat with your data, which is a really cool hello world.
• 11:59
  But you're looking forward to where are we going from here?
• 12:03
  What's the next step that we can create much more complicated
• 12:07
  applications with this that we do not have to worry about?
• 12:11
  Hey, is this going to go down? We have these production ready
• 12:14
  things and there are still quite a lot of hang ups.
• 12:18
  When it comes to using. ChatGPT at a high scale, right?
• 12:25
  Or even any of these models. And so if you're using open
• 12:30
  source, you have your own can of worms that you got to get into
• 12:34
  and maybe we can go into that. Like what are some of these
• 12:38
  hangups that you've seen and then how do you feel like those
• 12:41
  are being combated now? So if you look at it in the
• 12:45
  closed source models, if you're using Open AI, the the biggest
• 12:50
  challenge almost everybody faces the error rates.
• 12:53
  In fact in our benchmarks across and we do about 3 million API
• 12:57
  requests a day, we're doing about a 5% error rate on the
• 13:01
  opening I based API itself, which is which is crazy because
• 13:05
  five out of 100 requests are going to fail and then you need
• 13:09
  to build the systems in place to be able for it to either retry
• 13:13
  or fall back. And there's so many other
• 13:14
  options that you need to do and it's madness, right?
• 13:17
  Who? Which engineer in 2023 is
• 13:19
  expecting a 5% error rate, But that's fair.
• 13:22
  I mean, you're you're. Interested is so much.
• 13:25
  Yeah Oh my That's outrageous. Exactly.
• 13:30
  And I think people are now beginning to deploy tools to
• 13:34
  production and then users come back and say, hey, it didn't
• 13:37
  work for me because five out of 100 users are having a bad
• 13:40
  experience every single time. It's just yeah, it's insane.
• 13:43
  And if you look at the open source models, I think, and I've
• 13:48
  heard, I've been to so many conferences these last two
• 13:50
  months, almost every enterprise is like his scaling.
• 13:53
  This is really hard. The compute resources it starts
• 13:56
  to utilise is just crazy and unless you have very solid
• 14:01
  defined use cases and a consistent traffic pattern, it's
• 14:06
  very hard to handle the spikes and the troughs, right?
• 14:09
  I think it's almost akin to saying let me host my own server
• 14:12
  farm in today's world without any tools available.
• 14:15
  And I think it's the exact same problems of here.
• 14:18
  I have spiky traffic, my model goes down.
• 14:21
  I don't think I'm getting the best performance.
• 14:23
  I don't think it's more most efficient.
• 14:26
  It's all of those same problems which I think nobody wants to
• 14:29
  deal with. OI think on both ends.
• 14:32
  At the very base level, just deploying a model reliably is a
• 14:38
  challenge. And then you go above a little
• 14:41
  bit more saying, OK, how am I supposed to cash this out?
• 14:45
  Because caching doesn't work the same way as it did in the older
• 14:48
  world. So I need to build a new method
• 14:50
  of caching and that's where semantic caching came up or I
• 14:55
  have multiple models that I need to keep switching between.
• 14:58
  So how do I build a quick model router?
• 15:01
  And so, you know, problems keep stacking up.
• 15:04
  And I think in my brief stint, almost in five months, it's been
• 15:08
  that once you solve a problem, the next one emerges and the
• 15:12
  next one emerges. So it's just a string of things
• 15:15
  that need to be sort of solved for you to build a solid
• 15:19
  production level application. Yeah.
• 15:23
  Yeah. The, the bottlenecks are real
• 15:25
  and you're facing them. You're kind of forging this new
• 15:28
  path right now. And so I I would love to hear as
• 15:33
  you're going through these 3 million requests per day or as
• 15:37
  you're working with customers. You know you spent a lot of time
• 15:41
  on the report with me and we have a lot of use cases that we
• 15:47
  highlighted from the survey, but that was now dare I say three
• 15:52
  months ago and potentially all that data is stale considering
• 15:56
  how fast things move, right. But I would love to hear what
• 15:59
  are you seeing in the wild as far as use cases go as far as
• 16:04
  like what customers are doing and how is the?
• 16:10
  Use case as you're like breaking through these bottlenecks.
• 16:13
  How is the complexity of the apps that people are able to
• 16:18
  build? Evolving, right?
• 16:21
  I think so. I always look at it from 2
• 16:25
  frames of mind, right. So you have the introduction
• 16:28
  application. So they've been people have
• 16:30
  experimented for a while, they've figured out the best
• 16:33
  parts, they figured out value and then we're seeing stuff in
• 16:36
  production. And then there's still things
• 16:38
  that are in the experimentation phase.
• 16:39
  So let's probably talk about both a little bit, right?
• 16:42
  So in the production phases, I have seen three kinds of
• 16:46
  applications and it's almost everything that I've seen fits
• 16:50
  into these three applications, right?
• 16:52
  One is you either have a summary summarization type application,
• 16:55
  which is here is an active PDF. Give me a summary.
• 16:59
  Here's my financial report. Give me a summary, Here's a
• 17:02
  YouTube video. Give me a summary.
• 17:03
  So those are I think very basic examples.
• 17:05
  But then going deeper into meeting summaries, convert the
• 17:09
  summary into an action item list for me.
• 17:12
  Can you also put it into my To Do List?
• 17:14
  So those are things that are becoming Philips, you know,
• 17:17
  summarization is a concept really popular.
• 17:20
  The second one is this Q&A. And I think this is the most
• 17:23
  popular use case right now. So if you look at RAG, RAG is
• 17:27
  probably the defining production feature in almost every software
• 17:31
  application that exists today. So be it.
• 17:33
  Customer support, data inquiry, sales, internal IT use cases,
• 17:40
  almost every software that exists is just hustling to add
• 17:44
  rag to their applications. And I think the key advantage
• 17:48
  people are seeing is now you can have.
• 17:51
  There was always this utopian world shown by chatbot companies
• 17:55
  earlier on saying that can I just chat with all of this
• 17:58
  information, all of this knowledge I have stored and for
• 18:01
  the first time it's becoming possible and to a very large
• 18:04
  extent accurately possible. This is also the one use case
• 18:08
  that is not prone to hallucination too much because
• 18:11
  if you give all of the context to a foundational model then end
• 18:15
  U returning really great answers.
• 18:17
  So that's where I've seen people find a lot of ROI and people are
• 18:22
  willing to pay for tools like this as well.
• 18:25
  So I think that's the Q and A use case.
• 18:27
  Rag use case is probably the most popular right now.
• 18:31
  And the third kind of use cases, I almost categorise them as
• 18:34
  generative use cases, which is what the word even started,
• 18:38
  right? So let me a blog post brainstorm
• 18:41
  the pros and cons of this decision.
• 18:44
  Here's an e-mail. Can you change the dawn?
• 18:47
  Can you fix the grammar? I think 1 surprising thing which
• 18:51
  is I'd love to tell you is that among all of the customers I
• 18:54
  talked to fix my grammar or fix my tool seemed to be the most
• 18:58
  useful use cases. And you would imagine here these
• 19:00
  are like super primary and Grammarly would do it etc.
• 19:04
  But I think these two use cases see the highest usage amongst.
• 19:10
  You know, if you launch 10 features and you have these two,
• 19:12
  these two will probably get used the most, which is very
• 19:15
  interesting. Yeah, And especially I think
• 19:18
  that works so nicely for people who English isn't their first
• 19:22
  language. Yeah, And so you can feel more
• 19:24
  confident in what you're doing. And knowing that you've got the
• 19:29
  right tone. For me, it's horrible.
• 19:32
  I should probably use it because my tone is so bad all the time.
• 19:37
  But I just am informal. A step below informal, always.
• 19:42
  You can never expect any formalities out of me.
• 19:44
  And if I do. Say anything formally, like you
• 19:49
  should probably report it as somebody's hijacked my account,
• 19:52
  because it doesn't really work. But I I do see the value of
• 19:57
  that, the brainstorming, the talking with the different
• 20:00
  things. I you did gloss over this, but.
• 20:04
  It's the idea of agents and hey, take the take the meeting notes
• 20:10
  and then convert it into A to do list.
• 20:13
  Put it on to my To Do List type thing.
• 20:16
  It feels like agents are still a little bit flaky.
• 20:20
  I know there's so much energy going into it.
• 20:24
  How have you been experiencing them?
• 20:27
  What have you been working with to try and like help that?
• 20:31
  Or do you not even touch it like well explain to me your relation
• 20:35
  with agents? Yeah, I think agents is a
• 20:38
  another. Phenomenal tech.
• 20:41
  I wouldn't even call it tech, right.
• 20:42
  It's more a framework right now. So this framework is almost at
• 20:45
  the same accuracy level as GP3 was maybe in January 2021.
• 20:50
  So it's about 70% accurate because essentially you're
• 20:54
  propagating the error down every step.
• 20:56
  So the more steps in the branches the agent takes, the
• 21:00
  more error rate it's prone to. So I think that's the reason we
• 21:04
  haven't seen agents in production almost at all.
• 21:07
  But it's one of those tech where a lot of experimentation is
• 21:10
  happening because everybody's expecting these individual
• 21:13
  pieces to come down to, I mean the accuracy of the individual
• 21:17
  pieces to become really high. So if you're nearing though, if
• 21:22
  all of the accuracy of the individual individual steps
• 21:25
  trends to be 99.9% and more. Then you can have an agent
• 21:29
  successfully perform tasks. In areas where accuracy that
• 21:33
  much is more important. So you know here's a code piece
• 21:36
  right a unit test case. Try it.
• 21:38
  If it fails, try again where you can have a recurring use case.
• 21:42
  We are starting to see agents come up.
• 21:44
  I think code interpreter is probably the best example of an
• 21:47
  agent an open adjust. Releasing it out to the world
• 21:51
  gave people a lot of ideas on how you can really play with
• 21:54
  agents in production. Saying we're going to start to
• 21:57
  see a lot more of agents, a lot more of chains, a lot more of
• 22:00
  merging factual data APIs with probabilistic data nodes And if
• 22:07
  you see I think. Both Lamma index and Lang chain
• 22:10
  which are probably the primary orchestration libraries today
• 22:13
  they are focusing on drag. In the direction of agents, so
• 22:19
  they like hey today it's RAG but very soon it's going to be
• 22:22
  agents. So you sort of every company is
• 22:24
  also building up to that. So it's like information
• 22:27
  retrieval, just Q&A and actions on top.
• 22:30
  So you have Q&A with actions and then this if you can do it multi
• 22:34
  step then it's an agent implementation in production.
• 22:38
  I think that will be the path for.
• 22:40
  At least enterprises, companies that are really playing with
• 22:43
  this with real world users and not just hobbyists who are
• 22:48
  experimenting with with this tech today.
• 22:51
  Yeah. I wonder, man, I wonder where we
• 22:54
  are in the like S curve of our agents going to be able to get
• 23:00
  to that point where we can reliably use them.
• 23:03
  And if so, when? Is it a few months out or is it
• 23:07
  a few years out or is it a decade out?
• 23:09
  You know, that's the big question because everything that
• 23:13
  you're saying is, is very true. If you're getting this 5%, if we
• 23:17
  go back to what you're getting on those 3 million API calls and
• 23:21
  you're at 5% as soon as you want to do like 2-3 steps, that 5% is
• 23:27
  much higher than 5% error rate. And so you can't predictably
• 23:32
  rely on anything. And I don't know, do you see
• 23:35
  that 5% going to .01% anytime soon?
• 23:40
  That's the big question. I think it is going there and
• 23:45
  with the launch of fine tuning in, this is all very recent
• 23:48
  right now it's almost a new architecture stack is evolving
• 23:53
  which is. Let's not use a generic model to
• 23:56
  do everything. Let's try to train very specific
• 24:00
  models on very specific tasks, which will then be high
• 24:04
  accuracy, low latency. And I think this is something
• 24:07
  that I'm very excited about like will there be a worker be queen
• 24:11
  bee concept where you have a queen bee that orchestrates
• 24:14
  these smaller? Extremely high accuracy models
• 24:18
  which are cheaper to run but then only do one task.
• 24:22
  And then if you have this orchestration worked out really
• 24:25
  well. Then is when the agent comes
• 24:28
  out. So I think it's in the.
• 24:30
  In the evolution of when an agent becomes useful, you need
• 24:34
  this base infra to be sorted out first.
• 24:37
  Yeah, it's like I'm having deja vu, man.
• 24:40
  All of a sudden we're back to small models.
• 24:42
  What the hell happened? We went through the boom and
• 24:46
  now, Oh yeah, the these large language models are cool, but
• 24:49
  small large language models are actually where it's at.
• 24:54
  That's true. And you know, if you think about
• 24:56
  it, these small language models might still not be small.
• 24:59
  They might still be individual, you know, 20 billion, 70 billion
• 25:03
  models, which are still fairly large, but because we've seen,
• 25:08
  you know, 1 trillion parameter models, so a 70 billion model
• 25:12
  feels smaller. It's also much faster, cheaper
• 25:14
  to maintain, but then you could probably have.
• 25:17
  Ten of these models fine-tuned for really specific tasks and
• 25:21
  they outperform the best model any day.
• 25:24
  In fact, there are studies that came out that said.
• 25:28
  I think the number one use case for open source models today
• 25:31
  ends up being. Ohh what's this I mean?
• 25:36
  N to SQL. And so many companies have now
• 25:39
  tried fine tuning Lamma 2 on an Alto SQL, saying that hey, it
• 25:44
  performs better than GBD 4 or any other analytic SQL model
• 25:48
  that's out there. Which is fantastic because now
• 25:51
  individual companies can train their natural language queries
• 25:55
  to SQL on their own data. On a really small, fast open
• 25:59
  source model and get accuracy levels that are nearing that
• 26:04
  9899% mark, which is when agents now become super useful, right?
• 26:08
  Because then you can say. The code is my query converted
• 26:12
  to SQL reliably get the data from my DB reliably.
• 26:16
  And then convert it to an inside, that's useful for me.
• 26:19
  So I think all these steps, the individual accuracy levels,
• 26:23
  there are multiple I think research wings at play to
• 26:26
  improve the accuracy of each of the individual steps.
• 26:29
  I would, I would say it's probably.
• 26:32
  I think six months out. But given that everything moves
• 26:35
  so quickly, it might be sooner. Ohh, I like it.
• 26:40
  I like you given a date. That's bold.
• 26:42
  Man. That is a a lot more than I
• 26:44
  would go off and say. But let's talk a little bit more
• 26:48
  about just a lay of the land as far as where you're sitting with
• 26:53
  Porky and what you're doing, what you're seeing as far as the
• 26:59
  other tools around you. And below you in the stack, I'm
• 27:04
  going back to that market map as the reference guide as what we
• 27:10
  created. And again, there's the
• 27:12
  foundational model, there's the vector database, there's the
• 27:14
  sticks or the orchestration layer like the LUMA index or the
• 27:19
  link chain or Haystack, all that fun stuff.
• 27:22
  And then there's the other piece where I think you play and so
• 27:26
  give us the breakdown on that. Yeah, absolutely.
• 27:29
  So I think. We decided from from our
• 27:32
  experience, the thing we're good at is when you want to take up A
• 27:36
  to production. So you've done experimenting,
• 27:39
  taking into production, and now you want to really find value
• 27:42
  once you go to production? Is where you need an op stack
• 27:46
  that helps you with four major things, right?
• 27:49
  And this is what we've come up with so far.
• 27:50
  I'm sure this changes every month, but right now we say
• 27:53
  first you need observability. You need visibility into how
• 27:56
  your systems really performing. So you need the classic logging,
• 28:00
  tracing, monitoring and that's the observability stack that's
• 28:04
  really important. The second piece is, are you
• 28:08
  also saying though observability of the data too or are you not
• 28:11
  going that far? You're saying, hey DevOps
• 28:13
  observability like APM type? Things even of the data, yeah.
• 28:17
  So I think in this case I think they become merged because you
• 28:21
  want to observe the latency, the cost and the accuracy.
• 28:25
  So I think in previous DeVos worlds, latency and cost would
• 28:29
  almost imply accuracy because the the API calls always truthy,
• 28:34
  so if it returns the 200 means it's a positive response.
• 28:38
  So in fact fun story, sometime back we were discussing that
• 28:41
  maybe we need to invent a new HTTP code which is a 224 or 25.
• 28:46
  I don't know what it is but it's basically saying the API
• 28:48
  returned successfully but the data may be false.
• 28:52
  That's a new HTTP code that almost needs to exist now
• 28:55
  because. APIs no longer adjust successor
• 28:58
  error. It's everything in between, yes.
• 29:01
  Oh, interesting. But just going back to a little
• 29:05
  bit, it's not necessarily looking at the data streams or
• 29:11
  the, you're not observing the data flows, it's more just the
• 29:17
  data and how it's in there, yes, on the prompt specifically.
• 29:21
  That's true. That's true.
• 29:22
  Yes. Yes.
• 29:22
  OK. So once the model is ready, so I
• 29:25
  think there will be observability.
• 29:26
  When you're deploying a model and you want to measure the, you
• 29:30
  want to observe a model that's very different.
• 29:33
  This is more about observing all of the calls that you're making
• 29:36
  to the model and then making sure they're all, what's the
• 29:39
  cost, latency and price of it. Yes.
• 29:42
  Yeah. I think it's observability.
• 29:44
  Very different layer. It's not observability on the
• 29:47
  base Foundational Model itself. OK, cool.
• 29:50
  So that's one. The second piece is you need a
• 29:53
  way to be able to connect to multiple of these models and try
• 29:57
  them out as simply as possible. So when a Mistral launches and
• 30:02
  you're doing LAMA, you shouldn't have to completely rewrite your
• 30:05
  systems to see. OK, let me now try, Mr But can
• 30:09
  you do a 5% AB test with Mistral?
• 30:12
  Tested out and see if it really works for your audience or not
• 30:15
  with the same prompts or throw it in new prompts or what does
• 30:19
  that look like? Depends, right?
• 30:21
  With some of the cases you could try the same prompt itself.
• 30:24
  In some cases you have to change the format of the prompt and
• 30:27
  some case you have to change the entire structure of the prompt
• 30:30
  is also to rewrite the prompt. So I think in all three cases
• 30:32
  you need a system that allows you to do that transition from
• 30:36
  one to the other, test out, rollback these all systems,
• 30:40
  right. So that's the second piece.
• 30:42
  Where we calling it? That's the AI gateway.
• 30:46
  Which enables you to do all of this, Yeah.
• 30:50
  Hey everyone, my name is Aparna, founder of Verizon, and the best
• 30:53
  way to stay up to date with ML OPS is by subscribing to this
• 30:56
  podcast. The third piece is the prompt
• 31:03
  management and experimentation. I don't know how far this goes,
• 31:06
  but today almost every company that's interacting with alms,
• 31:10
  they have a prompt management stack.
• 31:12
  So how do you define your prompt library?
• 31:14
  How do you define experiments on them?
• 31:16
  Etc. The last piece of security and
• 31:19
  compliance, which is how do you make sure none of your data is
• 31:22
  getting leaked? You have so much important data
• 31:26
  that's being put in the prompt or in the vector database.
• 31:29
  How is it not leaked? How do you stay compliant with
• 31:32
  your sock too, your GPRS? So that entire stack, right.
• 31:36
  And this is what we're calling the complete full stack LLM OPS
• 31:41
  and that's the ecosystem that's evolving.
• 31:43
  And if I were to tell you where it's placed, you have
• 31:46
  foundational models if you're doing.
• 31:48
  And so do you have the deployment layers.
• 31:51
  Then you have the orchestration layer where you're writing code
• 31:53
  essentially on top of it. And this orchestration layer and
• 31:57
  the LLM oops layer now probably need to work together.
• 32:02
  O which is where you've seen a lot of like Lang chain is doing
• 32:05
  Lang Smith which is yeah, coherent, right?
• 32:08
  You have the orchestration layer and then you have the LLM OPS
• 32:11
  layer. Similarly, I think a lot more
• 32:14
  needs to be done in the LLM OPS layer, which is where port key
• 32:16
  placed. Yeah?
• 32:18
  So if I'm understanding this correctly, and I like these four
• 32:23
  pillars that you're talking about, hey, we need
• 32:25
  observability. We need to understand is it
• 32:27
  working and if it is working, there's almost that element of
• 32:32
  evaluation comes into it. Yeah.
• 32:34
  How well is it working? It's not just yes or no.
• 32:38
  There's a big grey area here. And so we have to figure out
• 32:41
  what that entails. That's the first pillar, the
• 32:45
  second pillar you mentioned. And you're going to have to
• 32:48
  remind me because I want to go through these and I remember the
• 32:52
  1st and the last one, of course, that's how the mind works.
• 32:54
  I think the last one, let me see if I if I got that right.
• 32:57
  But the last one is something along the lines of where, where?
• 33:05
  OK, now I don't even remember the last time.
• 33:08
  Yeah, I think last one was security and compliance.
• 33:10
  There we go. That one I I knew it was
• 33:12
  important, especially because my friend Diego has been hitting me
• 33:16
  up with all of this fun stuff where they're talking about how,
• 33:21
  you know, panda packages or just regular like and packages that
• 33:29
  normal people will be using have been wrought with spyware or
• 33:33
  whatnot and how the compliance is a big thing that gets
• 33:37
  overlooked. And the other piece to that is
• 33:41
  exactly what you're saying on that you have all these GDPR
• 33:44
  compliance, whatever. But then you have, hey, if I'm
• 33:47
  going to be using one of these models, how do I make sure that
• 33:52
  it's not spitting out this data later on?
• 33:55
  And how do I make sure that the right people have access to the
• 33:59
  right data and not anymore or any less?
• 34:03
  And it feels like with these large language models, it's a
• 34:07
  bit of a crapshoot right now. So that's a hard problem that I
• 34:10
  would love to dig into. But let me hear the two in the
• 34:12
  middle too. What were the other two pillars?
• 34:14
  That you had was so one is prompt management, so prompt
• 34:17
  management and experimentation, and the other is the AI gateway.
• 34:22
  So which is basically. Like flexing the multiple LMS
• 34:25
  and making sure you are the best of you know you can fork out to
• 34:30
  more Lums than one and then even keep trying out Ulms coming back
• 34:36
  etc. And the API Gateway sounds like
• 34:39
  it could get really expensive really fast.
• 34:44
  So the AI gateway is almost something we're saying becomes
• 34:47
  the central piece of any LLM product.
• 34:51
  So if you're building an application that uses SLMS, at
• 34:54
  some point in time you will have to build an AI gateway which is
• 34:57
  almost your gateway to the entire adlem world.
• 35:02
  And this can be a really fast proxy which introduces really
• 35:07
  low latency, scales very well, but then allows you to load
• 35:11
  balance, setup fallbacks, do testing.
• 35:14
  Connects to your monitoring stack.
• 35:18
  Make sure there's a firewall attached to it so you know all
• 35:20
  of those things. Again, traditionally going from
• 35:23
  if I talk about traditional engineering, engine X was almost
• 35:27
  a requirement when you went to production.
• 35:29
  Or going API gateway or a cloud flare.
• 35:32
  These were all things that were almost marked as checklist items
• 35:36
  saying if you need to go to production, have a firewall in
• 35:39
  place, have an engine X server in place.
• 35:42
  This is where an AI gateway will also come to place where if you
• 35:45
  go to production with an AI model.
• 35:47
  Having the AI Gateway in place, yeah, yeah, that makes a tonne
• 35:53
  of sense. So that AI gateway just points
• 35:57
  the different prompts or the different requests to the right
• 36:02
  models and it allows you to quickly experiment, rapidly
• 36:07
  iterate depending on whatever models come out or the the model
• 36:11
  of the month basically or if you're having these fall over or
• 36:14
  failover. I can't remember how you put it
• 36:17
  those requests. Then you're getting a bit of
• 36:21
  redundancy and you're getting that back up to know that,
• 36:24
  alright, if I am getting the whatever timeout error from Open
• 36:29
  AI, yeah, my service is not going to be affected because we
• 36:33
  can now hit Claude or we can go and we can hit our Llama model
• 36:39
  that we have up and running. Absolutely, exactly.
• 36:44
  In fact, most people what I've seen is they would have a load
• 36:48
  balancer set up between Open API and Azure Open AI because
• 36:51
  they're essentially the same model and you get more rate
• 36:54
  limits there than or you've set of fallbacks from say a more
• 36:58
  costlier model to achieve a model whenever for some reason
• 37:02
  the costlier model is off. So that is where a lot of this
• 37:05
  comes into play. I think more than anything we
• 37:08
  talked about the 5% error rate. We're now seeing that if you put
• 37:12
  a strong evaluation engine in place and set up these retries,
• 37:16
  load balancing and fallbacks. With a solid AI gateway in
• 37:20
  place, you reduce that error rate almost 2.02%, which is
• 37:24
  phenomenal, right from 5% to 102%.
• 37:28
  And this is just data we were crunching and I think we're
• 37:31
  going to release all of this data out in the public saying
• 37:34
  and it's not just Portia and you could do this yourself, but then
• 37:38
  you can reduce that 5% error rate significantly which trends
• 37:42
  to the point you were making earlier which is how do you make
• 37:45
  agents successful is when you have a you have something like
• 37:48
  this in place. You have strong evaluations in
• 37:50
  place and then you can start to trend that error rate near 0 and
• 37:56
  make agents really, really successful.
• 37:59
  Well, that that was one of the questions that I wanted to get
• 38:03
  to was you mentioned the worker bee and the queen bee type model
• 38:08
  here. Is this a bit of foreshadowing
• 38:12
  for that? Do you feel like, hey, with the
• 38:15
  AI gateway, we're going to be able to first hit the big model
• 38:20
  and make sure that we know which one it is and then that big
• 38:23
  model can use the AI gateway, it can leverage it and say, boom,
• 38:27
  now we know which model to go to net?
• 38:29
  Exactly, yes, and I think this will be on a on a triad of cost,
• 38:33
  accuracy and latency, because in a lot of use cases you can't
• 38:38
  have a lot of latency as well, so you might want.
• 38:40
  It's OK with a little less accuracy, but I want it to not
• 38:44
  cost me so much or not be so not take so much time.
• 38:49
  So it's going to be like a interesting mix of all of these
• 38:52
  three which gives you the best output and the best user
• 38:55
  experience. It may not always be about
• 38:57
  accuracy, yeah, yeah. What are you optimising for?
• 39:01
  Yeah, and recognising that you can have it fast, cheap or easy,
• 39:08
  but you can't have all three. Yeah, I mean there's just not,
• 39:12
  yeah, there's those trade-offs And so you know that.
• 39:16
  Alright, well, if it's coming from this request, we're going
• 39:19
  to be optimising for speed. If it's coming from this
• 39:21
  request, we're going to be optimising for accuracy
• 39:23
  etcetera, etcetera. So that makes complete sense.
• 39:27
  Let's dive into this security and compliance aspect.
• 39:31
  What were some of the hard problems that you solved with
• 39:35
  the security and compliance and how did you go about that?
• 39:39
  Sure. I wouldn't say we've solved a
• 39:41
  lot of problems because there's just a tonne of things to be
• 39:44
  done in security and compliance. And I think we might just end up
• 39:47
  partnering with some of these companies who are only focused
• 39:50
  on security compliance. We've done the basics, so you
• 39:54
  need to have. You know Pi anonymizer at your
• 39:59
  end. You need to make sure that none
• 40:01
  of the sensitive data crosses your boundary layer.
• 40:04
  So how do you do that? So we implemented a π
• 40:07
  anonymization scheme, and that's extremely hard because you have
• 40:10
  to implement that in a compliant manner.
• 40:12
  Again, you can always call GPD 3 and ask it to anonymize.
• 40:15
  It will do that, but then implementing the anonymizer in a
• 40:19
  compliant manner is the hard problem.
• 40:21
  Yeah, that's not sock to. Exactly there is not socked to
• 40:26
  compliant. Exactly so that is, that's that.
• 40:29
  Then how do you solve issues around data residency?
• 40:32
  So if you need to be GDPR, but opener is only in the US, then
• 40:36
  how do you really prepare for data residency?
• 40:40
  Which means can you use Open AI as a stateless server which
• 40:44
  isn't storing any information and the information is always
• 40:47
  stored in the new server. So the data does go out, but
• 40:51
  none of the sensitive data goes out.
• 40:54
  No PII ever goes out, it just processes.
• 40:57
  A bunch of text comes back and then you're able to derive value
• 41:01
  from it at the other end. So.
• 41:04
  I think those would be the two key pieces, the third being
• 41:07
  stopping DDoS attacks. So we will implement it some of
• 41:11
  Cloudflare solutions so that you don't get Ddosed.
• 41:14
  The problem with DDoS that we've heard with customers is that.
• 41:17
  A lot of times their key gets compromised because of a GitHub
• 41:21
  repo or something, and then somebody just makes a lot of
• 41:24
  calls and you know. It's a direct cost problem to
• 41:29
  you because now you've probably spent $1000 in in a day which
• 41:34
  you really didn't intend to because somebody got a hand of
• 41:36
  your key. O we've launched, you know, we
• 41:39
  launched something called Virtual Keys that allows you to
• 41:42
  store your open air anthropic keys in a secret vault.
• 41:46
  We will give you these virtual keys almost like virtual credit
• 41:49
  cards setup. Spend limits on it distributed
• 41:52
  to developers. You could even do a hackathon,
• 41:54
  give out 10 credits to everybody.
• 41:57
  Nice. And then?
• 41:59
  The key management happens at the port killer, and it's so
• 42:03
  much simpler because you can always decommission a key, you
• 42:05
  can remove a key, replace a key. Rotation is easier because now
• 42:09
  we've sort of built an entire layer for it.
• 42:12
  So I think those are the three things that are that's something
• 42:15
  that we've done, but I can see a lot of applications around.
• 42:19
  And I'd love to talk to startups about how do you do access
• 42:22
  control and permissioning, especially with vector
• 42:25
  databases. I think today no vector database
• 42:27
  really looks at permissioning really closely.
• 42:30
  So data leakage is a sure short possibility.
• 42:34
  I can essentially do a prompt engineering.
• 42:39
  To get data from a vector database I wasn't supposed to
• 42:42
  retrieve. So how do you put those cheques
• 42:45
  and balances in place? So my permissions, I mean, I
• 42:48
  don't exploit my permissions. So those are things that are
• 42:51
  becoming important. So you know, leakage is a
• 42:53
  problem. Prompt injections are a problem.
• 42:57
  So there's I think a bunch of things that companies are really
• 43:00
  interesting problems that companies are solving.
• 43:02
  We've probably done the basics, which absolutely need to be
• 43:05
  done, but we're looking to partner with some of these
• 43:08
  companies to do more. Yeah.
• 43:12
  It's funny because it reminds me of a lot of the talk and about a
• 43:16
  year, year and a half ago, the idea of the data mesh got really
• 43:22
  popular. Yep.
• 43:23
  And treating data as products. And that was a way to combat the
• 43:29
  these data access problems. And you had the consumers and
• 43:32
  you had the producers, et cetera, et cetera.
• 43:35
  Yeah. And I've heard horror stories of
• 43:38
  people joining an enterprise and not having the right access for
• 43:43
  data or just for systems, even six months after they've been
• 43:48
  working at the enterprise. And so taking all that into
• 43:53
  account and then recognising now we're just going to kind of let
• 43:58
  that data loose with the LM, it could be a recipe for disaster,
• 44:03
  right? Exactly.
• 44:05
  Exactly. Imagine you take your
• 44:07
  confidential internal data and fine tune a model on it just to
• 44:11
  get great benchmark results. You do?
• 44:14
  But then I am probably a new joinee and I am now able to
• 44:17
  access every information about the company because the model
• 44:20
  already has seen all that information.
• 44:22
  So that's just a nightmare and you can't really.
• 44:25
  I haven't seen a good enough solution right now that can put
• 44:28
  that can control the output of an LLM and say, hey this is not
• 44:33
  data that you have access to because there is no data control
• 44:36
  within another level. Yeah, it's almost like you need
• 44:40
  to do the reinforcement learning with human feedback on top of
• 44:46
  the fine-tuned model. Yeah, it will just be your like
• 44:50
  compliance layer and say as an AI with your security
• 44:55
  credentials you cannot, I cannot tell you that type of thing.
• 44:58
  Exactly. Exactly.
• 45:00
  And I don't think that's going to be an easy problem to do
• 45:02
  because there's this hundreds of permissions that the AS got to
• 45:06
  learn about. Yeah, 100% man, that is wild.
• 45:10
  That that's really like, it makes you think twice about the
• 45:14
  use case that you're trying to solve for.
• 45:17
  And this one comes down to what is the worst that could happen,
• 45:23
  right? Like, ohh, yeah, we've got this
• 45:25
  chat with our data and we think everything's fine.
• 45:28
  But what's the worst that can happen if all this data is now
• 45:31
  free to anyone who is prompting The model is that maybe that's
• 45:36
  really bad or maybe it's not. So really thinking through that
• 45:41
  isn't a fascinating one. And then also the idea of like
• 45:44
  having these digital keys. I like that too, because it's
• 45:48
  like you're giving out Brex cards for the open API keys
• 45:52
  almost. Exactly, Yeah.
• 45:54
  Yep, that's cool. So there is another thing that I
• 45:58
  wanted to touch on with you, which is the idea of forward
• 46:01
  compatibility. I know that you mentioned you
• 46:05
  were thinking a lot about this. I'm not super clear on what it
• 46:08
  is, so please explain it to me like I'm 5.
• 46:11
  Absolutely. So I'm sure you've already
• 46:15
  thought about it in some form of fashion, but I think the term
• 46:18
  forward compatibility, let's look at it right.
• 46:20
  So whenever you're designing any system, you always worry about
• 46:23
  backward compatibility. So anything I design now should
• 46:27
  be supported on every other system that my users are going
• 46:30
  to see it on, right? So it has to be compatible with
• 46:32
  browsers and machines and whatnot.
• 46:35
  Yeah, I think with. With AI, the pace at which it's
• 46:39
  moving, one constant fear I hear from enterprises is that.
• 46:44
  You know, any investment we make might just get wasted two months
• 46:49
  from now. So should I be even fine tuning
• 46:51
  my model now? Should I even build my own
• 46:53
  foundational model? So, And in that conundrum
• 46:57
  almost, there's inaction. So you can't take a decision,
• 47:01
  you can't move forward because you're like hey the there's no,
• 47:03
  the dust hasn't settled and I can't move fast enough to really
• 47:08
  launch anything into production, which is where a lot of
• 47:11
  enterprises seem to be they want to invest, they are investing,
• 47:14
  but then they they're constantly caught on.
• 47:17
  You know, we'll, we'll keep chasing a trend that changes too
• 47:20
  fast for our own likeliness. Yeah.
• 47:23
  And I think that's where the concept of forward compatibility
• 47:25
  comes in, where we're saying that.
• 47:28
  You probably need to have a system in between your
• 47:31
  application and generative AI that allows you to keep up with
• 47:37
  the new models, the new databases and the new frameworks
• 47:42
  that come into play. So.
• 47:44
  So how were you able to? So let's take this example,
• 47:48
  right, GPD launched 0613. They said, hey, this is the
• 47:52
  latest and greatest model we have.
• 47:54
  Go try it out and that's it, right?
• 47:57
  But then how does an enterprise really try it out?
• 47:59
  They're not used to change management by just switching API
• 48:02
  endpoints. They have a testing procedure,
• 48:05
  they have user focus groups and whatnot.
• 48:07
  Tonnes of things to do. O We're seeing this is where
• 48:10
  probably a smart AI gateway can come into place, where now the
• 48:14
  enterprise goes and says OK. G's non 0613.
• 48:18
  We're seeing a good enough success rate with the previous
• 48:21
  version, but now we want to test this out.
• 48:24
  So they say, OK, let's route 5% of our traffic.
• 48:28
  In a very dark mode, we're sampling some traffic and
• 48:31
  sending that out to GPD 613. The result that comes back is
• 48:36
  evaluated with stuff that is being shown to the user, which
• 48:40
  is from the older model. You do a quick evaluation over
• 48:44
  the course of a day, decide if this is worth shipping or not.
• 48:47
  So it's almost like a private benchmark that you're running in
• 48:50
  real time. Once you're satisfied, which is
• 48:54
  almost like a CICD test, you've ascertained that nothing has
• 48:58
  regressed. You can then say, OK, let's do a
• 49:00
  5% rolling release to my users. So 5% of my users see the newer
• 49:06
  model, 95% see the older model. When you're happy, you can ramp
• 49:10
  it up to 100% and now this entire orchestration can then be
• 49:14
  managed by an AI gateway. And the only way this is
• 49:17
  possible is if you decouple your AI gateway from your code, so
• 49:22
  you can continue to have lane chain lamma index your own code
• 49:26
  that is making these calls to alms.
• 49:28
  But then you have something in between that lets you very
• 49:32
  quickly swap out and swap in components.
• 49:35
  O. If you're using Lamma Lamma 70B,
• 49:37
  you're fine-tuned. It can you also very quickly
• 49:40
  find Tune a Mistral. On the family.
• 49:43
  Tested out. Launch it in a phased manner to
• 49:47
  your customers and that's I think.
• 49:50
  What we're talking about as forward compatibility.
• 49:53
  Wow, Yes, that makes 100% sense. And the idea of that fear is
• 50:02
  real. I have heard people talk about
• 50:04
  it too. It's like by the time I actually
• 50:07
  get my the Mistral fine-tuned is they're not going to be a new
• 50:11
  model that is the flavour of the month.
• 50:13
  And I don't even want to mess around with minstrel or how do I
• 50:17
  know which? Do I Do I find Tune Llama or do
• 50:20
  I find tune Mistral? Do I find tune Falcon?
• 50:24
  And the ability to say, you know what, I'm going to try all of
• 50:29
  them and see if I can do it real quick and then see which ones
• 50:33
  are giving me the best preliminary results.
• 50:36
  And then we double down on that one.
• 50:38
  Yep. And so you said something there
• 50:42
  that's that's also interesting that I would love for you to
• 50:44
  dive into more, which is decoupling the AI gateway from
• 50:49
  the code. Can you explain that more?
• 50:53
  Yeah. Sure.
• 50:54
  So I think the way to getting to this stage of forward
• 50:59
  compatibility would be let's not call.
• 51:04
  Let's not hardcode the open API call within my code.
• 51:08
  O within my code I should decouple it to say OK.
• 51:13
  I need to make an LLM call. With this prompt ID with this
• 51:17
  data that I have and expect an answer.
• 51:20
  That's it. The AIR gateway can then
• 51:23
  transform this prompt data and expected API call into the best
• 51:29
  possible model that's out there based on the rules I've defined.
• 51:33
  And then on the way back again transform the response into a
• 51:36
  consistent format that the code understands.
• 51:38
  O. Now my code never needs to
• 51:40
  change when I'm switching between models, when I'm
• 51:43
  switching between rooms. And that exists on a very thin
• 51:47
  fast layer, which is the gateway I see.
• 51:53
  And it's almost like you can add in what we were talking about
• 51:57
  earlier on what are you optimising for.
• 52:00
  And so that the model or that call knows, OK, we're optimising
• 52:05
  for speed, we're going to go for this smaller model, we're
• 52:08
  optimising for cost, we're going to go for the open source model
• 52:11
  etcetera, etcetera. And you don't have to put that
• 52:16
  specifically in the code, you just put pointers to the AI
• 52:20
  gateway and the AI gateway knows what to do with it.
• 52:23
  So it's basically the AI gateway is starting to make sense for me
• 52:26
  now. The AI gateway is like a catch
• 52:29
  all for whatever you're doing. Nothing goes through your walled
• 52:37
  garden. Nothing can make it through to
• 52:41
  the LM, whatever LM it is, unless it goes through the API
• 52:45
  Gateway. Yeah, Yep, exactly.
• 52:50
  And it makes sense why you would want to put the security and
• 52:54
  compliance layer there also. Yes, true.
• 52:58
  Because then it's like, hey, make sure that nothing ever hits
• 53:02
  the LM that I don't want to. Exactly, yeah.
• 53:07
  From once it hits the LM, it's kind of too late.
• 53:10
  Then it goes straight to output. It doesn't go back through the
• 53:13
  AI gateway. So it also goes back through the
• 53:17
  I gateway so you can ensure that no data leaves your walled
• 53:20
  garden that you didn't intend to leave.
• 53:23
  And whatever comes back from the AI can then be evaluated, tested
• 53:28
  on. Checked for moderation.
• 53:31
  All of that and then send back. So it's basically just on both
• 53:35
  sides. You're making sure that.
• 53:38
  We've picked the best prompt, the best model and send the
• 53:43
  request out and then on the way back in we've tested everything
• 53:47
  and made sure this is the best response that's possible and
• 53:50
  then send it back to the user. The magic is in how fast can you
• 53:54
  do it and how accurately can you do it and that is where I think.
• 54:01
  That's where we are most excited about saying how do you build
• 54:05
  the best AI gateway? And essentially you can almost
• 54:09
  think of how does this connect to a firewall that somebody else
• 54:12
  is building? How does this connect to?
• 54:14
  Maybe we have our own observability there, yes, but
• 54:17
  you want to connect this to a data dog and here I want to do
• 54:20
  all of my monitoring data log. Perfect.
• 54:22
  The gateway connects to Data Dog as well.
• 54:25
  So that then evolves as the one product enterprises will use.
• 54:29
  If they want to be confident about deploying LMS in
• 54:32
  production, that's it. Yes, alright, Fascinating.
• 54:38
  And then I guess the fun part there too is thinking about all
• 54:42
  of the possibilities that you now have if you're able to.
• 54:47
  Say, whenever I use the Mistral model, I have specific prompts,
• 54:54
  and the API Gateway knows what prompts work best for Minstrel,
• 54:57
  and so it takes that into account as it throws the data or
• 55:03
  the whatever my like call is. And if I'm using 3.5, then cool,
• 55:10
  I I have my specific prompts for that too.
• 55:13
  If I'm using four, I have my specific prompts.
• 55:16
  If I'm using cloud, whatever it may be, and you don't really
• 55:20
  have to in the code, specify any of that because the API gateway
• 55:26
  knows. Alright cool, we're routing this
• 55:28
  based on whatever conditions we need.
• 55:32
  We're routing it to Claude, so we know this is the prompt that
• 55:37
  we're going to use for this type of scenario.
• 55:39
  Exactly, exactly, yes. Yes, dude, I love it.
• 55:44
  I I knew you were working on some cool shit.
• 55:46
  I didn't realise it was that cool.
• 55:48
  And I had seen the idea of this API Gateway, because I think it
• 55:55
  came up in one of the threads in Slack, right?
• 55:58
  And that idea seemed like something that people were very
• 56:02
  much on board with. So that's awesome.
• 56:05
  Before we jump man, I want to know what was one of the hardest
• 56:11
  design decisions that you've had to make since creating import
• 56:15
  key. Yeah.
• 56:17
  Ohh. I would say it's something that
• 56:20
  we're still debating, is the. Is going open source or not?
• 56:24
  It's not a design decision almost.
• 56:26
  I think it's more a company design decision.
• 56:30
  Do you want to do it open source or not?
• 56:32
  And I think we've. We did a lot of thinking.
• 56:36
  We brainstormed with Shaggy about it, tell us the pros and
• 56:39
  cons of open sourcing this or not.
• 56:42
  I think we ended up deciding that hey, the AI gateway is
• 56:45
  probably one of the pieces that we should open source and we
• 56:48
  have gone ahead and open sourced.
• 56:50
  We've not really gone ahead and announced saying our AI gateways
• 56:54
  open source, but Rubias is the AI gateway that we've opened up
• 56:58
  that's got transforms to so many different AI models out there.
• 57:02
  It has caching. Retries, fall backs, all of
• 57:06
  these capabilities built in. And we'd love for the community
• 57:09
  to come in, look at it, build their own gateways, following
• 57:13
  those principles or contribute back to it, etc.
• 57:16
  But that's been really hard for. We're not a nonprofit
• 57:21
  organisation, so how do you decide what to open source?
• 57:24
  What makes sense and not just for OK, because the trend is to
• 57:28
  open source will open source something, but what?
• 57:30
  Is it meaningful enough that we open source into the world?
• 57:33
  And then what is meaningful enough that we keep it internal?
• 57:38
  So yeah, I don't know if that answers your question, but that
• 57:40
  was probably a very long and hard discussion internally.
• 57:45
  Yeah, it. It's fascinating to me thinking
• 57:47
  about that because yeah, at the end of the day, a man's gotta
• 57:52
  eat. And you wanna make sure that if
• 57:56
  you're open sourcing something, it's providing value to people.
• 57:59
  But then you're not cannibalising the business by
• 58:02
  open sourcing something or you don't open source too much.
• 58:06
  And if someone's going to pay for your product, what is that
• 58:11
  extra value add that you're going to be offering to the
• 58:17
  different customers? So I feel you, man, I'm sure
• 58:21
  it's something that you still are thinking about and you're
• 58:25
  trying to navigate your ways and it doesn't help that everything
• 58:29
  is so new and everything is changing and and there's a new
• 58:32
  tool that's coming out in the stack every other day.
• 58:35
  And so it's like where do we fit where like that identity crisis
• 58:41
  almost and recognising if you are an open source player or if
• 58:44
  you're a a closed source play and how that looks.
• 58:48
  Absolutely. Yep.
• 58:50
  Yep. That's true.
• 58:52
  We're ahead, dude. Thank you so much for coming on
• 58:54
  here and explaining all of this to me.
• 58:56
  I enjoy every time that we talk you know this.
• 59:00
  I always appreciate all the help you've given me in the community
• 59:03
  and helping me understand things from a fundamentally different
• 59:07
  level. I also cannot thank you enough
• 59:10
  for the help you did on the report and hopefully we're going
• 59:15
  to make another one of these with the evaluation report and
• 59:17
  that's right up your alley. Stuff that you're working on
• 59:20
  right now. For everybody out there that is
• 59:22
  listening, check out Port Key, show some love.
• 59:25
  And the open source project was called What again?
• 59:28
  It's called Rubius. You can just search for Rubius.
• 59:31
  Yeah, RUBUS on GitHub, Rubias, here we go.
• 59:36
  So go check that out, play with it, have some fun with it and
• 59:40
  we'll talk to you later, man. Absolutely.
• 59:42
  Always a pleasure speaking with them.
• 59:48
  Hey, I'm Vishnu. I'm a data scientist at first
• 59:50
  hand and I definitely think that you should subscribe to the ML
• 59:54
  OPS Coffee Sessions podcast. It's the best podcast out there
• 59:58
  to stay on top of what ML OPS actually is to talk to the true
• 1:00:01
  thought leaders in the space. And oh by the way, Demetrius is
• 1:00:05
  absolutely hilarious. What a weird guy.
• 1:00:07
  You should definitely subscribe to the podcast.