Latest / Agentic Conversations (formally mlops.community) / How to Systematically Test and Evaluate Your LLMs Apps // Gideon Mendels // #269

Transcript

How to Systematically Test and Evaluate Your LLMs Apps // Gideon Mendels // #269 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Gideon Mendels, I'm the CEO and Co founder of Comet.
• 0:05
  I make my own espresso at home, I grind and weigh my beans and
• 0:11
  depending on my mood I either drink just an Americano or a
• 0:14
  cappuccino. What is happening?
• 0:17
  Good people of Earth? We're back for another ML OPS
• 0:22
  community podcast. I am your host, Dimitri OS, and
• 0:27
  I've got a little bone to pick with some of y'all.
• 0:32
  There is this trend going around and instead of saying I'm just
• 0:38
  following up or I'm circling back, tell me if you've noticed
• 0:42
  the new trend is just a gentle reminder, Just a gentle ping on
• 0:48
  this. There's nothing gentle about it.
• 0:50
  You're fucking the shit out of me.
• 0:51
  I'm not answering for a reason. Get over it.
• 0:54
  No more gentleness. Let's be brute.
• 0:57
  Let's just be crap. So talking with Gideon today, I
• 1:04
  had a blast because he's been in the traditional ML world and he
• 1:09
  recently open sourced OPIC which is in the Gen AI world.
• 1:15
  It is all around evaluation and it's a tool to help you evaluate
• 1:20
  your LLM or Gen AI product output.
• 1:26
  And we talked at length about how these two worlds of
• 1:30
  experiment tracking that he's been doing for the past 8-9
• 1:35
  years and the evaluation tracking or world, those two
• 1:42
  worlds, they're not different. They're actually eerily similar.
• 1:47
  And so let's talk to him about it.
• 1:50
  A huge shout out to the Comet team for sponsoring this
• 1:53
  podcast. I got to say it, they are the
• 1:56
  reason that we are able to do what we are able to do.
• 2:00
  Thank you to them and all of our sponsors here at the MROPS
• 2:04
  Community Podcast. Let's jump in to this
• 2:07
  conversation, and as always, a gentle pig.
• 2:13
  A gentle reminder to pass this on to 1 friend.
• 2:25
  So. First thing we got to talk about
• 2:27
  is what is that solar panel behind you?
• 2:32
  It's a photo of an astronaut. I think they're doing a
• 2:34
  spacewalk working on the ISS probably.
• 2:38
  I'm not sure actually. But yeah, it's, it's in our
• 2:40
  office. I should probably ask where this
• 2:42
  originated from. I'm literally looking at it.
• 2:47
  And for the people that are just listening, in some world, some
• 2:51
  alternate reality, before you told me what it was, I saw it as
• 2:55
  a doorway behind you. And it was like, I thought you
• 2:59
  were back in like the laundry room and then there was some
• 3:01
  random solar panel that was stashed behind you overhead that
• 3:05
  probably was decommissioned or something.
• 3:08
  Thankfully the start up, the early start up days of working
• 3:11
  from the laundry room are, you know, 6-7 years behind us.
• 3:16
  But yeah, hey, it's never too late.
• 3:19
  You've matured as a company. You don't need to both be doing
• 3:23
  your laundry and building your startup.
• 3:26
  Yeah, life goal. Exactly.
• 3:29
  So I'm excited to talk to you about OPEC.
• 3:32
  I also want to just like dig in evaluation and evaluation
• 3:36
  metrics and what you've been seeing out in the field.
• 3:40
  Also, I think it's probably worth talking a bit about
• 3:43
  experimentation because you're kind of the OG in that.
• 3:45
  First off, let's kick it off man.
• 3:47
  With like you've been out there, you've been talking to a tonne
• 3:51
  of people, what are some key evaluation metrics that you've
• 3:54
  seen people using? Yeah.
• 3:57
  So kind of taking it from the top right, as we for for some
• 4:02
  context, like when we talk about evaluation matrix, right, like
• 4:06
  like you said, we've been doing ML experiment tracking for seven
• 4:09
  years and evaluation is not a new concept, right?
• 4:13
  I think it, it definitely behaves a little bit differently
• 4:16
  when it comes to LLMS, you know, and kind of more supervised
• 4:19
  models depending on the task. We have, you know, accuracy F1
• 4:23
  scores, you know, LLMS by itself have kind of more traditional
• 4:27
  metrics like such as perplexity, which is not just the company
• 4:30
  name, it's a, it's an LLM metric or language model metric.
• 4:35
  But yeah, like I think in the context of people building on
• 4:39
  top of LLMS, so not necessarily training or fine tuning, things
• 4:44
  look a little bit different, right?
• 4:45
  If you have labels in a supervised model, then all the
• 4:48
  kind of classical missionary metrics make sense.
• 4:51
  When we talk about LLMS, there is essentially the way we view
• 4:56
  it and it's it, it's definitely task dependent.
• 4:58
  There's kind of like three types of metrics you could use.
• 5:01
  And so let's take up an example use case.
• 5:05
  If we're building a rag chatbot, internal or external.
• 5:10
  So the the the basic one are just like the deterministic
• 5:14
  assertion, right? Like I don't want the model to
• 5:17
  ever say I'm an AI model so I can search for that string, you
• 5:21
  know, assert does not contain. So those are like the classic
• 5:24
  ones from kind of the software engineer world.
• 5:27
  But as we know, the output of an LLM could look very different
• 5:33
  from a string perspective. A semantically could mean
• 5:36
  exactly the same. So these deterministic
• 5:38
  assertions break very quickly. And then it brings us to like
• 5:42
  the two other sets of metrics which are the call it a
• 5:46
  heuristic distance like metrics. So for example, if I have a
• 5:52
  question that I want my model to answer right, say what is the,
• 5:58
  you know, envelops community podcast about right?
• 6:01
  So I'm fine if the answer doesn't look exactly the same
• 6:04
  every time, as long as it cover the main things that I wanted
• 6:07
  to. So I can measure the distance
• 6:10
  was something like bird distance or, you know, any other any any
• 6:13
  type of distance between two embeddings between my preferred
• 6:17
  answer and what I got now. So those are the second set.
• 6:20
  And then the last one, which is a slightly newer concept is
• 6:23
  using another LLM to produce a metric on in our original LLM
• 6:29
  response, which is what's called LLM as a judge.
• 6:32
  So, you know, we can use it for the same example earlier instead
• 6:36
  of measuring the distance and you know, with cosine
• 6:38
  similarity, I can ask another LLM.
• 6:40
  Hey, does both of these answer look the same?
• 6:44
  So that's a high level. That's what people use.
• 6:47
  There's still a lot to uncover in that space.
• 6:51
  It's still hard, but I would say if those are the three main
• 6:54
  ones. So one thing that I want to
• 6:57
  point out that you mentioned and articulated beautifully is
• 7:02
  there's a difference between the traditional ML evaluation
• 7:07
  because it was more evaluation of the model when you were
• 7:10
  training it, right? And now with LLMS you have
• 7:14
  evaluation, but that's once it's out in the wild or once it's
• 7:18
  already been trained, it's already outputting something.
• 7:20
  You're evaluating the output on that.
• 7:24
  Yeah. Well, so you're always
• 7:25
  evaluating the output, right? If you have a simple
• 7:29
  classification model, you compute the metrics based on the
• 7:34
  expected output versus what the model actually gave you.
• 7:38
  But you're right in the sense that like it's no longer a model
• 7:41
  that you even have control over in terms of training and fine
• 7:44
  tuning. You could be using Open it, Open
• 7:47
  AI, you know, ChatGPT 4 O or something, but you still need to
• 7:51
  evaluate how well you're doing. So you don't control the model,
• 7:54
  the model weights fine tuning, but you do control the prompt,
• 7:59
  right? And in some cases some call it
• 8:01
  production hyper parameter such as temperature and seed maybe.
• 8:05
  But yeah, so it is. It is quite different.
• 8:08
  So then the other piece that you were talking about is this, if
• 8:13
  you understand the question that's being asked or if you
• 8:16
  understand what you're going for, you can use Burb metrics,
• 8:20
  right? And that's if you know what you
• 8:22
  want. But a lot of times you have no
• 8:25
  idea how things are going to be used or you, you maybe it's not
• 8:31
  that you have no idea. You kind of assume or expect
• 8:34
  folks to use your AI feature or your AI product in one way, and
• 8:39
  then maybe somebody comes out of left field and starts using it
• 8:43
  in a different way. So you can't really use those
• 8:47
  burp metrics. Yeah.
• 8:49
  So it's a good point, right? It's very use case dependent.
• 8:52
  And sometimes you're right, like there are cases where you have
• 8:55
  no idea, just have to put it out there and see what people do
• 8:58
  with that. If you're building a rag or
• 9:00
  something, I would argue that in most cases you should be able to
• 9:03
  create like a, call it a goal data set of some examples,
• 9:07
  questions and answers. But either way, your points very
• 9:11
  valid. Because even if you created that
• 9:13
  and you have some kind of testing and evaluation before
• 9:16
  you release it, once it's in production, people will, you
• 9:19
  know, definitely surprise you. So that's, that's a big part of
• 9:23
  like what OPEC does, right? Because, and, and you know,
• 9:25
  there's definitely other solutions.
• 9:28
  They do that, but the idea is like you can bootstrap these
• 9:31
  data sets, right? So you put your what's called a
• 9:34
  chatbot in production. OPEC does all the tracing of all
• 9:41
  the responses, the questions, the chat sessions, everything
• 9:43
  happens automatically. And then you basically get a
• 9:47
  data set that you can either manually label human label.
• 9:51
  So person can go and say, hey, this answer was against our
• 9:55
  company policy or this answer had bias or this answer was just
• 9:59
  pure wrong. And then you can also use some
• 10:02
  of the, the built in evaluation like LLM as a judge metrics and
• 10:06
  then you bootstrap this data set by by by automatic and manual
• 10:11
  labelling. So the next time you iterate,
• 10:14
  you can actually run through those questions and answers.
• 10:16
  So that's like, I think like when, when I speak to teams and
• 10:24
  companies trying to deploy, especially RAG applications, I
• 10:28
  would say that's the majority of the LLM, not the only ones, but
• 10:31
  the majority of LLM use cases. That's kind of what separates I
• 10:35
  would say like those who made it to production and production
• 10:38
  could be an internal solution, right?
• 10:40
  Like, but still like with the real active users versus the one
• 10:46
  they put it out there and it kind of blew up.
• 10:47
  So it's this more data science oriented methodologies versus
• 10:53
  kind of more software engineering ones that makes
• 10:56
  sense? Yeah, explain that a little bit
• 10:59
  more because I do like what you're saying there because the
• 11:05
  software methodology is, I know exactly what I do not want.
• 11:10
  So I'm going to say if you pump out this string, just cancel
• 11:14
  that, right? But there's going to be a
• 11:16
  million different ways that an LLM can say that same thing.
• 11:19
  So all those edge cases, you're going to have to either spend a
• 11:22
  tonne of time trying to cover them all in your test cases or
• 11:29
  you do something else like I think and that's what you're
• 11:31
  getting at with the data science methodology.
• 11:34
  Yeah, yeah, absolutely. So like first of all, like it's
• 11:37
  really fascinating to see like the, you know, there's, I would
• 11:41
  even say the majority of these people that I spoke to building
• 11:45
  LLM applications are software engineers, which is great
• 11:48
  because like good being a good engineer helps you move so much
• 11:52
  faster. There's obviously data
• 11:55
  scientists who are amazing engineers as well, but it's like
• 11:57
  a slightly different paradigm. But to your point is, it's very
• 12:01
  accurate, right? On the software world, testing
• 12:03
  is a huge component, right? Like there's books and
• 12:06
  methodologies and unit testing, integration station smoke
• 12:09
  testing. Like there's, there's a lot of
• 12:11
  way to test software, but essentially all of those break
• 12:16
  when you try to test an LLM application.
• 12:19
  Because your point, I can write assertions, unit test,
• 12:23
  integration test assertions expecting a string, but an LLM,
• 12:29
  you know, your LLM provider updates their version, you're
• 12:32
  going to get a different string and probably hopefully a better
• 12:36
  one. But you don't want to break the
• 12:38
  bill because of that. And you don't want to go and
• 12:39
  change all these strings again. So that's where you need that
• 12:43
  fuzziness, if you call it, which adds risk as well, right?
• 12:46
  That's but it's it's this deterministic versus non
• 12:49
  deterministic nature of software and data science.
• 12:53
  So the fun part there is the idea of software engineers
• 12:59
  moving more towards understanding data science
• 13:02
  paradigms and then vice versa, data scientists moving more
• 13:06
  towards understanding software engineering paradigms, which I
• 13:09
  think a lot of people on the Internet and on social media
• 13:13
  have been screaming about for a good couple years.
• 13:16
  Like data scientists should learn how to code, or they
• 13:18
  should at least learn how to use Git or software engineering
• 13:23
  principles at the very least. You're the first person that
• 13:26
  I've heard to say vice versa. Like software engineers, if
• 13:30
  you're dealing with LLMS, you want to start to understand the
• 13:33
  nuances of that fuzziness. Yeah, yeah, exactly like, so by
• 13:38
  all means, I think data scientists should pick up as
• 13:42
  many software engineering skills as possible.
• 13:44
  The the best data scientists I've met in my career were like
• 13:49
  brilliant software engineers as well, right?
• 13:51
  It's just it goes together. But I think so first of all, I
• 13:56
  think what we're seeing is the two fields converging.
• 14:01
  Like, don't get me wrong, you're still going to have the pure
• 14:04
  research, data science, machine learning people like deeply
• 14:09
  understand the math of, you know, these optimization
• 14:13
  algorithms and, and lost functions.
• 14:16
  And there's still definitely going to be that group, but I
• 14:21
  think the majority of the use cases around machine learning
• 14:24
  and AI will be this convergence of, call it, software engineer,
• 14:28
  data scientist, but being able to use those things.
• 14:32
  But there's still a lot in the data science methodologies, just
• 14:38
  way of thinking that I think software engineers have never
• 14:41
  been exposed to that before could benefit a law firm, right?
• 14:44
  Like the in data science. And it doesn't matter which task
• 14:47
  you're working on or which you know, algorithm or model you're
• 14:51
  using, but it's a, it's an experimental approach and
• 14:56
  specifically a metric driven approach.
• 14:59
  So you generate assuming a task, you, you have a data set,
• 15:03
  hopefully with gold data set, so you know what the right answer,
• 15:07
  what the right output should be. And then you start changing the,
• 15:11
  the in this case, the prompts the pipeline, these parameters.
• 15:17
  And every time you run one of these experiments, you test it
• 15:20
  against the data set. Now you still want to look at
• 15:23
  the data and all of that, but it's not these like this vibe
• 15:26
  check approach. It's more of a, you know, metric
• 15:30
  driven approach. And then with that, it opens up
• 15:33
  this like, you know, kind of warms of all the things you
• 15:35
  should be aware of, like, you know, overfitting and such.
• 15:38
  It's it's still a risk in this case, right?
• 15:40
  Even if you're not controlling the LLM, if you're, if you have
• 15:43
  a small gold data set and you're, you know, really tuning
• 15:47
  the prong to like 10 or 15 answers, you might overfit,
• 15:50
  meaning it will look perfect on those 15 answers.
• 15:53
  But then when you start putting in production, it will fail.
• 15:55
  So that way of thinking is just, you know, it's just, it's really
• 16:00
  the only way to go here. So I'm really, it's, it's
• 16:03
  exciting to see a lot of software engineers picking up
• 16:06
  and that's a very huge focus of what OPEC is.
• 16:08
  It's kind of trying to bridge those two paradigms.
• 16:11
  Have you seen a way to unit test LLMS that are super helpful?
• 16:20
  Yeah, so I guess the it's again the, yeah, the, the pure
• 16:24
  definition is of unit testing of like, hey, let's, I mean,
• 16:29
  there's two sides of, you know, unit testing.
• 16:31
  There's a philosophy you should test what you should test.
• 16:34
  And how does it look like Obviously in an LLM you don't
• 16:37
  control the internals, but I think the idea here is again,
• 16:43
  having some deterministic simple assertions is very helpful, but
• 16:49
  they're very brittle in that case.
• 16:51
  So that's where you want to start, including more heuristics
• 16:53
  and L on as a judge once. And talk to me about LLM as a
• 16:58
  judge, because I know there's fifty different ways to do that.
• 17:04
  Have you seen certain ways that folks are gravitating towards
• 17:08
  more? Yeah, So it's right like LL
• 17:12
  judge has has its benefits and definitely some disadvantages.
• 17:17
  I wouldn't say there's like the first specific judge call it
• 17:22
  though, hallucination detection or bias detection.
• 17:25
  I wouldn't say that there's necessarily a way how everyone
• 17:30
  does it right. Like the prompt LLM of the judge
• 17:33
  is a fancy name. It just basically means you take
• 17:35
  the output of your original LLM and then you tap on it a prompt
• 17:41
  that asks a different LLM or even the same one.
• 17:43
  Hey, does this contain bias? Right.
• 17:46
  But so obviously people develop those prompts and those prompts
• 17:49
  are also perform differently on different LLM providers.
• 17:54
  So I wouldn't say there's necessarily a method that
• 17:59
  everyone uses. It's really excited to see Open
• 18:02
  AI in their last dev day actually released some form of
• 18:07
  like in their playground and LLM is a judge mechanism there.
• 18:13
  So they, they wrote the Proms for them and knowing Open AI,
• 18:17
  they probably did a good job developing those prongs.
• 18:20
  So that's, that's great to see some form of consolidation
• 18:23
  there. The one thing I, I, I, I would
• 18:27
  say I, I've seen people do LM the judges and LLS in general
• 18:33
  are depending on the scale, could be very expensive.
• 18:37
  So I've seen customers use like a lower or cheaper LLM in
• 18:42
  production, but then use an expensive judge to test it.
• 18:45
  So then you kind of get best of both worlds.
• 18:47
  That's a. That's one approach I've seen.
• 18:51
  Interesting. No coming back to OPEC because I
• 18:56
  wanted to dig into one of the pieces that you have around it,
• 19:01
  which is that track function. And for those who don't know I
• 19:07
  my understanding of it after reading the GitHub page, because
• 19:10
  it's fully open source, which is awesome to see, is that you can
• 19:15
  track just about anything that you want when it comes to these
• 19:20
  LMS as long as you tag it with like hey, this is important to
• 19:25
  me, I want to track it. Is that correct?
• 19:28
  OPEC is fully open source from a functionality perspective with
• 19:32
  exactly the same functionality we have in our cloud hosted
• 19:36
  version. So it's not a crippled product
• 19:38
  or anything like that. You can actually run it for
• 19:41
  production use cases at very large scale just from the open
• 19:47
  source project. But but to your point, yes.
• 19:49
  So it comes with built in integrations with like the major
• 19:53
  LLM providers, like, you know, the major libraries like yeah,
• 19:57
  Llama, index bank chain, Ragas and so on.
• 20:02
  But even if you're not using any of those things you pointed out,
• 20:06
  we have this really cool way of auto instrumenting your
• 20:10
  pipeline. So just by adding a notation at
• 20:13
  track above your function or class.
• 20:16
  We will track all the basically everything is going on in your
• 20:20
  call it chain or, or pipeline. So yeah, I think you you had the
• 20:26
  right idea there. And we've seen people track
• 20:28
  pretty, you know, different use cases and products.
• 20:31
  With it. So that's everything basically
• 20:35
  on on your side or if I'm creating an app, I can track
• 20:39
  whatever on my pipeline. I also wanted to get into can we
• 20:45
  track the responses of folks or signals that will help us
• 20:52
  understand if the response was valuable?
• 20:55
  Like something like if someone copy and paste this or there's
• 20:59
  the thumbs up, thumbs down, which I think we as a community
• 21:03
  have come to understand that it's not that valuable just
• 21:07
  because people that give you thumbs up or thumbs down may not
• 21:10
  be the best user segment that you actually are trying to shoot
• 21:15
  for or whatever. There may be a lot of reasons
• 21:17
  why that can be biassed or not the best signal, but a great
• 21:21
  signal could be you get one answer and then you don't ask
• 21:25
  for any follow up questions and you copy paste.
• 21:28
  And so that could be something that is really interesting or if
• 21:31
  there are follow up questions, then you can see what parts of
• 21:36
  the follow up questions it's trying to dig into and all that
• 21:41
  stuff I can imagine could be valuable for the app developer.
• 21:45
  Have you seen any of that? Yeah, absolutely.
• 21:49
  Right. So it depends on the use case,
• 21:51
  right. Like OPEC is not just for call a
• 21:53
  chat bots or rag applications, it's for essentially any LM
• 21:57
  powered feature and by default will track the, you know, the
• 22:03
  input of output of that LLM to definitely in the chat example
• 22:06
  you gave, you'll definitely see the responses.
• 22:09
  Now I think where you're going to is like, Hey, can we feed an
• 22:13
  additional call it features, BI features or or or other ones so
• 22:19
  we can later understand how well the response was.
• 22:24
  I haven't seen people use a copy paste one.
• 22:26
  That's actually a it's a cool idea, but we've seen definitely
• 22:30
  people feed in like a downstream, call it product
• 22:34
  analytics event. You can create essentially all
• 22:38
  these quality evaluation criterias on OPIC both again,
• 22:43
  that you can manually tag, but also automatically, right.
• 22:46
  So in, in your example, I think you said, hey, did this user
• 22:50
  based on this answer did like an activity in the product, You can
• 22:53
  add that, as I call it, categorical annotation.
• 22:57
  And then again, you bootstrap this data set.
• 23:00
  So later when you're reworking and improving your prompt, you
• 23:03
  can actually test how the new prompt does on those things.
• 23:06
  So it's pretty cool. Obviously you wouldn't be able
• 23:08
  to simulate whether the user did that or not, but it's a it's a
• 23:12
  very good, very good proxy for quality.
• 23:15
  So, yeah. And I think it, you know, brings
• 23:19
  the question, can we actually feed it into the training
• 23:21
  process down the down the road? But that's that's an open
• 23:25
  research question. Oh, yeah, that's fascinating to
• 23:28
  think about how. And the other piece to think
• 23:31
  about is not just, all right, we get that information, but then
• 23:36
  with that information, what do we do with it?
• 23:38
  Right. So cool.
• 23:39
  Yeah, I understand that when we have this certain snapshot of
• 23:46
  Proms and temperature and rag retrieval, whatever the whole
• 23:50
  system, we have that snapshot that is tracked and we see that
• 23:55
  it performs higher on these types of queries or this type of
• 24:01
  task. And we we think that because
• 24:04
  we're seeing more copy paste or we're seeing whatever that
• 24:07
  metric is that we're tracking downstream like you said, like
• 24:09
  the product analytic metric, now we have that.
• 24:14
  What do we do with that? Yes, I think that it's, you
• 24:19
  know, we're talking kind of hypothetical.
• 24:20
  It's it's very use case dependent.
• 24:22
  But like, for example, if you're doing like an AB test, right, so
• 24:27
  you're running two different LLM variations or prompts and
• 24:32
  production, and you're testing these downstream product
• 24:36
  analytic methods. Like very quickly, you know, you
• 24:38
  can know in production which one's better, which is pretty,
• 24:43
  pretty big, right? When you think about it now, you
• 24:46
  know, that's why I'm saying it's very dependent.
• 24:48
  In some cases you don't have those things.
• 24:51
  And then like, I think in some scenarios maybe, and I'm sure if
• 24:56
  we look on archive, there is 50 PhD students working on this.
• 25:01
  But I think in some cases where you have those kind of
• 25:07
  production labels, call them, you can start thinking of how
• 25:11
  you merge both like the traditional ML approach of
• 25:16
  things and this LLM approach. So just thinking out loud,
• 25:22
  right? If you have an embedding vector
• 25:25
  of features, right? So like user user information,
• 25:29
  for example. And then you have some kind of
• 25:33
  downstream supervised task like did the user, you know, buy the
• 25:38
  product or not? I'm just trying to keep
• 25:40
  something simple. Then sure, you can add it in the
• 25:43
  prompt today in many ways, those like those that context
• 25:46
  information, which is what people are doing.
• 25:48
  But it'd be pretty cool to think like, can we actually fine tune
• 25:52
  the model with these features? Like, you know, there's similar
• 25:55
  to how people do RLHF, which is, you know, in this case, it's
• 26:00
  it's like human, but or I think there's direct approaches DPO,
• 26:05
  if I'm not mistaken. You basically, you know, train a
• 26:07
  model on, you know, on these. Yeah.
• 26:11
  So it'd be cool to see, right? Like it is so early and that
• 26:14
  this space is moving so fast. So I think as companies and and
• 26:19
  and use cases start hitting the limits of what they can get by
• 26:23
  just changing up the prompt or their RAG and or the vector
• 26:27
  database parameters, we'll start seeing more of these
• 26:30
  sophisticated approaches. It's almost like because it's
• 26:35
  been quote unquote democratised, we all can have our wacky ideas
• 26:41
  and just test it really quickly. And so that's where you get,
• 26:46
  hey, you know what if we use a graph database, it kind of works
• 26:50
  better. Let's try this.
• 26:51
  And now we've got this thing called the graph rack, right?
• 26:53
  And somebody can come out with a random paper and next thing you
• 26:58
  know, a few folks are testing it and seeing if it's reproducible
• 27:03
  and it's it's actually giving them some lift in what they're
• 27:06
  doing. And so I, I wonder if that is an
• 27:10
  area that we can see more exploration in that would be
• 27:12
  really cool just to marry those product analytic metrics with
• 27:17
  more of the training process to say, can this give me a better
• 27:21
  LLM? Or maybe there's nothing there
• 27:24
  and it's it's absolutely a dead end, but it's definitely, it's
• 27:28
  fascinating to ponder and I'm sure it's going to be somebody
• 27:32
  else that's doing it and neither you or I are going.
• 27:35
  To go and try and figure that out.
• 27:37
  But if anybody out there is listening and they are working
• 27:39
  on this, I would love to talk to him because it's really cool to
• 27:42
  think about. Super cool to think about and
• 27:46
  that's what I love, you know, but kind of the research world
• 27:49
  and like by past life before comet, that's what I did.
• 27:52
  It's just like, yeah, like in in many cases, like we're throwing
• 27:55
  spaghetti on the wall and seeing what sticks.
• 27:58
  But like, once someone finds an approach that shows promise,
• 28:02
  like people build on top of it and that that's like how we got
• 28:05
  to where we are, right? Like Transformers and the, you
• 28:08
  know, the attention and all this.
• 28:10
  All you need paper is a mix of like, you know, you know, a
• 28:16
  bunch of different approaches that we already had at that at
• 28:20
  that time, right? I was, I was actually at Google
• 28:22
  when that paper came out. I had nothing to do with it, but
• 28:25
  it was written like a few buildings from where I was.
• 28:28
  Nice. And, you know, we're using at
• 28:30
  the time LSD NS with attention and sure, there's like, you
• 28:35
  know, a lot of differences, but also it's not that different.
• 28:39
  And I think now people are talking and starting to show
• 28:42
  that in the right scale, you can even make those like older
• 28:48
  algorithms like RNNSLSDMSGRUS perform closely to Transformers.
• 28:54
  So yeah, it's, it's pretty cool to see how how much we progress
• 28:58
  we made based on like, you know, thousands of people just testing
• 29:01
  different ideas and, and approaches.
• 29:04
  Yeah, getting the ability for it all to fan out and then you have
• 29:10
  survival of the fittest happening and so.
• 29:12
  It's. Cool to see.
• 29:14
  And it, it also kind of dovetails into this next topic
• 29:19
  that I wanted to talk to you about around version control and
• 29:24
  and almost like experiment tracking.
• 29:28
  You have parallels with evaluation, right?
• 29:34
  And it feels like they're, they're almost like brothers or
• 29:39
  sisters. They're siblings in a way.
• 29:41
  Because you want to be able to, in experiment tracking, you want
• 29:44
  to be able to have a snapshot of everything so you can go and
• 29:46
  reproduce it and be able to get that model or that same output.
• 29:51
  And the same with like LMS and evaluation and being able to
• 29:57
  have a snapshot and say OK, now I want to be able to get that
• 30:02
  output all the time. I don't want it to be just some
• 30:04
  random thing that because I had one prompt that was tuned really
• 30:08
  nicely and I forgot what I saved it as or I forgot exactly how I
• 30:14
  worded it. Now I can't ever get that output
• 30:16
  again. Yeah, no, that's, that's amazing
• 30:19
  because like that's really how we got to to build OPEC, right?
• 30:24
  Because you know, Comet, their, their original focus is on the
• 30:28
  envelope side and experiment tracking, right?
• 30:30
  Like we have over 100,000 data scientists using that and we
• 30:35
  power some amazing ML teams that like Netflix and Uber and
• 30:39
  stability AI and Cisco and many, many more.
• 30:43
  And what we started seeing is like our own user or customer
• 30:47
  base kind of trying to hack the, the existing experiment tracking
• 30:51
  solution for these use cases because like when you think
• 30:55
  about it, you're right. Like they're very, very similar.
• 30:57
  So in traditional ML you have what do you control as a data
• 31:02
  centres? You control the data set, you
• 31:03
  control the hyper parameters, you control the code or the
• 31:06
  algorithm. And then you part with
• 31:11
  intuition, part with like methodologies.
• 31:13
  You keep changing those things until you get a metric that's
• 31:16
  satisfactory. And I'm not even getting yet to
• 31:19
  like the reproducibility side of it.
• 31:21
  Just like the nature of it. You're testing many, many, many
• 31:24
  things and you're trying to get a better result, right?
• 31:28
  And then when you think about these like LLM applications, so
• 31:32
  yes, you control different things, right?
• 31:34
  You don't control them. I mean, let's start with what
• 31:36
  you do control. You control the prompt, you
• 31:39
  control some LLM parameters. If you're building a rag, you
• 31:44
  have all these, call it hyper parameters around chunking and.
• 31:48
  The pipeline. Configuration of how the the
• 31:51
  database does the the re ranking and so on.
• 31:55
  And then you can create a test data set like we spoke before,
• 31:59
  and you can continue iterating on those things until you get a
• 32:02
  result that's satisfactory. Yeah.
• 32:04
  So the embedding model too that that's one that you.
• 32:07
  Could. Yeah, yeah, yeah.
• 32:08
  Then better. Absolutely, absolutely.
• 32:10
  You want to test different embedders, you want to test
• 32:11
  different LLM providers. Like it's, it's, you know, it's
• 32:14
  a, it's a, you know, it's a, you're searching that space.
• 32:18
  It's very, very similar. Yes, you're changing different
• 32:20
  things, but you're still blindly searching a space, right?
• 32:24
  You can't run gradient descent on it.
• 32:25
  Like you have to blindly with intuition of course, and
• 32:28
  knowledge start tweaking things until you get a result that's
• 32:32
  good enough. So it was really fascinating to
• 32:35
  see users trying to use a common experiment tracking for that,
• 32:40
  right? And, and you can get a pretty
• 32:43
  like pretty close to what you need.
• 32:46
  But then we want to track the traces and the span so people
• 32:49
  can look at the data, right? And we want to include these
• 32:52
  built in evaluations because that is what we saw customers
• 32:56
  needing to manually implement all the time.
• 32:59
  So, so yes, so like very similar.
• 33:02
  And then that's like the main layer of like the main value for
• 33:05
  position of experience tracking, whether it's an OPIC or comet EN
• 33:10
  which is I want to help you build a better application,
• 33:13
  better model. But on top of it and you called
• 33:16
  it out, right? Like you have this notions of
• 33:18
  like reproducibility, right? Like I have multiple people
• 33:21
  working on this project. I want to pass it to someone
• 33:24
  else. They need to know exactly what I
• 33:26
  did to get to this result. And then the the other piece is
• 33:30
  collaboration, right? Because if you don't use
• 33:34
  something like OPEC or comment EM like it's basically
• 33:38
  impossible to collaborate. I mean, you can copy paste logs
• 33:41
  to slack all day long. But if I Dimitri, if we're
• 33:44
  collaborating on something and I want to show you, hey, this is
• 33:47
  the prompt. These are the outputs like how
• 33:49
  do I share this with you today? Like I put in an excel sheet.
• 33:52
  Sure, you can use Excel for everything, but so it's really
• 33:56
  cool to see like 2 products that are like in, you know, there's
• 34:01
  so much in common and it's, you know, also from a user and
• 34:04
  customer base perspective. So when we identify this about,
• 34:08
  I call it like 18 months ago, we decided to double down on OPEC
• 34:14
  and, and, and build it. And, you know, we wanted to give
• 34:19
  it back to the community in this case because we have a, you
• 34:21
  know, strong foundation with our existing platform.
• 34:24
  So we made it completely open source as well.
• 34:27
  It's almost like the microcosm of the macrocosm we were just
• 34:31
  talking about. I want to highlight that real
• 34:33
  fast where what you're doing when you're exploring this
• 34:37
  unknown space in your own prompt pipeline or your own LLM
• 34:43
  application or creating a traditional ML model and doing
• 34:50
  your experiment tracking, that's exactly what we were just
• 34:52
  talking about with the advancements of AI.
• 34:57
  Is it? And it's just that for this,
• 35:01
  it's me going through on my computer working on it.
• 35:04
  But then for the advancements of AI, there's hundreds or
• 35:08
  thousands or hundreds of thousands of us that are all
• 35:11
  trying to figure out what the best ways to optimise this are.
• 35:15
  And we're all just exploring this unknown space together.
• 35:20
  And then whatever we see hits, hits and we we kind of run
• 35:25
  towards it. And so it's cool to think about
• 35:27
  that from the micro to the macro and how there's, there's those
• 35:32
  parallels. But the other piece that I
• 35:34
  wanted to mention is this collaboration factor, which I
• 35:38
  think is a huge part of what's going on, since collaborating
• 35:46
  is, is almost like a must in certain ways because you don't
• 35:51
  want to just have it all for yourself or you want to see if
• 35:54
  you can get a fresh pair of eyes on something.
• 35:56
  What is it that I'm missing here?
• 35:59
  What do I not know? And can I ship it over to
• 36:02
  someone and see it? Yeah, that's a great question.
• 36:04
  So essentially OPEC has, let's call it like 3 main views.
• 36:11
  I'm talking about the UI, right? Like there's, there's a very
• 36:14
  robust SDK and an extension to π test that allows you to add
• 36:18
  these evaluation metrics. But so it, it depending what
• 36:22
  we're trying to collaborate on, right?
• 36:23
  So let's start with the, let's start with the basics.
• 36:27
  Like we have a data set, we created a data set.
• 36:30
  So I can share you a link to that data set.
• 36:33
  You can have, you know, obviously multiple ones in the
• 36:35
  UI, you can look through it, see all the inputs, expected outputs
• 36:39
  and so on. Whatever columns we have,
• 36:40
  they're not that different from an Excel table at this point.
• 36:44
  But then you want to start annotating things, right?
• 36:47
  Because we established like you really want to have these, this
• 36:52
  notion of what's good and what's not good.
• 36:54
  So I can share this with you. You can go in and annotate, you
• 36:59
  know, different samples. I can annotate samples.
• 37:01
  I can send it out to a, you know, external, you know, human
• 37:04
  annotator as well. And then we can also collaborate
• 37:08
  on which annotation categories that we care about because it's
• 37:12
  never as simple as like this is good, this is bad, right?
• 37:15
  Like it's, it's more complex than that.
• 37:18
  And then the last like call it the third main kind of view is
• 37:24
  the call it the observability view.
• 37:26
  So we put this in production and like, you know, you're a product
• 37:30
  builder and engineer, you want to see how people are using it.
• 37:34
  So with OPEC, you can actually see all these sessions and then
• 37:39
  for each one, you can actually see a full breakdown of the
• 37:41
  traces in the span. So if you have tools that you're
• 37:46
  calling within your chain, you can see what their response is.
• 37:49
  So you get this full observability of of how people
• 37:52
  are using it. And then, you know, I can select
• 37:54
  A specific trace and, you know, share this with you Demetrius
• 37:57
  and say, hey, like this seems odd or we should fix this.
• 38:00
  So yeah, it's been pretty cool to see how many people use it.
• 38:04
  The product's only been officially out for we're, you
• 38:07
  know, recording this in October 4th has been like 2 weeks
• 38:11
  already reached I think 1500 stars in Hithub and hundreds of
• 38:16
  users. So pretty awesome to see how
• 38:18
  people use it. Yes.
• 38:20
  And I definitely saw it when it came out and I was very curious
• 38:25
  and that's why I'm happy that we're having this conversation.
• 38:27
  I also wonder if you can, because I wasn't clear on this.
• 38:33
  Are you are you getting a snapshot of someone's whole?
• 38:39
  Like we, we talked about how there's all these X factors or
• 38:44
  knobs that you can tune when you're dealing with your rag
• 38:49
  pipeline. Let's stay centred on like a rag
• 38:53
  use case and you've got the the way that you're chunking it.
• 38:56
  You've got the RE reign career, you've got the embedding model,
• 39:00
  the model that the actual LLM you're using, the provider that
• 39:04
  you're using. All of that feels like it should
• 39:07
  be caught in a snapshot if you want to reproduce what you want
• 39:11
  to reproduce. Are you getting all that
• 39:13
  information too? Yeah, absolutely.
• 39:16
  We have this notion of experimental configuration,
• 39:19
  which is exactly those things, right?
• 39:21
  Like it's a snapshot of all these knobs and parameters so
• 39:27
  you can easily reproduce them. But then we go like deeper and
• 39:31
  we can see like, you know, in a rag use cases you want to see,
• 39:36
  you don't know what just want to see what the question and answer
• 39:38
  was. You want to see like what is
• 39:40
  the, you know, what did the vector database return?
• 39:43
  What was kind of pasted into the prompt template So you can
• 39:48
  really dig in and see all those things.
• 39:50
  So which helps iterate on a specific parameter versus the
• 39:56
  whole system as a whole all the time, right?
• 39:58
  Like maybe it's your trunking, maybe you want to add
• 40:01
  traditional BM25 input as well, which is, by the way, it's seems
• 40:06
  to be extremely powerful and useful to to include those
• 40:10
  things. So yeah, called experiment
• 40:13
  configuration and. Then that is exactly what I was
• 40:17
  thinking about is how valuable that can be in the collaboration
• 40:21
  aspect, because I can send you over all of that experiment
• 40:26
  configuration and you can pick up where I left off.
• 40:29
  And so then you can be like, yeah, oh, you know what?
• 40:32
  Maybe you have some inkling of something that two weeks ago you
• 40:37
  were playing around with a different configuration and you
• 40:40
  saw it was cool if you switched out one of these parameters and
• 40:45
  then boom, you see a lift and you get to see some gains there.
• 40:50
  I also wonder about the actual output and that that data set
• 40:57
  that you're talking about and then being able to zoom into
• 41:01
  different pieces of that output and say, you know what?
• 41:05
  This is doing really well on everything except for maybe this
• 41:09
  slice. Yeah.
• 41:11
  And I imagine you've got something like that because that
• 41:13
  also doesn't feel so different than the experiment tracking
• 41:17
  world, right? Yeah, absolutely.
• 41:19
  Right. So on the data sets or the
• 41:21
  experiment side, there's one of the things you always want to do
• 41:25
  is compare against something where there is a goal data set
• 41:28
  or experiment A versus experiment B, right?
• 41:31
  So you have the evaluation metrics and you can look at
• 41:35
  where does you know how does this experiment perform versus
• 41:39
  the other? But when you look at responses,
• 41:41
  you also want to be able to see like that actual text if right,
• 41:44
  like this is the how they responded differently.
• 41:47
  And and then, yeah, you can kind of slice and dice space and
• 41:50
  like, I guess show me all the all their data set items where
• 41:57
  this evaluation score is equal, like 0, for example.
• 42:01
  And that's really in a base. And then you look at the ones
• 42:03
  where you failed in and then you go back and iterate on the
• 42:06
  prompt and you try to fix it. So that's exactly the process
• 42:09
  which is very much, that's our previous point, very much a data
• 42:13
  science oriented process versus the software engineering 1.
• 42:17
  So I can really take that data that is output and I can dig
• 42:26
  deep into each little piece of that and understand what it was
• 42:33
  when it was output this way. OK, All of these, like you said,
• 42:36
  whatever it is, if the evaluation score is 0 and I'm
• 42:39
  looking at all these examples, I can know exactly what the what
• 42:44
  was pulled from the vector database or what kind of
• 42:46
  chunking happened. I can understand if there is
• 42:50
  something in my pipeline that is not working.
• 42:54
  It's a little bit easier to debug that.
• 42:57
  Absolutely, absolutely. That's that's the goal.
• 43:00
  And then you, you mentioned something that's wrong in your
• 43:04
  pipeline, right. So we're, I think that most of
• 43:06
  the, what we're talking about is call it in like this development
• 43:10
  phase, right? And then maybe you have, you
• 43:13
  know, it's an innovative process.
• 43:14
  You put in production, you start getting new production data, you
• 43:18
  label it, you go back to development, right?
• 43:20
  A huge piece here that like I'm very excited about because I
• 43:27
  haven't seen anyone try to approach it the same way we do
• 43:31
  is like the CICD aspect of things, right?
• 43:34
  Because in software engineering, you know, you after you build
• 43:39
  all your tests, every time you release something not even to
• 43:42
  production, right? Like every time you create a
• 43:43
  pull request on GitHub, it, it, there's a build and it, it runs
• 43:46
  through all the tests and it gives you so much confidence
• 43:51
  that whatever you did didn't break someone else's work or
• 43:54
  some old functionality. But it's like, how do you do
• 43:58
  that when you have an LM in place?
• 44:01
  Like, right. So yes, we had, we talked about
• 44:04
  the evaluation metrics, you know, whether they're
• 44:08
  deterministic or heuristic based.
• 44:10
  But like like we really worked with our customers to try to
• 44:14
  figure out like, OK, but what is it?
• 44:17
  How do you use this in the CICD flow?
• 44:20
  So what we ended up doing is we built an extension library for π
• 44:24
  test. So essentially within your
• 44:28
  existing like testing code, you can add additional tests and
• 44:32
  say, OK, this is a heuristic test.
• 44:36
  I am OK if the distance between the answer in my goal data set
• 44:41
  versus the distance from the LLM is lower than some threshold.
• 44:46
  So, and then you get that additional confidence that you
• 44:49
  didn't break anything, right? So even if it's not LLM related,
• 44:53
  if you change something, let's say you're, you know,
• 44:55
  something's broken in your vector database pipeline, like
• 45:00
  yes, you can write some software tests for that, but this is
• 45:03
  really a really great way to have this confidence to ship.
• 45:07
  And then that just allows you to move so much faster because
• 45:10
  you're not worried all the time you broke something.
• 45:12
  So it's pretty, it's really been like really cool to have the
• 45:15
  opportunity to take a concept that is so meaningful for
• 45:18
  software engineering and try to bring it to the LOL world.
• 45:22
  And I'm, I'm just excited to get more feedback.
• 45:24
  I'm sure there's a lot we should fix and improve.
• 45:26
  So it's been really fun. Well, the thing that keeps
• 45:30
  screaming in my head is feature flags.
• 45:33
  And have you been thinking about how to incorporate because it's,
• 45:38
  it's like a natural thing that you would try and want to have,
• 45:44
  right? Is that all right?
• 45:45
  Cool. We're going to test out this new
• 45:46
  one and we're going to slowly roll it out into production or
• 45:50
  we're going to roll it out just in this area or that area
• 45:53
  because we have 100% confidence that it's doing well here.
• 45:56
  And then we might do a little bit more of a roll out.
• 46:01
  So you're, you're AB testing on that roll out or I heard some
• 46:04
  people call it like whatever the champion challenger kind of roll
• 46:08
  out. Have you thought about any of
• 46:10
  that or is it you're just playing in the evaluation side
• 46:14
  and then reincorporating it back into the the CIC pipelines, but
• 46:19
  not quite going as far as the deployment piece?
• 46:25
  Yeah. So we open doesn't or comment
• 46:29
  for that matter. We don't do any work around kind
• 46:32
  of production AB testing. A lot of people when we say
• 46:36
  experimentation assume that's what we do, but but we know.
• 46:40
  So now it, it might change, obviously, right, depending on
• 46:43
  what users and customers are asking for.
• 46:46
  But that's like the existing platforms and tools for AB
• 46:52
  testing And essentially every company uses something, right?
• 46:57
  I think like translate well here.
• 46:59
  So we're not trying to like like build something that is, you
• 47:04
  know, super established and people have been done before and
• 47:06
  just kind of like put an LLN tag on it, right?
• 47:10
  Like if it's if, if the existing tools work well, like by all
• 47:13
  means, let's use them. It's about like where the
• 47:16
  existing tools don't work as well for that, for this new
• 47:21
  paradigm where we get excited about adding more functionality,
• 47:24
  that's like does the history of the company as well.
• 47:28
  We had Git version control and but it just didn't cover all
• 47:32
  these other things we needed from experiment tracking.
• 47:34
  So that's kind of so, yeah. So I guess the short answer is
• 47:37
  like we don't do any AB testing feature flags functionality, but
• 47:42
  there's some great tools out there.
• 47:44
  Yeah. But it's like you work well with
• 47:46
  that. And then along those lines, are
• 47:48
  there ways that you've seen folks architect their pipelines
• 47:53
  to have them be more bulletproof or resilient, maybe not
• 47:58
  resilient, but just be more accurate and have more success
• 48:02
  in their AI products? Yeah, I mean, there's a bunch of
• 48:08
  ways we can break this apart, right?
• 48:10
  Like, and there's obviously a lot of knowledge in data
• 48:15
  engineering, software engineering of you know, how to
• 48:18
  build a reliable data pipeline. So I probably not much to add
• 48:23
  there. I would say like without, you
• 48:25
  know, try not to repeat myself, but like I think the main thing
• 48:29
  that adds this new layer as around including automated
• 48:33
  evaluation testing for your LLM outputs, that's like the new
• 48:38
  thing that is hard to do with the existing knowledge that we
• 48:44
  have around data and software engineering.
• 48:48
  And I've seen teams that adopt that approach and suddenly just
• 48:52
  move 10X faster. So that's been really amazing.
• 48:55
  And there's this notion of the the evaluation.
• 49:03
  Basically, one thing that's hard with evaluation is the human
• 49:08
  labelling and how expensive that can get, right?
• 49:12
  Have you been seeing anything that will help that besides the
• 49:17
  LM as a judge type of thing? Or is it just, you know what?
• 49:22
  The best teams hold labelling parties and they they embrace
• 49:28
  the whole labelling piece of it. Yeah, I mean, there's sure,
• 49:32
  there's like ways they can get closer and but at the end of the
• 49:35
  day, you're right. Like good human label data is
• 49:41
  priceless. And it's it's very similar to
• 49:44
  traditional, right. It's like there's definitely
• 49:45
  companies that will outsource that work for you if you're a
• 49:51
  smaller team. We don't have budget labelling
• 49:53
  parties are always a great idea. But yeah, like having even like
• 49:58
  it's just, you know, crazy. Even a small amount like even
• 50:01
  like 50 or 60 samples on like on AQ and a rag use case will help
• 50:08
  you so much like so definitely recommend.
• 50:11
  If you don't have that, just go ahead and label a little bit of
• 50:14
  data. It also gives you a lot of
• 50:16
  insights once you start doing that about how to design your
• 50:19
  prompt. And have you seen differences
• 50:22
  between like internal versus external facing LLM applications
• 50:27
  and how to evaluate things there?
• 50:31
  Because I, I feel like you can get into hot water on both of
• 50:35
  them. And so maybe there's, there's no
• 50:38
  difference. Maybe with the internal stuff
• 50:42
  you have a little bit more leeway because you're not going
• 50:45
  to get thrown on the front page of the Internet, or at least
• 50:50
  unless you're. Employed.
• 50:53
  Yeah, yeah. But but I don't know, maybe you
• 50:57
  you there isn't a difference and it should be taken with a lot of
• 51:02
  rigour on both cases. Yeah, I mean, it depends on the
• 51:06
  use case, right? Like, and I think most companies
• 51:09
  for that reason start with like internal use cases.
• 51:12
  It's less, there's less risk in there.
• 51:15
  But I mean, look, there's, you know, there's companies building
• 51:18
  internal HR chat bots or rag chat bots.
• 51:22
  And theoretically like one can just answer questions about
• 51:26
  other people's compensation, right.
• 51:27
  So there's, there's risk there as well.
• 51:29
  That's that's one thing. The, the bigger difference and
• 51:33
  the main one I would say is the usually the scale, right?
• 51:37
  Like if you're putting something in production in front of like
• 51:40
  hundreds of thousands or millions of users, it's just,
• 51:44
  you know, it gets much, much harder in many ways.
• 51:47
  So that's more of an engineering thing.
• 51:50
  Yeah. But it is very cool to see like
• 51:51
  some of our customers starting with internal use cases, getting
• 51:55
  a lot out of it, moving to like beta where some some of their
• 52:01
  customers have access to it, and then finally putting in a
• 52:04
  production. It's such a good feeling to see
• 52:07
  like someone does that and like, you know, as a builder, like
• 52:10
  it's so hard to build things that are good and bring value
• 52:14
  and people are excited about and seeing our customers do that.
• 52:17
  Like, like literally what gets me up in the morning.
• 52:21
  And when you talk about the scale between the two internal
• 52:25
  versus external, I also since we are really harping around
• 52:30
  evaluation, how do you manage the scale and doing proper
• 52:37
  evaluation at scale? Yeah.
• 52:41
  So the first thing we had to do when we built OPEC is like
• 52:46
  support that skill, right? Because if, if you have, you
• 52:49
  know, hundreds of thousands of users or millions of users and
• 52:53
  you know, I don't know a very significant number of QPS like
• 53:00
  we and we want to do the observability side for that.
• 53:03
  Like we have to support that. It's like, so OPIC was built
• 53:09
  from the ground up based on a lot of the knowledge we already
• 53:12
  had around model observability to support that.
• 53:15
  So that's the first thing that's hard, right?
• 53:17
  Like how do we collect all these instrumentation information from
• 53:21
  you in real time without slowing down your application in scale?
• 53:25
  So that's the first piece. I mean, if if you are listening
• 53:29
  and you like that kind of work, definitely take a look at what
• 53:32
  we did there on the implementation side.
• 53:35
  We would love to get some feedback.
• 53:37
  We're we're pretty excited about it.
• 53:39
  But then, but then you're right, I mean just because like the
• 53:42
  evaluation is also really hard. So all these queries we talked
• 53:46
  about slicing and dicing the data, running automated
• 53:49
  evaluations, it just becomes much harder and scale like many
• 53:53
  things. And but this is where this world
• 53:56
  is going to, right? Like we're talking about
• 53:59
  features and, and use cases that will be served to like millions
• 54:03
  of customers. And maybe it's a nature of our
• 54:06
  existing customer base where we serve, you know, some of these
• 54:09
  very, very large, you know, I think about Netflix and Uber and
• 54:13
  the amount of users that they serve that we have to think
• 54:16
  about those things. But it's it's really hard and
• 54:19
  we're still early on both OPIC in an industry in general.
• 54:23
  So I think if we meet again in a year from now, I'll be able to
• 54:27
  tell you a lot more and what breaks when you get to that
• 54:30
  scale. And one thing that is classic
• 54:33
  observability type of feature or best practise is setting up
• 54:39
  alerts. Have you, do you have that
• 54:42
  ability with OPIC and have you thought about that type of thing
• 54:46
  because that's a good. Question.
• 54:48
  It's also, yeah, but too, right, because you don't really know
• 54:52
  how do you alert like when when your software goes offline,
• 54:55
  that's one thing you can alert on.
• 54:57
  But when the LLM is giving kind of bad output, are you going to
• 55:01
  alert for that? How do you alert for that?
• 55:03
  What does that even look like? Yeah, it's a good point.
• 55:06
  So we don't, OPEC doesn't support alerts today.
• 55:09
  It very likely will soon. Alerts, like you said is are
• 55:15
  really hard because you very quickly start going into this
• 55:19
  world of like alert fatigue or, you know, too many false
• 55:22
  positives, right? And we've seen we have a
• 55:24
  reservability product for traditional MO around feature
• 55:27
  corruption model and concept drip.
• 55:31
  So it's very hard, right? Like if I, if I tell you 6-7 or
• 55:34
  even five times that something's wrong in your model and you
• 55:37
  didn't take a look and it's fine.
• 55:39
  And the 6th time where something's actually broken,
• 55:42
  you're just going to ignore the the alert.
• 55:43
  Yeah, it's the boy who cried wolf.
• 55:45
  Yeah, exactly, absolutely right. So I wouldn't say we have
• 55:49
  figured that out exactly, but the idea is like using OPIC is
• 55:54
  like, yes, we give you a lot of built in evaluation functions
• 55:58
  and metrics, but you can and probably should in many cases
• 56:02
  implement your own. So if you know of something
• 56:05
  that's absolutely critical for you, like I don't know, I've
• 56:09
  seen one of our customers wanted to make sure that chatbot does
• 56:13
  not mention their competitor. Fair enough.
• 56:17
  So that's in a simple assertion, right?
• 56:19
  Does not contain X then you can definitely do alerts on that.
• 56:23
  So yeah, it's a good point. I think it's on the road map of
• 56:26
  the on the open open source road map there.
• 56:29
  But if not, I'll make sure to add it later today.
• 56:32
  Well, also the thing about alerts, I was trying to think
• 56:37
  how you would be able to today just capture value from
• 56:46
  understanding the output of the models.
• 56:51
  And there are those hard lines that you don't want to take,
• 56:57
  such as don't mention a competitor or if this is some
• 57:02
  kind of question answer by don't like respond to people if they
• 57:06
  say give me the answer in Jason or in Python, you don't want to
• 57:10
  be doing that because as you saw probably with the Amazon sales
• 57:16
  bought it was doing that. And then that's a great way to
• 57:18
  end up on the front page of the Internet, as we say.
• 57:21
  But one thing that I was thinking about that could be
• 57:25
  very valuable is if all of a sudden I'm seeing a lot of folks
• 57:34
  ask the same question or it feels like there's some kind of
• 57:39
  anomaly in the output, maybe that could be worth looking
• 57:44
  into. Like, hey, all of out of nowhere
• 57:47
  in the past X amount of time, minute or hour, we've seen an
• 57:55
  unusually high rate of this or that.
• 57:59
  That could be valuable because it could be another.
• 58:02
  It's like a proxy signal, right? It's some telling you something
• 58:06
  is is happening differently than usual and maybe you want to go
• 58:11
  and look into it. And that was the only thing that
• 58:13
  I could think of that could be a nice alert if you understand the
• 58:16
  output. Absolutely.
• 58:19
  Yeah, We've seen a lot of those things on the our MPM model
• 58:23
  production monitoring product, which is, you know, designed for
• 58:28
  like teams that control the model versus like an LM
• 58:31
  application. So yeah, it'll be actually
• 58:33
  pretty cool to see how those things going to converge.
• 58:37
  I would say like the majority of use cases I've seen out there
• 58:40
  are like very early meaning like that kind of observability is
• 58:45
  they're not even thinking about this.
• 58:47
  They're just trying to get something working and it's basic
• 58:49
  form. But I have no doubt we'll get
• 58:52
  there, right? Like if you think about on the
• 58:55
  software side, like the APM, the data dog with neural relics in
• 58:58
  the world, I have no like no doubt we'll like absolutely get
• 59:03
  there eventually. Well, I remember 1.
• 59:05
  Year, it might take 10, but it will get there.
• 59:08
  Yeah, I, I just say this because I remember the conversation that
• 59:12
  I had with Phillip from Honeycomb probably a year and a
• 59:18
  half ago. And he was talking about how it
• 59:21
  became very obvious to them once they put the AI product or AI
• 59:27
  feature into their product when somebody was trying to prompt
• 59:30
  inject the feature because it was almost like a programmatic,
• 59:37
  there was a lot more queries per second.
• 59:40
  And or there was just like, it was pretty obvious it wasn't a
• 59:44
  human that was trying to do it. And so those are things where
• 59:49
  you can set them up. Maybe you don't need a tool like
• 59:52
  an evaluation tool. You can just set those hard
• 59:55
  rules like hey, if a user all of a sudden is querying 50 times
• 1:00:00
  per second, then that's probably not a human.
• 1:00:02
  Yeah. And I think in that case they
• 1:00:04
  also have, it's an interesting LLN use case because like
• 1:00:10
  you're, I think if I'm not mistaken, they're generating
• 1:00:13
  queries for the user like a text to sequel ish use case though
• 1:00:18
  you could you actually know whether the query is like
• 1:00:21
  syntactically correct, right, for example.
• 1:00:24
  So that's a super easy alert, right?
• 1:00:26
  Like if your model is generating non syntactically correct
• 1:00:29
  queries, I I'm not even talking about whether the query returns
• 1:00:33
  what the user expected. That's that's harder, but like,
• 1:00:36
  you know, syntactically correct. And that's like classic OP use
• 1:00:39
  gets, implement your own metric. I think we even have one for for
• 1:00:45
  SQL query, but implement your opaque metric and add something
• 1:00:49
  around that. And then you can have these
• 1:00:51
  alerts or even just this information on the UI.
• 1:00:54
  That's like, these are the use cases that break.
• 1:00:59
  It's a while. So it's almost like you can
• 1:01:02
  today get alerts, you just have to do a little bit more creative
• 1:01:05
  thinking. Yeah, yeah, there's no alerts.
• 1:01:08
  It's like built in. I mean, you can forward the logs
• 1:01:11
  and so on. If you want to hack it there,
• 1:01:13
  I'm sure there's a way to do it. But I I definitely urge the
• 1:01:17
  listeners if you want to go and open a pull request for an
• 1:01:20
  alerting. System.
• 1:01:21
  There we go. Integration to slot pager duty
• 1:01:24
  and such, we would love to accept it.
• 1:01:26
  Yeah, it's an opportunity. Right now.