Latest / Agentic Conversations (formally mlops.community) / Product Metrics are LLM Evals // Raza Habib CEO of Humanloop // #320

Transcript

Product Metrics are LLM Evals // Raza Habib CEO of Humanloop // #320 · Agentic Conversations (formally mlops.community)

• 0:07
  how you are looking at product metrics and combining those with
• 0:13
  evals. So I think that there's no real
• 0:16
  difference between product metrics and evals, or at least
• 0:19
  like the best evals are the same as product metrics.
• 0:22
  What do I, what do I mean by that?
• 0:24
  So I think a lot of what you're trying to solve with evals is
• 0:27
  you're trying to get a measure of the quality of your AI
• 0:30
  system. And what's changed with
• 0:32
  generative AI and large language models is a lot more of the use
• 0:35
  cases are very subjective to measure performance on.
• 0:38
  If you are like writing an e-mail or summarizing a document
• 0:41
  or answering a question, there isn't like one right answer
• 0:44
  anymore. And so it's hard to say like
• 0:46
  what even is the correct answer. Like take a concrete example of
• 0:49
  summarization. Meeting summarization is so
• 0:51
  contextual as to whether or not it's a good summary.
• 0:54
  If you have a sales call, you want a different type of
• 0:56
  summary. Different salespeople will care
• 0:58
  about different things like did you pick up key information?
• 1:01
  So there's no one correct answer anymore.
• 1:03
  So in some sense, like the best eval, like the value would you
• 1:06
  wish you could have in practice would be the end user outcome.
• 1:10
  Like did the user get the outcome that they wanted?
• 1:12
  Were they able to achieve their goal?
• 1:14
  Did they use the tax that they like it that they, you know,
• 1:17
  depending on the use case that did the agent achieve the goal
• 1:19
  it was supposed to do for the person and the way that they
• 1:21
  expected? Was the customer support message
• 1:24
  actually helpful? And what evals are really trying
• 1:27
  to do is give you a proxy for that during development because
• 1:31
  you can't have that necessarily before you put stuff into
• 1:34
  production. And so hopefully those two
• 1:37
  things are very correlated and product metrics and evals are
• 1:41
  not that different. And a lot of the evals that
• 1:43
  people are creating with LM is judge or like automated evals
• 1:47
  are often run as production metrics as well, right?
• 1:50
  So on every utterance they're actually scoring it with either
• 1:53
  a classifier or a piece of code or an LM is judge and then
• 1:56
  monitoring that for how things are drifting over time or to be
• 2:00
  alerted if something changes. And do you see this as like the
• 2:02
  happy path in whatever the product is?
• 2:06
  There's a next step that I'm trying to take.
• 2:09
  And if that gets accomplished after I have some interaction
• 2:13
  with AI, then that's a win. And you can easily see that's
• 2:18
  black and white, like I took that step.
• 2:20
  I I made the next best action or I didn't?
• 2:24
  Yeah. So in practice, the we see like
• 2:27
  3 very common types of sort of like in production evals as like
• 2:33
  default end user metrics and then and then ones that are more
• 2:35
  bespoke or kind of use case specific.
• 2:37
  So the most like basic ones that everyone does, you know, you see
• 2:42
  it, you see them in lots of apps.
• 2:43
  So like the simplest thing is just end user feedback, thumbs
• 2:45
  up, thumbs down or scores, things like that.
• 2:48
  They're somewhat useful. You don't actually get as much
• 2:49
  of it as you would want, and it biases towards the extremes.
• 2:52
  I always laugh at those because it's like the people that are
• 2:55
  given the thumbs up and thumbs down are just a very small
• 2:58
  subset of your user base. But it's still it's still useful
• 3:01
  data, right? So something we'll often see
• 3:02
  people do in human loop is like look at the examples that got a
• 3:05
  bad rating because they want to try and understand like really
• 3:08
  where are the failure modes? Like where is the system going
• 3:10
  wrong? Then you get corrections.
• 3:14
  So if people are like generating text or getting an answer to
• 3:18
  something, you know, they're editing a legal document or
• 3:20
  something like that and being given suggested edits, then they
• 3:23
  probably like edit the text before they use it a little bit.
• 3:26
  And so having that like correction can be very useful
• 3:28
  signal both like how much do they correct it, but also as a
• 3:32
  as a stored data point of what the right answer was in that
• 3:35
  case. And then finally what you said
• 3:37
  of like actions like what is the natural next action?
• 3:41
  And does the user take that? Do they send the message?
• 3:44
  Do they copy the text? You know, whatever it is
• 3:46
  contextual in your application. Those we see people use a lot as
• 3:50
  baseline. And then things that we see on
• 3:52
  top are people will often build custom LMS judge or code
• 3:56
  evaluators that are very use case specific.
• 3:59
  And those might be monitoring tone of voice, whether or not
• 4:03
  there is, you know, stuff is on brand or stuff like this that
• 4:06
  that may be a little bit more subjective to monitor.
• 4:09
  And they they might want that only in development, or they
• 4:11
  might not want that in development and in production as
• 4:13
  well. Yeah, because when you get into
• 4:15
  the enterprise, they can't just Yolo things into production.
• 4:19
  Yeah, exactly right. So, so when we built the first
• 4:23
  version of Human Loop, like the initial focus was all around
• 4:27
  helping people to measure how well the system was working in
• 4:29
  production and use that to guide improvements.
• 4:31
  And that's still a core piece of the platform.
• 4:34
  But as we went upmarket and we started to sell more to mid
• 4:37
  market and enterprise companies, then, yeah, you have this
• 4:39
  problem that like you have to be able to be confident that it
• 4:42
  works well in development before you're willing to, you know,
• 4:45
  roll it out even to a beta group or to all your users.
• 4:48
  And so you need some way of actually building a rigorous set
• 4:50
  of testing metrics in dev that hopefully are predictive of what
• 4:54
  will happen when you actually deploy to production.
• 4:56
  So basically you went in the human loop with that idea and I
• 5:00
  like that you talk about the evolution and you also see how
• 5:03
  things change. I talked to you what like 3-4
• 5:06
  years ago when human limp was starting.
• 5:09
  Just over 2 years ago, yeah. So it was about it was it was
• 5:12
  just a couple of months before ChatGPT came out actually, as I
• 5:15
  think we recorded that episode. Classic.
• 5:17
  Great timing and. Yeah, that that is.
• 5:19
  That is roughly how I split the world now, before and after
• 5:21
  ChatGPT. Some people it's Christ, the
• 5:25
  others exactly it's ChatGPT. Yeah, BC it was it was about 3
• 5:29
  months BC. Yeah.
• 5:30
  So what have you changed? How have you like seen the
• 5:36
  product develop and grow and you're thinking around AI in
• 5:42
  production in general? Yeah.
• 5:44
  So you know, when we started, LMS are still very new.
• 5:48
  They're mostly being used just by startups.
• 5:50
  The applications were much more limited, so people were building
• 5:53
  like the single prompt based apps.
• 5:55
  It was, you know, the earliest cases were copywriting or
• 5:57
  various forms of writing assistant user comes to the
• 6:00
  website, describes what they want, spits out a blog post or
• 6:02
  helps them, you know, brainstorm.
• 6:05
  And the models were kind of just about good enough for that.
• 6:07
  But for more complicated use cases like question answering or
• 6:10
  agents, people were trying and building demos.
• 6:13
  And you know, even even 2 1/2 years ago, like you had the
• 6:15
  first agent demos, but the models weren't smart enough to
• 6:18
  make that reliable. And so the first version of
• 6:21
  human Loop was really focused on helping people who are building,
• 6:25
  you know, this first generation of LM applications to understand
• 6:28
  how well they were working in production.
• 6:29
  Because if you can't measure it, it's very difficult to improve
• 6:32
  it, know how to make it better over time.
• 6:35
  And subjectivity and stochasticity was like the
• 6:38
  challenge, right? And it's still the challenge.
• 6:40
  But if you think about copywriting or writing
• 6:41
  assistance, you know, that's an extreme point of like evaluation
• 6:45
  is very subjective. And for a lot of engineers, it
• 6:47
  was the first time that they were building software where
• 6:50
  every time you run it, you get different results, right?
• 6:52
  People in the machine learning community, this is relatively
• 6:54
  normal for them. But I think that if you are not
• 6:57
  from a machine learning background and that was actually
• 6:59
  quite a new experience. And so the the first version of
• 7:02
  human was very simple. It allowed you to monitor what
• 7:06
  was happening in in production. So basically every time someone
• 7:08
  was interacting with your system to get the inputs and the
• 7:10
  outputs, gather and user feedback and match that against
• 7:13
  different prompts. So you could kind of do AB
• 7:15
  testing or comparisons of prompts in production.
• 7:19
  What's changed since then is as the models have gotten smarter,
• 7:22
  the complexity of the use cases has gone up enormously.
• 7:25
  So we've gone from like simple writing apps to people are
• 7:28
  automatically negotiating legal contracts, right?
• 7:30
  So people like File, Vine and Antra are our customers, or
• 7:33
  we've got people building language learning assistance at
• 7:35
  Duolingo or at Gusto. They're building, you know,
• 7:38
  complex agents that can actually go in and do tasks that
• 7:42
  accountants were previously doing in payroll software.
• 7:44
  So we've gone a long way from hey, just generate me a blog to
• 7:48
  actually good do something complicated for me.
• 7:51
  Multi. Step yeah.
• 7:52
  And so we've had to build out, you know, still the same.
• 7:56
  The, the fundamental problem is the same, which is that I'm
• 7:59
  trying to take this unreliable base thing, the LLM, and build a
• 8:03
  reliable, trustworthy application on top of it.
• 8:06
  But the stakes have gone up, right as we started to work with
• 8:09
  enterprises. They care a lot more about
• 8:12
  avoiding mistakes, about having guardrails in place.
• 8:15
  The complexity applications have gone up.
• 8:17
  So tracing and observability actually becomes way more
• 8:19
  important. The way that people build has
• 8:22
  changed as well. So original human loop was just
• 8:25
  this in production, gathering feedback.
• 8:28
  Since then, we built that out into a full tracing and
• 8:30
  observability system. So if you have an agent that's
• 8:32
  going off and doing things, it'll let you see the full path
• 8:35
  of what did that system do like so user asked the question, then
• 8:39
  every turn of the conversation or every step that the agent
• 8:42
  takes, you have that recorded and you can augment that with
• 8:45
  feedback or evaluation data. So you have this very rich data
• 8:48
  set that allows you to do both governance but also improve your
• 8:51
  system. Like what is and isn't working
• 8:53
  and how do I guide that? And then coupled to that, I
• 8:56
  think what's become the core of the application actually is an
• 8:59
  evals tool. And this is really used
• 9:02
  primarily in development, but what it allows people to do is
• 9:05
  to write down essentially the criteria that the system needs
• 9:09
  to pass to be good and have a data set of test cases that they
• 9:13
  can then iterate against. So LM development is very
• 9:16
  iterative. You build the first version of
• 9:18
  the pipeline as quickly as possible, get something that
• 9:20
  works end to end. And then you're looking to go
• 9:22
  from something that works end to end to something that's good
• 9:24
  enough to trust in production. And you're iterating over
• 9:27
  prompts, you're doing over tool definitions, over preprocessing
• 9:30
  and POS processing code. You're trying out different
• 9:33
  models, you're trying out different data, store the data
• 9:36
  sources for the context. If you don't have a way to know,
• 9:39
  as you make those changes, am I making things better or worse?
• 9:43
  That's very difficult to actually make progress and not
• 9:45
  just spin your wheels. And So what evals do is they
• 9:48
  allow you to get quantitative feedback on every change that
• 9:51
  you make so that you can actually tell, like, am I
• 9:54
  improving things? And that's really helpful during
• 9:56
  development to actually get to a system that works well.
• 9:59
  And I think we spoke alongside file via Net like recent AI OPS
• 10:03
  conference that you guys had organized and they were able to
• 10:06
  take a system, you know of just over 50% accuracy on doing
• 10:09
  information extraction to 90% plus.
• 10:12
  But it would have been impossible without this constant
• 10:15
  feedback loop. Yeah, like evals were the most
• 10:19
  important thing, huh? Yeah, at least at least for
• 10:21
  them. And I think, I think it's, it's
• 10:24
  one part amongst many, but it's it's kind of an essential
• 10:27
  baseline. Like if you don't have good
• 10:28
  evals, I don't think you're going to be able to get very far
• 10:32
  on the other pieces. So it's foundational.
• 10:35
  Stay with me here. Yeah?
• 10:38
  Have you heard the parable of the scorpion and the frog?
• 10:43
  I have, But you know, maybe we should, maybe we should remind
• 10:46
  the audience. All right, so, so there's this
• 10:49
  parable of a scorpion and a frog and the scorpions trying to get
• 10:53
  across the river. And so it sees a frog and it
• 10:56
  says, hey, will you give me a lift?
• 10:58
  I need to get across the river, so can you take me?
• 11:01
  And the frog is like, yeah, but you're a scorpion and you got a
• 11:05
  tail. And if you sting me, then I'm
• 11:07
  going to drowned and I'm going to die.
• 11:11
  And the scorpion said, don't be ridiculous.
• 11:14
  Why would I sting you if I'm on your back?
• 11:17
  And then you drowned, and I also drowned.
• 11:19
  And so the frog thinks about it for a SEC and goes, OK, I'll buy
• 11:24
  into that one. Yeah, get on.
• 11:25
  And so the frog's taking the scorpion across the river.
• 11:31
  The scorpion can't help itself. It just boom.
• 11:34
  At a certain point it stings the frog and they both go down and
• 11:39
  they drowned. And the reason I'm saying this
• 11:41
  as? They're drowning, the frog says
• 11:43
  to slip. And why did you sting me?
• 11:45
  And he says it's just in my nature.
• 11:46
  That's it. That was the that's the key
• 11:48
  part. Yeah.
• 11:48
  It's in my nature, right. And the reason I think about
• 11:52
  that story is because a lot of times I wonder if these LLMS
• 11:58
  were caging them and trying to put all of these guard rails
• 12:03
  around them and all of this different stuff.
• 12:05
  We're doing all this wild engineering, like evals and like
• 12:09
  everything that we can muster up when these hallucinations are
• 12:15
  just part of the nature of the LLM and we don't like them, but
• 12:21
  we have to figure out a way to like contain them.
• 12:24
  Yes, I feel like there's maybe multiple things going on here
• 12:27
  and there's there's a few things to talk about.
• 12:29
  Let's put a pin in the hallucination question.
• 12:31
  I really want to come back to that.
• 12:33
  But I think there's an extent to which sometimes what people are
• 12:36
  doing is working around the limitations in the current
• 12:39
  generation of models, right? So some of what people are doing
• 12:42
  is oh, the system is like not reliable enough or you know, the
• 12:47
  models aren't smart enough. So we have to figure out a way
• 12:49
  around that. And that's a stopgap solution.
• 12:52
  But I think even if you had the model to be arbitrarily smart,
• 12:56
  like let's say you just have an API to Einstein or whatever
• 12:58
  Dario says his data center of geniuses.
• 13:02
  I don't think the need to like try and iterate on your system
• 13:05
  to achieve your goal goes away. You still have to figure out
• 13:08
  like, what is the context the model needs to be able to do
• 13:11
  this goal? Is it doing the goal in the way
• 13:13
  that I expect? If I change things over time, is
• 13:16
  that still true? So, you know, opening eye
• 13:18
  release is a new model and I want to try it out.
• 13:21
  Like, what is the impact of that change on my metrics?
• 13:24
  Like maybe the accuracy of a system goes up, but people don't
• 13:27
  like the tone of voice as much or something like this.
• 13:30
  So it's not necessarily about mitigating limitations of the
• 13:35
  system so much as like even if you have the smartest model in
• 13:38
  the world, it's always going to be stochastic, right?
• 13:42
  Like by nature of these systems, and we're not trying to get rid
• 13:44
  of that stochasticity, that's often a benefit.
• 13:48
  And it's always going to have like slightly different
• 13:50
  characters with the models. Because I think the, again, sort
• 13:53
  of sticking to lines from Dario, since I think he has some good
• 13:56
  lines and he says we don't so much build these systems as grow
• 13:59
  them. There's a certain
• 14:00
  unpredictability in the character of different models.
• 14:03
  And that's a feature, not a bug, because it means we get lots of
• 14:06
  different personalities or styles or skills or variances
• 14:09
  and ability that I think people like and can make use of.
• 14:13
  I don't think that takes away the need for evaluation if
• 14:17
  you're coming. Like if you're a software
• 14:18
  engineer, you're writing unit tests, you're writing
• 14:20
  integration tests. Like why would it be any
• 14:22
  different with AI? It's just that the types of
• 14:24
  tests you need to write are different.
• 14:25
  And in fact, if you come from a machine learning background,
• 14:28
  like you were always building a test set of cases and measuring
• 14:30
  metrics on them, It's just that the types of metrics and how you
• 14:33
  do it are changing. And then on the point of
• 14:36
  hallucination, when I speak to people like this colloquially, I
• 14:39
  find it very hard to get out of people like a coherent
• 14:41
  definition of what they mean by hallucination as opposed to just
• 14:44
  the model being wrong. Like when you talk about
• 14:47
  hallucination, you're implying an intent to the model.
• 14:51
  And so, like most people, I don't think have a clear idea in
• 14:54
  their head of the distinction between those two things.
• 14:57
  Like maybe, maybe academics have defined a good distinction, but
• 15:00
  when people say the model hallucinated when it answered a
• 15:02
  question, what's the difference between that and it just got the
• 15:05
  answer wrong? Yeah, I guess hallucination is
• 15:09
  the wrong answer at the end of the day, So it makes sense that
• 15:13
  they're one in the same. But the the idea there then is
• 15:21
  mitigating the wrong answers, not mitigating the
• 15:24
  hallucinations. In a way it's like, let's just
• 15:27
  figure out how to be more consistent with right answers.
• 15:31
  And we can do that by, like what you're saying, having this
• 15:35
  overview of how the system is changing over time.
• 15:40
  I feel like when people use the word hallucination, it implies
• 15:43
  something about their mental model of like what the LLM is
• 15:45
  doing. And I'm not sure exactly what
• 15:48
  that mental model is that they're thinking of, but it
• 15:52
  feels at least conceptually muddy to me.
• 15:56
  Like, I don't think people are clear on what they mean, and I
• 15:58
  think it would just be clearer to say like that LLM is being
• 16:01
  factually inactive. It's getting things wrong.
• 16:03
  Yeah. I like that on the inside of
• 16:06
  what you've built, you've got this system running, right?
• 16:09
  And you're recognizing that there's changes that are
• 16:12
  happening to the system continuously.
• 16:15
  But then there's the production environment and there's logs and
• 16:20
  traces that are coming back, and it goes to those product
• 16:23
  metrics, like you're getting product metrics back, you're
• 16:26
  getting information back. So how are you using that to
• 16:29
  update your system and how are you making it easy for that loop
• 16:34
  to just continuously be happening and ideally like get
• 16:40
  better each time? Yeah.
• 16:41
  So maybe maybe this is a good point for me to kind of explain
• 16:44
  the different pillars of what's in human loop, because I think
• 16:46
  understanding like what we've put into the LM Evals platform
• 16:49
  and why we chose those pillars kind of answers this question.
• 16:52
  So there's like really 3 core pillars.
• 16:53
  We've talked about two of them. So one is the in production
• 16:56
  tracing and observability, The other is like having data sets
• 17:01
  and evaluators that you can run during development.
• 17:03
  And then the final pillar that we haven't talked about, but
• 17:05
  it's kind of the answer to this question is tools for managing
• 17:09
  prompts, iterating on them, collaborating between the
• 17:13
  technical people and the non-technical people who have
• 17:15
  the domain knowledge, maybe to be able to change a prompt or to
• 17:19
  to like look at an output and sort of score it in some way.
• 17:22
  So some concrete examples here would be, you know, we mentioned
• 17:26
  Filevine earlier. At Filevine, it's like lawyers
• 17:28
  who are doing a lot of the prompt engineering.
• 17:31
  At Duolingo, it's like language learning experts at fundraise,
• 17:35
  they have their kind of real estate experts and product
• 17:37
  managers be involved in writing the prompt.
• 17:39
  So, so at a lot of our companies, the people who are
• 17:42
  either looking at the evals and scoring them, you know, Gusto,
• 17:45
  its customer support staff, or the people who are writing
• 17:47
  prompts are not the engineers themselves.
• 17:50
  And so part of the platform as well is separating out the
• 17:54
  configuration, which is the prompts and the tool definitions
• 17:56
  from the code so that you can iterate, you can iterate on the
• 18:00
  prompts independently, but still have them versioned and tracked
• 18:03
  in the way that you would have with traditional software.
• 18:05
  So those are the components. And then now we can kind of
• 18:08
  answer your question of like, OK, well, I'm monitoring all of
• 18:10
  this stuff in production. How do I actually use that to
• 18:12
  make things better? And the usually the cycle is
• 18:15
  that someone will be looking at an eval or they'll be looking at
• 18:18
  logs from production and they'll have some hypothesis about why
• 18:22
  the system isn't working the way it's supposed to be working or
• 18:24
  something that's wrong. Like so they'll spot inaccuracy
• 18:27
  in a rag system, or they'll spot the tone of voices wrong or
• 18:30
  something like that. And they'll this.
• 18:32
  Person is usually technical, non-technical.
• 18:34
  We actually see it with both. So it's often PMS. So PMS sort
• 18:37
  of bridge that gap or it might be the engineers themselves
• 18:39
  looking at the the eval logs. They will then say, OK, I think
• 18:43
  what's going on is that the model is not getting the context
• 18:45
  it needs right. My guess is that the problem is
• 18:48
  that actually the RAG system is like not retrieving enough.
• 18:50
  We should go in like improve the recall of the RAG system.
• 18:55
  Well, they can then go make that change within human loop, right?
• 18:57
  They can actually update the configuration of the tool, or
• 19:01
  they can go and change a prompt, or they can trigger a fine tune
• 19:04
  of a model. And then they can run the eval
• 19:06
  and see like, did I actually make things better on my metrics
• 19:08
  or not? And then there's the ability to
• 19:11
  deploy that change from there. And so for a lot of the changes
• 19:15
  you might be wanting to make, which are changes to prompts,
• 19:17
  tool definitions, or trying to run some kind of fine tuning
• 19:20
  job, you can often do that without having to go into code.
• 19:24
  And so that allows the non-technical people to iterate
• 19:26
  very quickly and frees the engineers to do their thing as,
• 19:31
  you know, what they want to do from the PMS and the
• 19:33
  non-technical domain experts. And something we see a lot
• 19:37
  otherwise is APM wants to make a change to a prompt or wants to
• 19:40
  try something. And that might involve a full
• 19:42
  app redeployment. Maybe they have to open a ticket
• 19:45
  and a system like JIRA or Linear to be like, please update this
• 19:48
  prompt. And then an engineer will go
• 19:49
  like, edit that in code, then they'll do the app redeployment.
• 19:53
  It's like, oh, actually that made, you know, that didn't make
• 19:54
  the change I was expecting. Or if they don't have good evals
• 19:57
  in place, then people are nervous to make changes because
• 19:59
  they're worried about regressions.
• 20:00
  I like how you're bridging the gap between all these different
• 20:04
  personas. So it's as technical as you want
• 20:06
  to be, or as subject matter expert.
• 20:09
  It is really, it's really 3 personas that we interact with a
• 20:12
  lot. And maybe this goes as well to
• 20:13
  like what we've seen the most successful teams do when they're
• 20:16
  building products. And so you have your.
• 20:19
  Nice segue by the way. You have, you have your, your
• 20:22
  product engineers or the actual software engineers who are
• 20:24
  building the system, they're making the API calls to the LMS.
• 20:27
  They're actually, you know, building all the pipelines, but
• 20:31
  oftentimes they're not the right people to be able to define what
• 20:35
  good looks like. And they're not necessarily the
• 20:36
  right people to write the prompts.
• 20:38
  And the people who are usually best at that are the product
• 20:41
  managers. You know, writing a spec is not
• 20:44
  that different to writing eval is actually like there's some
• 20:46
  definition of what does good look like.
• 20:48
  And oftentimes there might be a third persona that may or may
• 20:52
  not be the PM themselves, which is the person who has the domain
• 20:55
  expertise to kind of define what the goal is and what good looks
• 20:59
  like. So sometimes that's just a
• 21:01
  product manager, but sometimes it's a separate person.
• 21:03
  And that's where the lawyer comes in and says, wait a
• 21:05
  minute, Yeah, You think this is OK?
• 21:08
  And so the way those people get used is either they're being
• 21:11
  used just to provide feedback. So sometimes what people will do
• 21:13
  is as part of the eval loop, they'll map the system over a
• 21:17
  data set of test cases. They'll have the lawyer or the
• 21:19
  domain expert, whatever the use case is come in, look at each
• 21:22
  output and score it as like pass, fail with a critique.
• 21:26
  So hey, this one is good, it's fine.
• 21:29
  This one was bad because and then that binary data of like
• 21:32
  which ones were good enough? And the reasons then become what
• 21:36
  informs the next round of either prompt engineering or updates.
• 21:40
  And sometimes the lawyer will go to the prompt engineering
• 21:43
  themselves. Sometimes the engineers or PMS
• 21:44
  will do it. But that feedback cycle is
• 21:48
  really useful. And often also it gets used to
• 21:50
  build the LMS as judge. So you know that human feedback
• 21:54
  is invaluable, but very expensive and hard to scale.
• 21:57
  And oftentimes what you can do is take this combination of pass
• 22:00
  fail feedback and critiques and then essentially kind of distill
• 22:04
  that into a prompted LM that you can now use at scale.
• 22:07
  And as long as it correlates highly.
• 22:09
  So the human judgement, you've now got a way to do more
• 22:14
  scalable evaluation that can also be done in production.
• 22:17
  So evals are being done primarily by folks that are
• 22:23
  RPMS. People talk about how data scientists can fill that role
• 22:29
  really well to create these evals to also help explore in
• 22:33
  the prompt exploration space, and it's very akin to what a
• 22:38
  data scientist is used to doing. It's very helpful to have
• 22:41
  someone on the team who has a background in stats or machine
• 22:45
  learning or data science just for the rigor of understanding,
• 22:48
  like how to do evaluation well. Like understands the distinction
• 22:52
  between sort of held out data versus not held out data and
• 22:55
  what kind of metrics you might want to calculate.
• 22:57
  And that you, you know, if you're dealing with stochastic
• 22:59
  systems, you probably want to run them multiple times and get
• 23:01
  averages and stuff that will be like obvious and intuitive to
• 23:04
  anyone from a stats background. But someone who's like not done
• 23:07
  that style of thinking might not even know to think about that.
• 23:10
  And so the typical makeup that we see is usually there's like,
• 23:13
  you know, the normal number of engineers you would have on any
• 23:15
  product team, maybe one data scientist and then the PMS and
• 23:20
  the domain experts and they're collaborating together, right?
• 23:22
  So each of those people is bringing something to the table.
• 23:25
  The data scientists usually don't have the relevant domain
• 23:29
  knowledge to be good at doing the prompt iteration.
• 23:32
  And So what you really want is to be able to give them tools
• 23:35
  that allow the non-technical people to come in and iterate
• 23:38
  without being blocked by either of these other groups and they
• 23:41
  can share information between them.
• 23:43
  The way this works in human loop is that when you want to run an
• 23:45
  evaluation, you map your your application or your LM system
• 23:49
  over a data set. It generates all the outputs and
• 23:51
  then it scores them. And one of the ways it can score
• 23:53
  them is with human feedback. So it creates a queue of jobs
• 23:57
  for the domain experts to come in and actually provide that
• 24:00
  feedback. And then do you also map it to
• 24:04
  the product metrics? So not during development, but
• 24:07
  during production, what happens is that people add some login
• 24:10
  calls or we have integration with open telemetry so they can
• 24:12
  send data to human loop of what is the agent doing.
• 24:16
  And then once those traces land on human loop, people can set up
• 24:19
  evaluators to be in monitoring mode.
• 24:21
  And So what that means is that every time a trace comes
• 24:23
  through, the evaluator says an LLM is judge evaluator will look
• 24:27
  at the output score it. And basically you're building
• 24:30
  this data set of traces augmented with product metrics
• 24:34
  or augmented with end user feedback.
• 24:36
  And so when APM comes into the app, the a typical workflow
• 24:39
  might be OK, show me the cases where the LLM thought this was
• 24:42
  bad. Or show me the cases where a
• 24:43
  user gave me bad feedback. I'm going to look at those and
• 24:47
  try and determine whether I need to go change a prompt or fine
• 24:50
  tune a model or update my rag system.
• 24:51
  I'm going to use that to guide where my failures are.
• 24:54
  And now you said that you can update the rag system from
• 24:57
  within human loop. So there's some there's some
• 25:00
  amount of that people can do within human loop and some that
• 25:03
  might require busting out into more powerful tools.
• 25:07
  So within human loop, people are often specifying configuration.
• 25:10
  So they're specifying chunk sizes like what metrics being
• 25:13
  used to like actually do the retrieval, whether it's like a
• 25:17
  traditional IR method or they're doing something with like
• 25:20
  semantic similarity. So they have a tool definition
• 25:23
  within human loop that's high level configuration for the
• 25:26
  retriever and they can edit that.
• 25:28
  And the way it works is we're kind of like a CMS for prompts
• 25:30
  and tool definitions. So in production, the way it
• 25:34
  works is the production application basically speaks to
• 25:36
  human loop and says, hey, what's the prompt that's currently
• 25:38
  deployed or what's the tool configuration that's currently
• 25:41
  deployed? Human loop sends that back and
• 25:43
  then the retriever gets called or it gets cached locally and
• 25:46
  then the retriever gets called. So you're managing the
• 25:48
  configuration which is the prompt and tool definitions from
• 25:51
  within human loop and that allows you to iterate quickly
• 25:54
  and make changes very fast on the application.
• 25:57
  And so that configuration, how are you tracking it over time or
• 26:00
  tracking changes? Is it just in GitHub?
• 26:03
  So we have sort of a version control system built into human
• 26:06
  loop that tracks it. And then we also have serialized
• 26:08
  versions of all of these. We call them files.
• 26:10
  So every tool in prompt in human loop has its file type and
• 26:14
  there's a dot prompt file format or a dot tool file format that
• 26:18
  people also check into Git. So people can have like a sync
• 26:20
  with Git if they want and still will be using git for
• 26:23
  versioning. But we're also keeping track of
• 26:25
  the history of changes. We've got audit logs, you can
• 26:27
  see who changed what on human loop natively.
• 26:30
  So if people are doing it through the UI, as a
• 26:31
  non-technical person, you have the same rigor as you would have
• 26:34
  if an engineer was committing code.
• 26:36
  Let's say I update a prompt or I start seeing something is going
• 26:40
  a little bit weird and so then I'm going to change out a model.
• 26:44
  What is the CICD process? So typically the workflow would
• 26:48
  be someone goes into human loop, they would make the change in
• 26:53
  our editor, save a version of that prompt and they'd trigger
• 26:56
  an evaluation. And sometimes these evals are
• 26:59
  actually triggered as part of like just typical CICD.
• 27:03
  So someone is running CICD, you know, and a GitHub action is
• 27:06
  going to call human lip to run the eval report and then they
• 27:09
  get back the metrics and the DIF from so they can set thresholds
• 27:13
  on like we have to be above certain metrics before
• 27:15
  something's going to be committed to prod and then it
• 27:17
  goes to prod. So it's actually like very much
• 27:20
  just the normal CICD people are used to.
• 27:22
  The only difference is that one of the GitHub actions or one of
• 27:26
  the hooks in your CICD system will be calling an eval report
• 27:29
  on human loop for that change. OK.
• 27:31
  What have you been seeing as far as use cases and actual like AI?
• 27:37
  That is, you're making people money.
• 27:40
  It's always easy and fun to talk about all these evals and
• 27:43
  everything that you can be doing, but at the end of the
• 27:46
  day, if you're not looking at what the real potential is.
• 27:52
  Or yeah, where's the, where's the business?
• 27:53
  Where's the business impact? The real use cases are and.
• 27:56
  So how have you been seeing folks succeed in this
• 27:59
  department? Yeah, absolutely.
• 28:01
  So we see more customer facing use cases.
• 28:03
  So there's kind of two buckets where I think people are either
• 28:06
  saving money or making money with with Gen.
• 28:08
  AI. So there's like internal process
• 28:11
  like automation. People are building customer
• 28:14
  service support agents, right? Whereas previously before they
• 28:16
  might have had humans take a certain amount of time, they can
• 28:19
  improve their deflection rates or increase the success with
• 28:22
  things like customer support. That's like a very obvious use
• 28:24
  case that at scale I know is saving companies 10s of millions
• 28:28
  of dollars. And I can't name names, but like
• 28:30
  at least at least one company I know is saving on the order of
• 28:33
  $30 million a year from what they had before.
• 28:36
  Not bad. But then what we spend a lot
• 28:39
  more time on is customer facing applications.
• 28:42
  So less internal process automation.
• 28:44
  And the reason why we naturally end up spending time on that or
• 28:46
  our customers are building that is in an internal app, people
• 28:50
  will tolerate a certain amount of bugginess or kind of it
• 28:52
  can't. It doesn't need to be 100%
• 28:54
  accurate, but as soon as you're putting something in front of
• 28:56
  customers, then reliability and like confidence that it's going
• 28:59
  to perform the way you expect goes up way higher and the need
• 29:02
  for evals and tracing and observability becomes an
• 29:06
  essential thing. So we mentioned earlier
• 29:08
  Filevine, which is a legal tech company.
• 29:10
  And the reason I'll talk about them is because we had a case
• 29:12
  study together that there's also shared.
• 29:14
  Shout out to Brianna. Shout out to Brianna from, from
• 29:17
  Filevine, but they're, you know, they've released something like
• 29:20
  6 new AI products in their contracts, sort of lifecycle
• 29:23
  management product and, and tools for lawyers across the
• 29:28
  last year and a half that are generating, I think 10s of
• 29:30
  millions of dollars of revenue for them, net new revenue and
• 29:33
  growing very quickly. So, so legal tech is like 1 like
• 29:37
  vertical that, you know, there are other companies, Harvey
• 29:41
  Antra that are like doing, you know, ironclad that are doing
• 29:44
  amazing things with LMS in the legal space.
• 29:47
  So those are applications that I think are generating significant
• 29:49
  revenue. Coding tools is like a vertical
• 29:52
  where we know there's a lot of revenue being generated.
• 29:54
  Cursor windsurf, now you've got cloud code from Anthropic
• 29:57
  natively, like Opening Eye has their own tools.
• 30:00
  GitHub copilot was the OG, right?
• 30:02
  So that's another set of vertical applications that we
• 30:05
  know people are getting a ton of value from.
• 30:07
  Can we pause right there for GitHub Copilot?
• 30:09
  Because talk about opening up a whole market and then just
• 30:13
  losing it. It's so incredible how much work
• 30:18
  they did to create this whole new field and then let Cursor
• 30:22
  come and eat their lunch. Yeah, You know, it's
• 30:25
  interesting, like we'll see how I think the game is still very
• 30:28
  early. Like I think it's too old to too
• 30:29
  early to call how it will play out.
• 30:32
  There's not that much differentiation even now between
• 30:34
  these different coding tools. Developers are very fickle.
• 30:37
  They swap models and they swap which tool they're using very
• 30:40
  quickly, true, so I wouldn't call it yet.
• 30:43
  Yeah. But yeah, it's GitHub Copilot.
• 30:48
  I feel like is it's true that you never know, it could come
• 30:52
  out of nowhere or rise like the phoenix from the ashes.
• 30:56
  Yeah, I mean, I'm, I'm interested to see what what the
• 30:58
  model providers themselves start to produce.
• 31:00
  Well, there was talks about windsurf being bought by.
• 31:03
  Rumors that opening eyes going to buy windsurf.
• 31:05
  Or it already bought them, who knows.
• 31:07
  And then you've got Cloud code, you've got opening eyes codec.
• 31:10
  So you know they're already releasing their own tools.
• 31:13
  They're very they're very clearly entering the game of
• 31:14
  competing directly on on coding tools.
• 31:17
  Yeah. And Copilot has some kind of
• 31:22
  partnership that I saw with one of the, I think it might be
• 31:26
  anthropic. Interesting because so early on
• 31:29
  they had a very close Co partnership with Microsoft
• 31:31
  because because Microsoft open eye relationship and the early
• 31:34
  get up copilot models were the early codecs models from open AI
• 31:38
  sort of checked over the wall like you know.
• 31:40
  All right, so continue. I I cut you off, you there's so
• 31:43
  in. Terms of vertical applications
• 31:44
  like legal, tech is definitely a domain where people are making a
• 31:46
  ton of money. Coding is 1 where people are
• 31:48
  making a ton of money. Another pocket that we see is at
• 31:51
  Tech. So we have a bunch of companies
• 31:53
  who are building tools for teachers or for students or AI
• 31:56
  tutoring. So one of our customers there is
• 31:58
  McMillan, who are building an AI tutor, but Duolingo for a kind
• 32:02
  of language learning love Duolingo and a bunch of and a.
• 32:05
  Bunch of others help. If you've ever tried to, you
• 32:08
  think you've got like a level 7 or level 10 and then you go to
• 32:11
  the country and you're like yo hablo espanol to hablas espanol.
• 32:18
  But I. Think I think that's going to
• 32:19
  get a lot better with improved AI tools.
• 32:21
  I hope right You can actually practice conversations.
• 32:23
  Exactly because I need to learn German.
• 32:25
  I've been living in Germany for too long to not be able to speak
• 32:28
  German. But you don't speak German.
• 32:29
  Yeah, exactly. I always.
• 32:30
  Assumed that because you were in Germany, you were German.
• 32:33
  I was Deutsch. No, I am very not Deutsch, and
• 32:36
  it's starting to show now. My daughter's six and she speaks
• 32:39
  German incredibly. And so I'm missing out on a lot
• 32:44
  of half the conversation because she'll speak to my wife in
• 32:46
  German. And then so Duolingo I'm going
• 32:50
  to start using it again and see if.
• 32:51
  See if you can get fluent in German.
• 32:53
  And because of human loop, I'm going to say, Raza, you saved
• 32:56
  me. No, you're the reason I speak
• 32:58
  German. I don't know if.
• 32:58
  We can take too much credit. I think the good, the good folks
• 33:00
  is these companies are doing the hard work, but we're we're
• 33:02
  hopefully facilitating. So yeah, those are, I think, 3
• 33:05
  concrete examples. I could probably, you know, name
• 33:07
  more from some. I think those are like still
• 33:12
  applications that are maybe like one generation back from the
• 33:15
  state-of-the-art in the sense that, you know, there's been a
• 33:17
  progression of complexity of things that people are building
• 33:19
  with LMS from right at the beginning, everyone was building
• 33:22
  these like writing assistants kind of very, you know, the only
• 33:25
  thing that really had traction in the beginning was like
• 33:26
  marketing automation. Copy AI was copy the.
• 33:30
  One Jarvis. That was the one.
• 33:33
  They raised a ton of money. There is a ton of money.
• 33:36
  I don't know. I don't know what's happened.
• 33:37
  To them and then we'll open AI. When and Chaji BT came out it
• 33:40
  was like oh we don't need copy AI anymore.
• 33:43
  But interesting, some of these companies have managed to keep
• 33:46
  reinventing themselves, like Writer I think is doing really
• 33:48
  well. And they started off in a very
• 33:51
  similar vein. But you know, what they did
• 33:53
  differently was that they were always focused on private models
• 33:56
  rather than building on top of the large public elements.
• 34:00
  So they seem to be going fine. But yes, you had kind of
• 34:02
  copywriting and then I think you started to get like rag systems
• 34:06
  and people realizing, hey, we can actually use this for
• 34:08
  customer support or for, you know, actual question answering
• 34:11
  in various ways. And we're only now seeing like
• 34:14
  more agentic applications really get deployed.
• 34:18
  And I think the first version of agents is still very much like
• 34:21
  the skew morphic. I'm going to go do what I used
• 34:24
  to do with a human with an agent.
• 34:27
  But we're starting to see some of our customers build things
• 34:29
  that go one step further and are imagining things that like
• 34:32
  wouldn't have been possible before LMS. So there's still a
• 34:36
  gentic applications, but they're not, I'm going to go replace a
• 34:38
  human there. I'm going to go do something
• 34:40
  that couldn't be done before. And a concrete example of this,
• 34:43
  one of our customers, a company called Windmill.
• 34:45
  And you know, the problem they're solving is that
• 34:48
  performance management in large companies sucks.
• 34:50
  You get a review from a customer like, you know, you start not a
• 34:53
  reason customer. You get like feedback from your
• 34:55
  manager and like a cycle like once every six months.
• 34:58
  The manager doesn't really understand like what's going on.
• 35:01
  It's it's not a great experience.
• 35:03
  But what you can do if you have an army of like LM agents is
• 35:06
  they can go out and they can speak to everyone in your
• 35:08
  company all the time. They can proactively ask them
• 35:10
  questions, they know whose colleagues you know are, and
• 35:14
  they can then aggregate that information back up and share it
• 35:17
  in both directions. The managers can have a much
• 35:19
  better idea of like on the ground what's happening.
• 35:21
  And also you can get like real time or continuous feedback from
• 35:25
  both your peers and your manager on how you're doing mediated by
• 35:28
  these LMS. They can effectively have, you know, conversations
• 35:32
  with everyone and it's still very early, but that's an
• 35:36
  example of like an application that I think wouldn't have been
• 35:38
  possible. You couldn't sort of imagine
• 35:40
  doing that before LLM. So I'm quite excited about
• 35:43
  people pushing agents in that direction.
• 35:45
  That's a fun thread to pull on because you can go in the
• 35:50
  direction of now that we have the ability to talk to people at
• 35:56
  scale, what can we do? Yeah, absolutely.
• 35:58
  I imagine they're talking of people in various formats.
• 36:03
  It's maybe that you get on a call and there's.
• 36:06
  I think in this case it's in Slack, but you could imagine it
• 36:08
  being a voice agent. You could, you know, and it's,
• 36:10
  it's, it'll just send you a Slack message with a few
• 36:12
  questions and it or it'll or it'll, you know, say, hey, we
• 36:14
  spoke to the colleagues who work most closely with you and they
• 36:18
  think you're doing a great job. But here's, you know, here's
• 36:20
  what you're doing. Well, but we've got this theme
• 36:22
  that came up across four different people where you might
• 36:24
  want to improve. And you know that wouldn't have
• 36:27
  been possible before. I take feedback horribly, so I
• 36:31
  would not like that tool at all. Yeah, I think, I think you can
• 36:36
  imagine like more pro human, pro social and less pro social
• 36:40
  versions. I think this is actually taking
• 36:42
  a very productive approach and just really trying to help
• 36:44
  people communicate better. Yeah, It reminds me of this idea
• 36:48
  that a friend of the pod, Fausto had, and he was saying, you
• 36:52
  know, what about when everyone joins the community, they get on
• 36:55
  a call with an avatar and they're asking quest.
• 37:00
  The avatar's asking questions on why are you joining this
• 37:03
  community? What is interesting for you?
• 37:05
  Are you looking for a job? Are you looking just to learn
• 37:07
  more? Are you looking for other people
• 37:09
  in the field? Because we kind of know the
• 37:12
  buckets that people will fall into when they join the
• 37:15
  community. And it's one of like the seven
• 37:19
  things and so. To what extent can you not do
• 37:22
  that just with the sort of form when they sign?
• 37:24
  Up because people don't fill out forms.
• 37:25
  That's the problem. OK. 9 out of 10 people just go,
• 37:29
  yeah fuck that. Like I'm not going to take 5
• 37:33
  minutes and fill this out, even though if they did and we knew
• 37:39
  more information and we knew like, wow, 5 out of 10 people
• 37:43
  who joined are looking to we meet others in the field.
• 37:47
  We, we have this program where we set people up one-on-one
• 37:51
  matching program with other people in the community.
• 37:55
  And so we can suggest that to them and make sure that they
• 37:58
  know that we have this program. But if you never get the data,
• 38:02
  you don't know. And so I can suggest it, but
• 38:05
  maybe it 5 out of 10 times it falls flat.
• 38:07
  Yeah. And one of the things that I
• 38:09
  think is interesting that you can do with this tech that you
• 38:12
  couldn't do with just a form is ask follow up questions and
• 38:16
  asked so follow up questions that are specific based, you
• 38:18
  know, adaptive forms. I don't think it's probably like
• 38:22
  possible before. So again, it goes back to being
• 38:24
  able to talk to people in a much deeper way at scale.
• 38:27
  Yeah. But building those systems, like
• 38:30
  there's so much added complexity, right?
• 38:31
  So the windmill PMS are kind of like iterating on, you know,
• 38:34
  they've got 10s or maybe more prompts at this stage, like
• 38:37
  hundreds of versions of those that each of those needs to be
• 38:39
  benchmarked. It's a complicated system, and
• 38:42
  being able to separate out, like iterating on the prompts from
• 38:46
  having that in your code base where they might be scattered
• 38:48
  all over is a is a very big win. I do appreciate that you
• 38:55
  extracted out of the code base so you're not going through
• 38:58
  lines and lines of codes or just like searching for that one
• 39:01
  prompt. You want to have a very clear
• 39:04
  picture of exactly what context you're sending to the model.
• 39:07
  So this is one of the beefs I also have with like a lot of the
• 39:09
  LLM frameworks, like agent frameworks and things like that,
• 39:12
  where, and I won't name any specific names, but people will
• 39:15
  know if they're using them. They all do this to a certain
• 39:18
  extent, yeah. Many of many of people have
• 39:20
  already talked about this on the Internet.
• 39:23
  But oftentimes they're hard coding prompts within the
• 39:26
  framework and so you as the end user don't have a clearview of
• 39:29
  exactly what am I sending the model.
• 39:31
  Prompt engineering and building with LMS is not complicated.
• 39:35
  Like all you need to really understand is like exactly what
• 39:37
  context of the model received so that you can reason about
• 39:40
  whether or not it's possible for the model to answer or achieve
• 39:43
  the task. And then what things do I need
• 39:45
  to change or give it more information on for it to be able
• 39:48
  to either adjust the tone or get closer to reliability.
• 39:51
  Like edge cases. I might not have thought of much
• 39:53
  like communicating with a colleague, but if there's a
• 39:56
  bunch of like hidden stuff that the models also being told.
• 39:59
  All this random body language. So I think like just making it
• 40:03
  really easy for people to see like exactly, exactly what is
• 40:06
  the context that's going to the model, which is the tracing
• 40:08
  observability and also having the prompt separated and
• 40:10
  centralized. So if you don't have them
• 40:12
  scattered around the code base. How often are you seeing folks
• 40:16
  use variables in their prompts? Oh, like all the time.
• 40:21
  That's a common pattern now, right?
• 40:23
  Yeah, so like we talk about prompt templates.
• 40:25
  So when we talk about prompts in human loop, we really mean the
• 40:28
  choice of the model, the hyper parameters and then the prompt
• 40:30
  template where that template is going to have, you know, at in
• 40:34
  production variables that are coming either from user or
• 40:37
  feedback or they're going to be being filled in from a retrieval
• 40:39
  system. So you have a a template with
• 40:43
  gaps you know people use maybe like mustache style sort of
• 40:46
  handles or ginger templating or something like this to specify
• 40:49
  the prompt template. Because it brings up the other
• 40:53
  piece that we were talking about earlier on the recommender
• 40:55
  systems. Yeah.
• 40:57
  So the, the context here was we were saying that the closest
• 40:59
  analogy to building with sort of very subjective use cases before
• 41:05
  LMS was probably recommender systems.
• 41:07
  A lot of machine learning has correct answers that recommended
• 41:10
  systems don't like what is the best movie to recommend or
• 41:13
  clothing item to recommend. There's not one correct answer.
• 41:16
  And in that way it's very analogous to building LM systems
• 41:20
  where, you know it's subjective as to what what good is.
• 41:22
  Then you were saying, well now when I want to go for a movie, I
• 41:26
  look at. 100% go to ChatGPT now. Yeah.
• 41:30
  I know, and I think you're an outlier.
• 41:31
  I probably I probably AM. We'll we'll ask the chat and
• 41:35
  we'll see what people say like listeners can let us know.
• 41:40
  But if you're not doing this, you should be.
• 41:41
  So my let me let me make my pitch quickly, which is just
• 41:44
  that if you're if you're a heavy user of one of these LM systems,
• 41:47
  ChatGPT clawed, whatever it might be like it's learning a
• 41:50
  lot about you. It knows a lot about your
• 41:51
  preferences through the memory. Now it also like, can you, you
• 41:55
  can describe to it what you want in a way that like is very hard
• 41:59
  to do with like a traditional Rex system that just says like,
• 42:02
  you know, how similar are you to the other people?
• 42:04
  So, you know, you can do weird things.
• 42:06
  You can be like, hey, I'm in the mood for a political thriller
• 42:10
  that's like, slightly dark. In the style of Guy Ritchie.
• 42:12
  Yeah, I recently and the style of Guy Ritchie, I recently
• 42:14
  watched a movie that I really liked and I'm like, I'm in the
• 42:17
  mood for something similar. Like what else is out there
• 42:19
  that's like this or I really like Christopher Nolan's films.
• 42:23
  Like which other directors also play on themes of time travel or
• 42:26
  whatever it might. You can't get that from a Rexis.
• 42:29
  No, yeah, I was thinking like, oh, here's the last five films
• 42:33
  that I really enjoyed. You can do this.
• 42:35
  Too. Give me yeah, give me five more.
• 42:37
  Or at least give me two, because I'm probably not going to watch
• 42:41
  all 5 right now. Yeah.
• 42:42
  And I don't think this is a replacement for traditional
• 42:44
  Rexis, but as a as a it is, it is interesting though to see,
• 42:49
  you know, you can do embeddings that are like conditioned on
• 42:53
  descriptions or text or things like that.
• 42:54
  So I think these two worlds will collide a bit.
• 42:58
  I think you're an outlier because of the cognitive load
• 43:01
  that takes. And even though for you, you're
• 43:04
  sitting there and you're like, no, this is really cool because
• 43:06
  I can be really creative and you can get a much better
• 43:09
  recommendation than just scrolling through the horizontal
• 43:14
  bars on Netflix. But then I was looking at it and
• 43:18
  I was saying, yeah, but man, clicking is so much easier than
• 43:23
  trying to articulate what movie. And, and really, I think that's
• 43:27
  what it comes down to is like, if I go on Netflix and I just
• 43:31
  scroll around for a while and it knows what I have watched
• 43:35
  already and it's already bringing me different films,
• 43:39
  then it's much easier to just go, yeah, all right, this looks
• 43:43
  good. I, however, I don't think I am
• 43:46
  that person either because usually I'll get my movie
• 43:49
  recommendations from folks on Tiktok.
• 43:52
  OK, interesting I'll see different people on TikTok
• 43:55
  breakdown like here's the top ten these movies this insert
• 43:59
  like favorite type of movie. But for the most part, the
• 44:05
  recommender systems where you're just clicking or scrolling.
• 44:08
  If we're going to a recommender system, like Tik Tok's got the
• 44:10
  best one, right? And so if you extrapolate it
• 44:15
  out, maybe for a movie, yeah. But if it was for a social
• 44:18
  network like TikTok, I'm in the mood for something funny.
• 44:21
  It has to do with, like, cats and this and that.
• 44:24
  And it's going to be much more effort to get something where
• 44:28
  you can just get those signals and scroll.
• 44:31
  Yeah, I think, I think that's probably true.
• 44:33
  This isn't something I've given a a great deal of thought to.
• 44:35
  Although something that occurs to me just off the top of my
• 44:38
  head is that like one of the hard problems with like
• 44:40
  recommender systems is that you're like very often in a
• 44:43
  small data regime, right? Like new person joins and you
• 44:45
  know very little about them. And like being able to go from
• 44:48
  like a description of that person or like looking some
• 44:51
  stuff up about them and then using that to like find which
• 44:54
  people are like them might be might be a very interesting way
• 44:57
  to overcome those cold starts. That's a cool way of plugging it
• 45:00
  in, yeah. But not not something I'm
• 45:02
  personally expert on, so I will avoid having two strong
• 45:05
  opinions. Well, Speaking of things you are
• 45:07
  expert on, as you look forward, what's around the corner for
• 45:10
  you? So there's there's really two
• 45:12
  things. Yeah.
• 45:13
  And one of them we're launching actually, you know, in the next
• 45:16
  week or two. So maybe by the time this
• 45:17
  podcast comes out, it'll be. It'll be by editing.
• 45:20
  In at least in beta and and sort of being used by people, but so
• 45:24
  one of them is we're trying to give much better support for
• 45:27
  people who are prototype and and building agents and agent
• 45:31
  evaluations as well are just much harder than evaluating
• 45:33
  simpler system. Because of the steps involved.
• 45:36
  Yeah, because there's like this branching tree of so many things
• 45:39
  that the agent can do. So having good coverage over
• 45:41
  what might the agent do is hard. Also oftentimes to evaluate an
• 45:45
  agent or even run one it you know they're interacting with
• 45:48
  users or or other systems, you have to be able to mock those
• 45:51
  systems as part of your eval. So if there is a user who is
• 45:54
  asking questions as part of the agent, you know, if one of the
• 45:57
  tools the agent has is like get information from user, then to
• 46:00
  run a good eval, you need to be able to like mock the user.
• 46:03
  Or if it's calling third party API is like, you need to have
• 46:05
  that integration into your eval system.
• 46:07
  So evaluating agents is just more complicated.
• 46:10
  And so we're, we're launching a new agent builder as well as
• 46:14
  much better tooling around how do I evaluate complicated
• 46:17
  agents? So today there's I think decent
• 46:19
  support for this in human loop, but this will just simplify both
• 46:22
  the DX and UX and I think be pretty strong there.
• 46:26
  And then the one I'm very excited about is auto
• 46:29
  optimization. We're trying to come up with a a
• 46:32
  better name for this and maybe you can help me out with names.
• 46:35
  We haven't. We haven't.
• 46:35
  Released this yet auto. Optimization, but the, but the
• 46:38
  premise is, you know, people are spending so much time in human
• 46:42
  loop today tweaking prompts, adjusting tool definitions, like
• 46:45
  iterating on these different components.
• 46:47
  And actually we've done a bunch of research now in prototype on
• 46:51
  having AI systems do a lot of this iteration for you.
• 46:55
  And because we have access to your evaluation scores and your
• 46:59
  data sets, we can run that. So once you've set up basically
• 47:02
  your data set and evals, we can have the AI proposed changes to
• 47:06
  prompts, update parts of the system, rerun the eval, look at
• 47:10
  the eval report. And then like a human data
• 47:12
  scientist would like sort of think about like, oh, what is
• 47:15
  the cause of the failure? Make an adjustment and rerun
• 47:19
  until the the score improves to certain certain point.
• 47:22
  That works better today than like we've benchmarked it
• 47:26
  against DSP, why we've been benchmarked it against it.
• 47:29
  Has a bit of DS. It's it's got a DSP vibe to it,
• 47:33
  but one we're looking at like entire pipeline.
• 47:36
  So not just kind of prompts or those parts.
• 47:38
  Because they do the auto prompt optimization, but you're saying
• 47:41
  the entire pipeline. What if it is the?
• 47:43
  It's the model that you need to change out.
• 47:45
  Yeah. And so let's test 5 different
• 47:48
  models here and see if that gives us any.
• 47:50
  Yeah. So there's and there's quite a
• 47:51
  lot of complexity to getting it to work well.
• 47:55
  And then we've benchmarked it against kind of the best things
• 47:57
  we can find in the academic literature.
• 47:59
  And you know, we're currently outperforming.
• 48:02
  So we're going to sort of be running that in beta starting
• 48:04
  very soon actually with sort of small group of early customers.
• 48:07
  And then I'm hoping in a few months or a couple of months
• 48:10
  time kind of launch that to GA. And so if something that you
• 48:13
  know, our top users are spending 10 hours a week or something you
• 48:16
  know of of this iteration now I think we can probably just
• 48:19
  automate all of that. Wow.
• 48:21
  All of it. A large fraction.
• 48:24
  So it's basically like you're going from the, you're going to
• 48:28
  more of a declarative, yeah, way of working.
• 48:31
  Just say give me your emails and I'll give you what you need.
• 48:33
  Give me give me your emails and that kind of first version of
• 48:36
  the system that works 80% accurate or whatever.
• 48:39
  And a lot of the iteration that you were doing to get from there
• 48:42
  to the system that's 90% plus, you know, all those hours that
• 48:46
  Brianna's team at file line spent in human loop with
• 48:49
  expensive lawyers like we can, we can maybe do a really good
• 48:53
  first cut of that or or at least automate a significant fraction
• 48:56
  of that work. So those two are the are the
• 48:59
  sort of new things that we're most excited about.
• 49:02
  Lawyers are expensive. Engineers are expensive.
• 49:06
  PMS aren't cheap. So.
• 49:08
  Especially good ones, especially in AI.
• 49:10
  If you're thinking about hey this 10 hours now open are
• 49:15
  opened up for the lawyer to be creating more evals.
• 49:19
  Or or just doing anything else? Lawyer work, whatever lawyers
• 49:22
  do. Yeah, go Bill some hours.
• 49:25
  Yeah. So I think that's going to be
• 49:26
  quite transformative. And I think that it's we have to
• 49:32
  build the foundations first. You can't actually, like someone
• 49:35
  couldn't show up today and they're like, hey, we want to
• 49:36
  build like the optimization system for AI products because
• 49:40
  the first thing they'd have to do is like, they'd be like, OK,
• 49:42
  well, what we need is the ability to like build evals and
• 49:45
  run them over data sets and put that in the like closed loop of
• 49:47
  the optimization. I mean, people are doing.
• 49:49
  It's like Anthropic has some experimental API's around prompt
• 49:53
  improvement. So if you go into the Anthropic
• 49:55
  console or if you go into their API, they're like, take a prompt
• 49:58
  and they'll rewrite that prompt for you sort of using best
• 50:01
  practices for Claude. But it doesn't tell you anything
• 50:04
  about like how good did it get better or for your use case.
• 50:08
  And I think having this eval suite coupled to the auto
• 50:11
  optimization gives us a unique advantage over anyone else's
• 50:15
  building in the space. It sounds a bit magical and so I
• 50:19
  want to see it when you release it.
• 50:21
  Like come and show us because. I would, I would love to come
• 50:24
  back. And I mean, what we're going to
• 50:25
  try and do, the reason why we're launching it in beta for us with
• 50:28
  small number of customers is exactly this, that we know there
• 50:31
  will be skepticism because the promise is so big.
• 50:34
  Yeah. And So what we would like to do
• 50:37
  is we want to launch it with like concrete case studies to
• 50:39
  kind of help people kind of get over that initial skepticism.
• 50:43
  But I know from having used it in kind of our research work and
• 50:46
  having a very good intuitive sense of what people are trying
• 50:49
  to do, I have very high confidence that this will work.
• 50:52
  And so are there things that whether it is creating a more
• 50:59
  fine-tuned prompt or just having more prompts that you can pull
• 51:03
  from or having more eval data points, does that add to the
• 51:10
  success rate? Or is it like whatever, give us
• 51:14
  that baseline and we'll add on to it with synthetic data, with
• 51:18
  some evals that we get and we don't even need you around?
• 51:24
  No, we still so we like the taste of the end user is not
• 51:28
  something that we can ever like replace and they express what
• 51:33
  they want, right. You said moving to a decorative
• 51:35
  approach in two ways, right, in the definition of the evals and
• 51:39
  the feedback they give us, right.
• 51:41
  So one of the things that we can do is much the same way that we
• 51:43
  might that the human teams today ask the domain expert for
• 51:47
  feedback in terms of score these outputs.
• 51:50
  That's something that we can ask for.
• 51:52
  And then the auto evals that are already there also guided.
• 51:54
  So we cannot like, you know, there's no, there's no free
• 51:58
  lunch. We're not magically able to
• 52:00
  build a better system than you like without your input.
• 52:04
  But what we're changing is that like all we need from you, as
• 52:07
  you say, it's decorative is like for you to express what you want
• 52:10
  clearly through like decisions and feedback.
• 52:13
  So like, yes, this one, not that one.
• 52:14
  Preference data, definitions of evaluation, some minimum
• 52:18
  criteria of what good looks like and a first version of the
• 52:21
  system from which we can infer your goal.
• 52:23
  And then from that we can run. So when you tell people about
• 52:27
  human loop, how do you describe it?
• 52:30
  So we describe Human Loop as the LLM eval platform for
• 52:33
  enterprises. And we're focused on evals
• 52:35
  because that tends to be the most foundational piece.
• 52:38
  It's not the only part of Human Loop, right?
• 52:39
  We're really trying to give you all the tools you need to do
• 52:42
  prompt management, to do observability, to do evaluation,
• 52:45
  to get a reliable AI system. And I think we're quickly
• 52:47
  becoming the kind of sort of industry standard for evals.
• 52:51
  So that's the bit we lead with because I'd rather be, you know,
• 52:55
  100% clear and 80% accurate.