Latest / Agentic Conversations (formally mlops.community) / Are Evals Dead?

Transcript

Are Evals Dead? · Agentic Conversations (formally mlops.community)

• 0:00
  Why do you even need evils in the 1st place?
• 0:02
  Right? We started to see where things
• 0:04
  were failing and started to build evils for that
• 0:07
  specifically. You know, it's easy to say let's
• 0:09
  just test the agent end to end. What does end to end mean?
• 0:13
  In the times of machine learning, we were training a new
• 0:16
  model and you roll it out. You know, now it's coming to
• 0:18
  agents because agents are being productionized.
• 0:27
  How are you evaluating? How has that changed over time?
• 0:32
  We learned a lot obviously in the past years that process.
• 0:34
  We've been building many agents for many applications.
• 0:37
  And for the lesson we quickly learned that you need to have a
• 0:42
  way to understand how your agent is doing.
• 0:44
  Like now it's very easy to by code something and get it in
• 0:47
  production, but that doesn't scale.
• 0:50
  If you want to release this to millions of users, you will find
• 0:53
  that many times they break. And this is also reason why
• 0:57
  people don't trust Asians 100%. But we're here to change that.
• 1:03
  The way we approach it is also from kind of scientific approach
• 1:08
  because I'm a data scientist myself by background and like
• 1:13
  it's easy to use models with an API, just run a prompt and get a
• 1:18
  response. But actually what you write in
• 1:20
  this prompt can change a lot of the results.
• 1:23
  And the way I see it is, is much more like a data science kind of
• 1:27
  task where when you write a prompt, you're kind of training
• 1:30
  a model 'cause you're determining the outputs of of
• 1:33
  the agent, right? So it's very important to to
• 1:38
  have good evaluation sets. And at the beginning you don't
• 1:41
  have really production data, so you need to kind of bootstrap
• 1:46
  this whole thing. And it was very important to
• 1:50
  have a very created set of examples.
• 1:55
  For instance, this could come from product manager insights or
• 1:59
  discussion in the team, or maybe this is a specific use case that
• 2:03
  you want the agents to we need to work on.
• 2:07
  And, and there we basically built all these examples.
• 2:11
  So at the beginning to bootstrap this, this agent, we created a
• 2:15
  curated set of of test cases. We approach it like a software
• 2:19
  engineering project. In this case, we wanted to have
• 2:22
  some flows that we were sure that we're gonna work.
• 2:24
  Those flows you can use in CICD pipelines and and everything
• 2:29
  like more more like you would do for a software development
• 2:33
  project. Yeah.
• 2:35
  And that you, you make sure you like the main use cases are
• 2:38
  covered, but you also might wanna see all the parts that
• 2:42
  need testing and make sure that nothing breaks along the
• 2:45
  pipeline. That was the first level that we
• 2:49
  did. So understand what goes wrong
• 2:51
  where, fix it like iterate fast. And then we started to test the
• 2:57
  system ourselves, like on a wider, wider team within the
• 3:01
  company. And getting feedback from users
• 3:04
  was very specific feedback because we had internal users
• 3:07
  that were helping us with this. And with that, we started to see
• 3:11
  where things were failing and started to build evolves for
• 3:15
  that specifically. Something we did was also
• 3:19
  building evolves for multi certain conversations, like for
• 3:23
  guardrails. This is all the obvious case
• 3:26
  because when you have single turn conversations, it's very
• 3:30
  hard to to see if the guardrails can be broken or not.
• 3:34
  Yeah, sometimes you need the user who's very persistent, like
• 3:37
  I'm not talking really bad stuff because that is usually covered
• 3:42
  if you're using foundation models.
• 3:44
  But there are things like platform circumvention that is a
• 3:47
  bit more sneaky, like trying to users trying to get information
• 3:52
  that is not supposed to be public.
• 3:53
  You know, there is a lot of of things.
• 3:55
  Yeah, that. For users trying to get
• 3:58
  discounts. I imagine.
• 3:59
  Well, yeah, you know, once you release it, people are going to
• 4:03
  try their worst or their best, so yeah.
• 4:08
  Wait and can you break down this multi turn conversation evals
• 4:12
  and how you were? Basically it was 4 guardrails
• 4:15
  but you also wanted to evaluate that it wasn't giving discounts
• 4:20
  when it needed to be or it wasn't telling secret
• 4:22
  information that it shouldn't be.
• 4:24
  Did you basically set up another agent to try and red team it?
• 4:28
  Were you specifically trying to red team it?
• 4:30
  How did that work? So we created personas.
• 4:35
  There is yeah, we created an agent that was supposed to
• 4:38
  impersonate a user who had certain intention and we were
• 4:40
  defining this at the beginning and the agent had to be very
• 4:44
  persistent to try and break our own agent.
• 4:48
  So for instance, yeah, no matter what the other agent's response,
• 4:52
  try to get this information, try to be per SE events one nice and
• 4:57
  sometimes it was working and yeah, that's where you know you
• 5:01
  need to to do something about it.
• 5:03
  How were you updating that You realized something went wrong?
• 5:07
  You realized that the agent gave away information it didn't mean
• 5:10
  to? What did you then do to you just
• 5:12
  change the prompts? Or did you also add specific
• 5:16
  like hard coded rules? Yeah, it depends on the case.
• 5:19
  Like sometimes we didn't encounter it in our case, but
• 5:24
  sometimes you might have security issues that did not
• 5:27
  surface at the time and then you need to to do changes in the
• 5:32
  actual code. Sometimes it's just prompting
• 5:34
  being a bit more aggressive. Or you can add a reviewer that
• 5:39
  evaluates the output and checks, you know, some sort of content
• 5:41
  moderation step before the response.
• 5:45
  There are different techniques and you know, always start
• 5:49
  simple and then build up if it doesn't work.
• 5:51
  You did talk to me a bit about the simulations that you were
• 5:55
  doing because I think that kind of is in line with what you're
• 5:59
  talking about now. You were simulating in different
• 6:02
  ways like. Can you go into that?
• 6:04
  So the simulation that we did, we're kind of multi turn with a
• 6:08
  pretty fine number of turns. So we didn't drag them for very
• 6:12
  long. We just wanted to see whether
• 6:15
  the agent could break our agent in like 4 or five turns of
• 6:18
  conversation. And we had few phases to define
• 6:24
  this, like first we came up with examples ourselves because we
• 6:27
  wanted to, yeah, just make sure basics are covered.
• 6:32
  Then we started to get feedback. And then after this phase you
• 6:34
  can move to production data and see maybe we can move to this
• 6:39
  other parts of Evol's, which is equally important, which is
• 6:42
  error analysis. So there you really want to
• 6:45
  pinpoint what's wrong and add this to your evaluation set.
• 6:50
  So for instance, if we see a case where a user managed to get
• 6:53
  to sneak some information from the agent, then you add it also
• 6:56
  to this set. And you make sure you run it
• 6:58
  every time you're updating the code, or periodically depending
• 7:03
  on how frequently you need it. One thing that I'm fascinated by
• 7:08
  is the evals. You're kind of doing offline,
• 7:11
  right? But then you have prompt changes
• 7:15
  that you roll out and those are almost like online.
• 7:19
  Do you think about how to update the production system with the
• 7:25
  new prompts, but almost like do it in a champion challenger way?
• 7:30
  Or is it done slowly? Or do you just change it and see
• 7:34
  where things break afterwards? If I understand your question,
• 7:38
  like is it it's about basically measuring the impact of the
• 7:43
  change in the? Production system, basically,
• 7:47
  and especially because you change the prompt.
• 7:53
  How are you rolling it out? Yeah, this is same as rolling
• 7:56
  out a new model. Like in the times of machine
• 7:59
  learning, we were you're training a new model and you run
• 8:02
  it out. So first of all, you want to see
• 8:04
  how it compares to the previous. So you have an evil set to run
• 8:08
  it and you see the if there are discrepancies, if it's better,
• 8:12
  if there are some some tests that now fail and it didn't
• 8:16
  before, like if you introduce new errors.
• 8:18
  This is really important to do. First of all, on a trusted evil
• 8:21
  set, you run it there and then you can start to roll it out.
• 8:26
  You can do an AP test for instance.
• 8:28
  There are, there is a lot of literature about this because,
• 8:34
  yeah, this is what data science community has been doing a lot
• 8:38
  before. And, you know, now it's coming
• 8:39
  to agents because agents are, you know, are you being
• 8:43
  productionized. So now all these little things
• 8:46
  are starting to connect finally. It was.
• 8:50
  I didn't see this as much before, but now, yeah, it's
• 8:54
  really nice that we can use all these techniques and port them
• 8:56
  in the world of agents. Maybe that's a nice thread to
• 8:59
  pull on. Is what are some things that you
• 9:03
  can take from the traditional predictive machine learning
• 9:07
  world, like those eval sets that you're testing offline and then
• 9:11
  the slow rollouts or the Champion Challenger AB testing
• 9:14
  when you do roll out the model. But now we're not rolling out a
• 9:17
  new model, we're rolling out new prompts.
• 9:19
  What are some other things that you're taking or what are some
• 9:22
  things that are completely different?
• 9:23
  Imagine you have an input of a black box, which is your model,
• 9:26
  your prompt, and then you have the output.
• 9:29
  So that part needs to be tested well, so that I approach like a
• 9:32
  data science problem more than traditional machine learning
• 9:35
  sense, even though it's there is a more, there is a bit more
• 9:40
  explainability there. We can make easier changes in
• 9:44
  the prompts, but still that that part is very analogous.
• 9:49
  We've seen. The other is the fact that
• 9:51
  you're integrating this into a big software engineering project
• 9:55
  that is more complex than just getting an output from a model.
• 10:02
  So for instance, tool, tool calling is really important to
• 10:05
  get it right. Sometimes tools are whole
• 10:07
  workflows by themselves, like tool can be almost an entire
• 10:12
  agent or sometimes agents. So there is much more complexity
• 10:18
  in an agentic system, so many more things that can go wrong.
• 10:21
  And if you start to, for instance, change API holes where
• 10:26
  you have always the probability that things might go wrong, that
• 10:30
  you're going to get the wrong output.
• 10:31
  So it's really important to find out what the steps are and test
• 10:35
  everything. It's yeah, you know, it's easy
• 10:38
  to say let's just test the agent end to end.
• 10:41
  What does end to end mean? That there are all these
• 10:43
  components and you really need to understand what goes wrong.
• 10:47
  Also, at the end, why do you even need evils in the 1st
• 10:50
  place, right? You want to see what's wrong,
• 10:53
  What to improve? Is the product good enough to be
• 10:55
  released? I think for ether software
• 10:58
  engineering team you want to know what to do next.
• 11:01
  Yeah, that's the most important. You want to iterate fast, get
• 11:04
  output and release new version quickly.
• 11:08
  So you need evils to understand where things go wrong, What are
• 11:11
  the errors? Why does certain things happen?
• 11:14
  Having agents use other agents as tools, those agents now are
• 11:19
  they can fail, and maybe you don't really recognize where
• 11:24
  that happens, so debugging becomes very complex.
• 11:27
  It's the core of evaluating agents because at the end, users
• 11:31
  care about the end to end experience, right?
• 11:33
  We want to give a good experience to the user.
• 11:35
  So end to end is really important.
• 11:38
  So there is a level in which you want to test like how things go,
• 11:42
  like how what's the experience for the user itself.
• 11:45
  Yeah. So that's very important.
• 11:49
  It's not the only part of it. It's very important to us to dig
• 11:53
  deep and see for every step what can go wrong.
• 11:57
  This is more similar to traditional software engineering
• 11:59
  where you have unit tests for functions.
• 12:03
  We want to have brand new visibility into what can go
• 12:06
  wrong. We say we have the end to end
• 12:08
  experience where you can for instance, say this is what the
• 12:12
  user types and this is what the agent responds like the agent is
• 12:16
  doing web search, these are the search results, things like
• 12:18
  that. Or the user is searching for
• 12:21
  gluten free foods. Did the agent return gluten free
• 12:24
  food or not? This is this is something
• 12:27
  important to check, right? Because there are allergies in
• 12:30
  case of food. So to give you some ideas.
• 12:34
  But then yeah, you check that the user experience is fine, but
• 12:38
  then from the developer side, if I want to know what to fix, then
• 12:43
  I need to take all these tool calls, see what goes wrong and
• 12:48
  where. And I think it's very important
• 12:51
  to to use the right tools for that.
• 12:53
  Also we find many times this project have complex workflows
• 13:01
  and traditional platforms for evals are sometimes not enough
• 13:06
  like to visualize data and outputs.
• 13:09
  We sometimes have to build our own tooling.
• 13:11
  Interesting. Tell me more about that.
• 13:14
  For reasons to evaluate conversations in order for
• 13:17
  people who are non-technical to understand and see if something
• 13:20
  was good enough or not, we needed a good way to visualize
• 13:24
  this. And you know you have agents and
• 13:27
  tools, most frameworks allow you to visualize these things, but
• 13:31
  sometimes in the tool you also have other things going on that
• 13:34
  are very domain specific. So we had to build apps for, for
• 13:40
  them to, to visualize conversations.
• 13:42
  And I think this stuff is sometimes overlooked, but it's
• 13:46
  very important because you really need domain experts, you
• 13:49
  need PMS to, to look at this data because they are they're
• 13:52
  guiding the development of the product itself.
• 13:55
  So they really need to understand what is going on and
• 13:58
  it needs to be easy for them. I heard about a friend who
• 14:01
  actually Willem, who came on the podcast probably three months
• 14:05
  ago, and he was talking about how they built heat maps for
• 14:09
  where the agents were failing. And once they built these heat
• 14:13
  maps, they could see very clearly what the agent was good
• 14:16
  at and what it was not so good at.
• 14:19
  Yeah, I think this is the this is the core of evils like
• 14:23
  understanding where things go wrong.
• 14:26
  So we had it at the beginning. We we had simple prompts to
• 14:29
  evaluate conversations, which were like, is the agent
• 14:34
  returning a good response? Is the tone accurate?
• 14:37
  Like all these things, This is when you.
• 14:39
  Were using LLM as a judge. Yeah, yeah, yeah.
• 14:41
  This is for LLM as judge. And it's very easy for the for
• 14:45
  the judge to think that everything goes well, but you
• 14:48
  really need to to instruct them to look at the errors, like find
• 14:52
  out what is wrong, because they don't have this capabilities by
• 14:56
  themselves. Like they're very eager to
• 14:59
  please you when you show them data, usually talking about
• 15:04
  various foundation models here that you can use to evaluate.
• 15:08
  But what you want is not that. So yeah, this is a very cyclical
• 15:12
  process. You need to keep people in the
• 15:14
  loop. You need to pull people from
• 15:17
  technical and product sides in the loop, because you have
• 15:20
  evaluation sets what to evaluate, but also how.
• 15:24
  What is the quality of the evaluation output?
• 15:27
  Because I think it's worse to get wrong outputs to not get
• 15:32
  them at all. Like you think everything goes
• 15:34
  well and then it doesn't. Of course you don't want that.
• 15:37
  Yeah, those silent failures is something that I wanted to ask
• 15:40
  you about too, because a lot of times maybe everything seems
• 15:43
  like on the surface it went well, but when you dig in and
• 15:47
  you look at that user experience, you go, wow, this
• 15:49
  was a horrible user experience. Yeah, I think nothing beats
• 15:53
  using the app and and testing it and.
• 15:56
  Being your own beta tester basically.
• 15:58
  Yes, we I think dogfooding is one of the most important things
• 16:02
  to do if you're developing a product, you'd need to use it,
• 16:05
  you need other people to use it, You need your grandma to use it
• 16:09
  too. It's really, yeah, it's really
• 16:12
  important to see the flow cuz open these apps involve UI
• 16:15
  elements. It's not just the agent itself.
• 16:19
  It's easy to sit in our corner and just evaluate the agent
• 16:22
  sitting in the tool. Well, yeah, but what if the user
• 16:25
  gets a completely wrong experience?
• 16:28
  So there, yeah, there is, especially at the beginning of
• 16:31
  this project, there is a lot of information sharing and yeah,
• 16:35
  like involving multiple people in this.
• 16:38
  Well, there is a fascinating thing here too.
• 16:40
  When it comes to the evaluations that you were talking about and
• 16:45
  the visualization that you were mentioning, you want to be able
• 16:49
  to visualize it in ways so that non-technical people can also
• 16:52
  get an idea of what's happening. I just mentioned heat maps with
• 16:56
  my buddy William. Did you have other ways of
• 17:01
  changing that data to visualize it for the others that you found
• 17:06
  particularly helpful? Yeah, I think distributions are
• 17:10
  really important. Sometimes you don't know what
• 17:15
  errors you're going to encounter.
• 17:16
  So it's also important to keep things free form like have the
• 17:20
  agents like summarize what went wrong simply and then aggregate
• 17:24
  everything later is very important because you can think,
• 17:28
  OK, what are the category of errors that I can get?
• 17:31
  What is the taxonomy? You can have a tool error.
• 17:33
  You can have, I don't know, time out error like there are many.
• 17:36
  You can start writing them all. But at the end you have, if you
• 17:40
  get our data, then you can just use that, right?
• 17:44
  I mean, these things are not mutually exclusive, of course.
• 17:47
  Yes, yeah. So that's a interesting way of
• 17:51
  putting it is why not just let the agent tell you what went
• 17:56
  wrong and summarize what went wrong and then you can have that
• 17:59
  as another data point? Exactly.
• 18:01
  Yeah, yeah. I imagine there's no shortage of
• 18:04
  data for this. So in and in a way you can
• 18:09
  always get more data. You can always ask the agent for
• 18:11
  more data and say what went wrong or show.
• 18:15
  Then you can also look at the traces and logs from different
• 18:19
  tools. You can look at the eval sets
• 18:21
  and if it's failing any of those, yeah.
• 18:25
  Is there like a 8020 here that we, if you're going into it,
• 18:30
  you're saying these I 100% need to have as data points and they
• 18:37
  need to be this granular or this sophisticated before I feel
• 18:42
  confident in pushing my agent out to production?
• 18:46
  Yeah. That's, that's a very bold
• 18:50
  question because it's, it's really hard, especially at the
• 18:55
  beginning to know how this is going to how this is going to
• 18:59
  look, right. Like especially if you don't
• 19:02
  have production data, you're kind of imagining how things are
• 19:06
  going to work until you write it out to user.
• 19:09
  Yeah, you, you don't, you don't have this information even like
• 19:12
  you can think these are all the use cases I want to cover.
• 19:15
  OK, 99% are cover, I will release or maybe 50% if you're
• 19:20
  more comfortable. But at the end you need real
• 19:24
  user data. So I would frame it the opposite
• 19:28
  way. Like I think that the, the goal
• 19:31
  is really get feedback as soon as possible and, and get and get
• 19:35
  this out. And if you release to some
• 19:37
  users, for instance, you will start to to have conversation
• 19:40
  logs and then you can really see where all the failures are.
• 19:45
  So there are some that you might want to accept, some that you
• 19:47
  might not want. In the case of search, there is
• 19:51
  always a precision, a recall problem.
• 19:54
  For instance, in like with information about food, you want
• 20:00
  to be precise because if someone has set certain that
• 20:02
  restriction, they don't want to give them wrong information.
• 20:06
  So it really depends on the project.
• 20:09
  You really need to sit down and with people from engineering
• 20:13
  products and and define what these things are.
• 20:16
  Yeah, maybe we haven't talked about it yet, but.
• 20:19
  But defining what metrics are comfortable with is really
• 20:22
  important at the beginning. So OK, you have an idea of what
• 20:27
  is good enough right? Will I tolerate 50% of the
• 20:32
  search is going wrong? Like what is?
• 20:35
  What is the tolerance level for errors?
• 20:38
  What do you accept? Yeah.
• 20:40
  Again, it goes back to this idea of like there are hard line
• 20:43
  things that we first sure cannot do, like giving free food away
• 20:48
  or telling a vegan that this food is vegan, but it's not.
• 20:54
  It's actually meat. Those are things that you're
• 20:57
  going to piss some people off very quickly with.
• 21:01
  And so you recognize along these lines we need very, very low
• 21:07
  margin of error. But over here we can be a little
• 21:10
  bit more. Yeah.
• 21:13
  Generous with our margins of error and.
• 21:15
  I mean, this is AI, we need to embrace it, right?
• 21:19
  It's not like user also expects the agent to be conversational,
• 21:24
  so they might get something wrong and then recover and and
• 21:28
  then give the right result at the end.
• 21:29
  You know, there can be some hiccups along the way,
• 21:32
  especially at the beginning. But I think with new AI
• 21:36
  products, people are a bit more comfortable with that by now.
• 21:40
  Of course then you want to improve it, but you cannot do
• 21:43
  that without enough data. So it's a bit.
• 21:46
  You need to be comfortable with this uncertainty and and
• 21:50
  leverage it. Like use the feedback and and
• 21:53
  build on top. Is there anything that you are
• 21:59
  particularly surprised about that you've learned along the
• 22:02
  journey of building these agents, putting them into
• 22:05
  production, that you want to share right now?
• 22:08
  Or that you can share with us that if you were going back and
• 22:12
  you were building this on day one, you would focus a lot more
• 22:17
  attention on. I think in general how complex
• 22:20
  these systems can be, like how many moving parts that can go
• 22:25
  wrong. I think at the beginning it's
• 22:28
  easy to underestimate. Like you, you're testing
• 22:30
  yourself a couple of use cases, it works and then you're good.
• 22:34
  I'm not talking about this agent in particular, like in general
• 22:37
  my experience building agents. But then yeah, you release it to
• 22:42
  users and they will find ways to.
• 22:45
  Multivariate. It, yeah, or not using it the
• 22:48
  way you think because you're someone building agents, so your
• 22:52
  mindset is completely different. So for me the most important
• 22:56
  learning point was to really trust users and get feedback
• 23:00
  from them. Trust users to break the thing.
• 23:04
  Yeah, like get get people that you're that with your opinion
• 23:09
  you value and and ask them to to test.
• 23:12
  Yeah. And how are you reincorporating
• 23:15
  that back into the product? One of the things you can do is
• 23:19
  to get a channel for feedback and have people posting feedback
• 23:23
  and then gathering all this feedback, aggregating it and and
• 23:27
  defining a road map based on that.
• 23:29
  Like what are the the main issues to fix?
• 23:31
  You know you can use LLMS in every step here.
• 23:36
  Yeah, you you can have an agent reviewing feedback and all and
• 23:39
  prompting you things to improve. When you're doing these
• 23:43
  different kinds of visualizations and looking at
• 23:45
  the distribution and all of that fun stuff and you find certain
• 23:49
  things that are out of whack, you're then just reincorporating
• 23:53
  that into your eval set, or I guess where is step one?
• 24:00
  Yeah, it's both. It's like when you find an error
• 24:03
  in in an application and you fix it like you find a bug fix.
• 24:06
  It usually good practice to add a unit test for that if you can.
• 24:11
  So I think having a curated set of of examples and edge case is
• 24:16
  also is really valuable here. So when you find something that
• 24:21
  is a bit off and it's a good idea to include it and depending
• 24:26
  on the severity, then you can you can fix it.
• 24:29
  Like this is more of a product problem.
• 24:32
  Then are you comfortable with that, yes or no?
• 24:36
  I think that's good. Is there anything else you want
• 24:38
  to hit on? If you want to work on put
• 24:40
  projects like this, we're hiring a process, so feel free to
• 24:44
  messages, send us your CV and yeah, let's keep building great
• 24:49
  stuff. Nice.
• 25:24
  Yeah.