Latest / Agentic Conversations (formally mlops.community) / It's 2026, and We're Still Talking Evals

Transcript

It's 2026, and We're Still Talking Evals · Agentic Conversations (formally mlops.community)

0:00
Yeah, I'm a detective. There's not a lot of users that
0:02
are going to tell you I don't like that setting at twenty
0:05
types of different evaluators that are not connected to your
0:07
goal. I would say it's instruction to
0:09
fail. Evils themselves are not very
0:12
hard. You say eval.
0:19
Yeah, it's such a broad term. And my question exactly is like,
0:22
we have a good eval, but then, you know, we have a issue in
0:25
production evil start before you ship the agent or before you
0:29
ship the product. And then you have to completely
0:32
drift from your approach to a different evil.
0:35
So you don't continue with the same evaluation system once you
0:37
have an agent in production because it doesn't match.
0:42
You have new users, you have, I think weird users.
0:45
We can say that some users, you know, you don't expect certain
0:49
types of questions or requests. And to give you an example, like
0:53
you ship an agent that is a shopping agent like, and you
0:56
test versus queries like I want to find Adidas size 37 and then
1:02
another user would in practice in real life.
1:04
So like I want to have shoes like LeBron James.
1:07
And you know, that's a different question.
1:10
And then you have to completely twist your evils from, you know,
1:14
test cases that you simulated yourself to something that user
1:17
created. And then you switch from
1:20
different fader modes to different fader mode, and then
1:23
you go from online evils to offline online evils.
1:27
And I think that a lot of teams do it.
1:31
They completely ship the product and they're like, OK, let's
1:33
start and do evils. And then everybody's like, oh,
1:35
what is evils? And then we go into and to be
1:38
honest, I've heard so many times word evils even when I'm talking
1:42
about it, you know, I get where the confusion comes from.
1:46
So yeah, it's really important to to create something.
1:50
But basically people ship their product and then they come to
1:53
you and they say, hey, can we set up some evals?
1:57
Yeah, that's my favorite phrase. OK, Maggie, we have this.
2:00
Let's run evals. And I'm like, OK, So what do you
2:02
mean? Evils should be constant within
2:07
the development team? Evils start the moment the idea
2:10
of the product starts. So this kind of ensures you that
2:14
when you ship something you are not, you know this is kind of
2:18
evil's pre production is for you not to get like burnt and evil
2:23
in production is to actually do deliver something of quality.
2:27
And by burnt you just mean like putting the right guardrails on
2:30
it? Burnt.
2:31
I mean, you don't, you know, you're not ashamed of your
2:33
product. What is in production?
2:36
Because it's so bad. Yeah, it's so bad.
2:37
It's, you know, you're over complicate things sometimes.
2:41
I think I also mentioned that you ship more features than you
2:44
actually evaluated. So features for me always comes
2:47
later. What is more important a product
2:50
that works, you know, as far as we are concerned good or a
2:54
product that has 1000 features, but you know most of them fail
2:59
and then you know you can ship more features.
3:02
You can, you can test ad hoc different things.
3:04
And that's what I'm also understand through what Thiago
3:07
said about iterations, then you actually set up a different
3:10
filter mode. So you, you actually discover
3:12
them once it's shipped. But the first product you should
3:16
build evals from the very beginning, simulate user
3:19
scenarios. Actually, you know, try to mimic
3:23
somebody's brain, create user profiles and not do it once, do
3:27
it multiple times. Because what is actually
3:30
interesting for me in current lens is that they don't they,
3:34
they understand what they're doing.
3:35
So they understand what the the task that they have to do, but
3:38
what it fails is after sometime they lose track or they do
3:43
something four times well and then the fifth time is
3:45
tragically wrong. So that's the fun part about
3:48
evils. That is not really deterministic
3:51
and it's also not a bag that you can systematically improve.
3:55
So the failure modes, it's something that you should really
3:57
focus on before and after, but that's a specific difference.
4:01
And are you talking about the LLMS losing track when you ask
4:06
them to simulate different personas?
4:09
Oh, that's as well. I mean, that's another point.
4:11
Like we evaluate LLMS with LLMS that are also problematic.
4:16
So yeah, they also lose track. But I think there was research
4:19
on that, like current LMS, they work very good after, you know,
4:23
certain, you know, tasks. So the coherence within the LLM,
4:27
how it works, it, it loses the track of what it was going to do
4:30
or working on. So yeah, that's the case.
4:34
But in terms of simulating your user, you're kind of how we do
4:37
it. I'm going to tell you how we do
4:39
it. You think of different user
4:41
personas, is it power user? Is it, you know, very lazy
4:46
users, busy professional. And you try to describe it as as
4:50
well as you can. And then you call an agent
4:53
yourself. So you simulate the scenarios
4:56
1510, sometimes 100 times if you need to, using different types
5:02
of personas to see how well your agent react to it and what is
5:05
the variance of the outcomes. So if your agent scales, you
5:09
know, 95 four times in terms of, you know, I don't accuracy, I'm
5:14
not really a fan of this performance metric, but yeah.
5:17
Why not? Yeah, it's too.
5:19
It actually measures the average of of the performance.
5:24
That doesn't really give you a quality of this.
5:26
So yeah, I usually use the TNRTPR.
5:29
So how well it's, you know, the human label versus the LLM, it's
5:32
more tangible. And yeah, and I also said that
5:37
it also performs well in terms of like statistical tests, not
5:41
really real user scenarios. And that's also the difference
5:44
between testing with LLM applications and, you know,
5:48
basic software solution. So yeah, it doesn't work well
5:52
anymore in my opinion. I think a lot of people would
5:54
say that as well, but I I very well would like to debate that
5:58
if you think differently. I got you off track.
6:01
Yeah, you were saying that's easy to be honest.
6:04
Like the LLMS? Yeah, yeah.
6:07
Sorry. Now.
6:08
We were talking about how you don't like the accuracy, yeah,
6:13
measurement, but you have the accuracy measurement with LLMS
6:17
as a judge. Yeah, I don't measure very often
6:20
accuracy, because what accuracy tells you about real
6:22
applications like my agent is accurate 95% of time.
6:27
What does it mean, you know? Yeah, I have no idea.
6:30
Yeah, me neither because I didn't define that and I don't
6:34
know what, you know, my agent with I don't know, responds to
6:39
to my users 95% of time. Well, when it asks for, you
6:42
know, food recommendation and there's a specific feathered
6:45
mode that I, I talked about it yesterday in the, in the, in the
6:48
workshop, like when my user asks for vegetarian pizza
6:52
recommendation and you know, we always give 5 recommendations
6:56
and one of them is pepperoni. That's a real use case, right?
7:00
So it's, you have to, that's why I think failure modes are really
7:03
important in that. So you catch a specific
7:05
failures. And what I think people don't do
7:08
very often is that they don't do error analysis because it takes
7:12
so much time and it seems like, you know, so much effort into
7:17
something that, you know, have to change anyway, iterate in a
7:20
week or two. Because as I'm also going to
7:23
come back to Thiago, Thiago said, yeah, you have to do look
7:25
for those fader modes all the time.
7:27
Yeah. So it's it's cumbersome.
7:29
It's really time consuming. And you're talking about trying
7:34
to simulate certain personas, You as the human, putting in
7:38
your time to think of what a different persona would do or
7:42
say. And then you do it various
7:45
attempts. Yeah.
7:46
What I do, what we do in a team is you design a product.
7:50
So as a person that designed a product and build it, you are
7:53
actually having in mind what kind of scenarios this person
7:56
would take. You design different
7:58
orchestrations, so you create different tools and then you
8:01
create different scenarios like, I don't know, 1520 scenarios
8:05
that your potential user would might want to execute with the
8:09
tool or with that product that you have.
8:11
And then what you create, you actually use LLM.
8:14
Then you describe the prompt LLM and describe the persona that
8:18
you might want to mimic. And then you run over those, you
8:21
know, scenarios that predefined and you see how well and what's
8:24
the variance and behavior of your agent.
8:28
So you basically you call an agent with an endpoint and
8:31
that's pretty interesting because usually like one time,
8:34
2-3 times it's go working very well.
8:36
But then you know, it takes, you know, different Rd. after the
8:39
word. So it's really it's tricky and
8:42
it's never the same. That's also another thing.
8:44
It's always fails in a different place.
8:46
Yeah. Oh, that's, yeah.
8:48
Let's get back to that real fast.
8:50
But the thing that I hear you saying in a way is like before
8:56
the product goes out there. What you're describing I had
9:02
thought were simulations. Yeah.
9:05
And then after it's out and you're just looking at the data,
9:08
that's like the offline evals. Is that?
9:11
Or do you see it differently? Yeah, I I get, yeah, I see a
9:14
different, not, not differently the same actually, but I would
9:17
expand on that. So you do the evaluations based
9:21
on the scenarios before the product, you know, is released.
9:25
You can also create a, you know, specific groups that are going
9:28
to test your product. You can ask your team members.
9:32
But I do believe that is a bit biased because you are
9:35
surrounded by developers usually.
9:37
So you create the your data set yourself by, you know, trying to
9:40
break your agent and then you test based on that and then you
9:46
come to production. And that's what I mean like, OK,
9:49
we're sure enough that our product is behaving well in the
9:53
circumstances that we thought of.
9:55
And then we think about, OK, we put it in production, we trust
9:59
it to some extent and then we got our what our user are asking
10:03
for. And that's another part of
10:04
evaluation because user are going to come up with completely
10:07
different things and completely different filter modes.
10:10
And then you can still simulate your agent.
10:13
For example, if you introduce a new feature, you make sure that
10:17
your feature does not ruin your product or you introduce, you
10:22
know, change to a system prompt for your agent.
10:24
That's also something that you should test as your AB test
10:28
before you push it, at least to some extent, because of course,
10:31
again, we're dealing with two different animals here.
10:33
So before and after. So you make sure that on your
10:37
side, you did everything to push the product certainty as high as
10:42
you can actually imagine. And then, you know, you put it
10:45
in the wild and you, you, you kind of, you know, allow it to
10:48
work in the wild and see what's going to happen there.
10:51
And that's another part of the story, so.
10:54
How much can you get from just trying to tell the LLM to be as
11:00
unhinged as possible? Is the real world always going
11:04
to be more unhinged? About now, I thought about it.
11:08
To be honest, that's not something I've put in before,
11:10
but if you say like if you actually try to teach your LLM
11:14
the unhinged behaviors of your previous users, you can actually
11:18
pick up on it quicker. Pretty far.
11:20
But yeah, I think it's always something that we're going to
11:23
come up with. There's always going to be some,
11:25
you know, an unexpected behavior of LLM that you want to catch.
11:28
But you're you're right. We can try to be unhinged, like
11:31
on purpose, even in the team, like, Oh yeah, let's have a
11:34
break at the agent exercise. That's a yeah.
11:37
I was thinking like this could be fun activity for the team.
11:41
Anybody that breaks it gets 50 bucks Amazon gift card or
11:44
whatever. You know.
11:46
So that you really are trying to get people thinking outside the
11:48
box on what they are doing. It's tricky in a way.
11:54
I think a lot of those topics are still too much conceptual,
11:57
not practical, and I think there is not really a DNA of evils
12:04
yet. A lot of people talk about it
12:07
all conceptually. We could do that, we should do
12:09
that, and then eventually it's this forgotten piece of
12:13
development. And why is that?
12:15
Just because it takes so much time and effort.
12:17
I think so. I think it's also not very
12:19
interesting to some extent. Yeah, it's kind of boring.
12:22
Yeah, it's boring. It takes a lot of time.
12:24
Yeah. Yeah, and it's, you know, I hear
12:27
sometimes like, oh, Maggie, why do you want to evals?
12:30
It's boring. I'm like, why do you think so?
12:32
It's not because it of course the time consuming exercises of,
12:37
you know, open eggs or coding or whatever you call it.
12:41
Yeah, you have to really read into it to to find a mistake to
12:45
fight a failure mode. So these part are not.
12:49
But I think the fun part about evils is when you try to
12:52
translate that into a working and production solution that
12:56
enables you to catch something sooner and then you like have a
13:00
hot moment. Oh OK, so I actually helped you
13:03
with catching it. Because you preemptively catch
13:06
it. Yeah.
13:07
And also, like, if you think about it, the fact that we have
13:11
so much work with evils to make it work, it's also for me,
13:15
motivation to in some way try to enhance it and automate it.
13:20
And I know it's also, you know, slippery slope because you
13:24
should not trust LLM with helping you out with something
13:27
that, you know, you judge another LLM.
13:29
But there are ways that you can actually do it.
13:31
And I think a lot of people get very much hesitant or like,
13:35
reluctant to go through the error analysis because first, I
13:41
don't think they even know how much they can get through it
13:43
because they never get through 1.
13:45
So there's like, OK, we never done that because it's, you
13:47
know, boring. It takes too much time.
13:49
But it's like, you know, trying something for the first time and
13:52
then afterwards, like, oh, damn, it worked.
13:55
Yeah. And then the second time is the
13:57
consistency. So we did it once.
14:00
And then I'm OK, now let's run evils.
14:01
And then nobody comes back to the topic.
14:05
So it's just like, hey, we did that already.
14:06
Yeah. Isn't it fixed?
14:07
Yeah, yeah. The educational matter in this
14:11
part is also very important. I think a lot of teams don't
14:13
understand that it's a duration. That's a process that you have
14:16
to come back to, and I struggle with that sometimes, you know,
14:21
when I do something once and they're like, oh, yeah, we put
14:24
so much time into it and I'm like, OK, like we're going to
14:27
meet again in three weeks and do it again.
14:28
I was like, what? When you're getting pushed back
14:31
because, oh, we did something, we put so much time into it, it
14:35
made the product better, are you just not able to show how much
14:40
better the product is? Yeah, to be honest, that's a
14:44
very interesting question because I think there's a lot of
14:46
noise as well there. So you pick up on a lot of
14:50
errors and then they're being corrected or, you know, you're
14:54
trying to prompt your agent differently.
14:56
And everybody is like, oh, we need evils, but we don't want to
14:59
put that much time into evils. So they see the outcome of it.
15:04
But I think a lot of development teams are quite messy and we're
15:08
working on a lot of things at the same time and you know, 5
15:10
different directions. So Eval's is not only trying to
15:13
keep up with that, but also catching up and also trying to
15:16
fix sometimes some stuff. So I think a lot of people see
15:20
value in it, but at the end of the day, it's still a mentality
15:25
that you're going to go and fix something else, then go and try
15:29
to develop. Also I've heard eval's, I think
15:33
the simplest way that a software engineer understands evals is by
15:39
saying, oh, it's like testing for AI and tests you kind of are
15:43
updating a lot but not that much.
15:46
And tests also aren't something that are going to take you a
15:50
half a day or two days of work to figure out unless I mean,
15:54
maybe some tests, but for the most part it's not that big of a
15:58
time investment. No, it's not.
16:00
And that's a big difference. We talk about unit tests versus,
16:04
you know, NLM testing and yeah, they usually, yeah, it's not
16:09
really the same answer. We're not looking.
16:11
I think a lot of people compare them and here we are asking
16:14
completely different questions for those tests.
16:18
So I think we're also, it's a big misconception for some
16:22
people. Like we did unit tests, we don't
16:23
have to do evals or it works well here, but we don't measure
16:27
actually the trustworthiness of our LLM.
16:29
We don't measure the. You know how well it behaves in
16:32
this circumstances. The semantic matter in this in
16:36
this is actually not measured for unit tests.
16:39
And Speaking of the evals and going back to the big question
16:42
at the beginning, what are good evals?
16:44
Yeah. What are things you're measuring
16:47
in a good eval to make it? Is it on a system level?
16:50
Is it all these different? Because I remember we had a talk
16:54
from a guy from Meta and he had 12 slides with 12 different
16:59
things you should be measuring in your evals and each slide had
17:02
like 5 or 6 sub components of what you could be measuring.
17:06
So it's like, if you really want to measure everything, it can
17:09
get very nuanced and very cumbersome.
17:13
Yeah, that's true. And you can get everybody
17:15
discouraged. Yeah, yeah, whatever you what
17:19
you measure in your evals is very much custom made for your
17:22
use case. And the first thing I always ask
17:25
myself when I start an approach a project with Evo's like what's
17:28
actually the goal of this? What's the goal of the product?
17:31
What do you want to achieve with the product?
17:33
Who are your users? And the most interesting
17:36
question and the, I think the one of the most important one is
17:38
what is the definition of good? So can you.
17:41
Give me an example of what that like actually played out in.
17:44
Yeah, like, OK, for our food ordering agents, there was a
17:51
case, we had a lot of interactions and you know, it
17:56
was interesting case because I think we also imagined how the
17:59
user would want to interact. So we've we came up with kind of
18:03
like surprise me intent and create a bit of an issue for
18:07
ourselves because how do you answer a question, surprise me
18:10
and then suddenly people ask surprise me and it's an
18:13
interesting case. Like how do you actually come up
18:15
with something that might be useful, but makes a bit of more
18:18
noise for you? So here, who is your user?
18:22
There are people that are probably, you know, undecided.
18:25
They are maybe in a rush. They are overwhelmed and
18:29
overestimated with number of options on your on your, you
18:33
know, homepage or just definitely they just don't know
18:37
what to eat or have an occasion to organize.
18:39
And that's your use case. And then you think like, how
18:42
would they behave and what's the definition of good?
18:44
So you have to think if it's better for me to deliver
18:48
something that is quicker of, of good quality, personalized.
18:53
And then you have to approach this case 1 by 1 and they ask
18:56
yourself those questions and then you think like, OK, So what
18:59
are my evils? So probably my evils would be
19:01
the recommendation, right? So what would, is it the
19:03
personalized accommodation? Is it also, you know, how well
19:08
we execute the task? What is the coherence, how well
19:11
we remember the context of a conversation?
19:14
If somebody is vegetarian, we don't, you know, offer meat.
19:18
And this is a specific use case. So would you apply the same, the
19:23
same evaluators, the same metrics to a use case about, you
19:27
know, car dealers? You know, that's why I feel
19:31
like, Oh yeah, this basic framework, you can put it like,
19:35
OK, I want to compare myself, which I also don't know how
19:40
sometimes because they're so different, but usually it's just
19:43
a very different set of evaluators and approaches, a
19:46
very set, different set of business measures that you align
19:48
it with. Yeah, you're waiting.
19:49
Certain pieces of this interaction stronger than
19:53
others. Exactly, Plus you know for you
19:57
know car dealers options that we our team is building an
20:01
automotive for example, probably the satisfaction measurement is
20:05
going to be different than for the food ordering.
20:08
So you also remember that you tied it with the with the
20:11
different business metrics. So in the evaluation of footer
20:15
ring agent, we try to combine it with the conversion.
20:19
So we also test happy paths. We take all of the conversation,
20:24
we match it with the conversion and we actually look which
20:27
conversation with what evaluator outcome ended up in conversion
20:32
or which ended up with frustration.
20:36
There are also certain cases when users use the product then
20:40
they're very frustrated but eventually end up buying
20:43
everything. So it's tricky.
20:45
It doesn't make sense. No, no.
20:47
You know what? It's so funny.
20:48
And I feel like you could easily catch a lot of frustrated users
20:55
if you just had some kind of simple metric for was caps lock,
20:59
caps lock on. Yeah.
21:02
And if you have that show up anywhere in any conversation,
21:06
you should know, like who we should look into this one a
21:09
little bit more. I agree and disagree, I think a
21:13
lot of people are just lazy and consider myself in this group.
21:16
Yes, I have caps lock when I'm not going to rewrite myself, you
21:20
know, and we got I also felt like first Oh.
21:23
You have false positives, yeah. I was thinking about it because
21:26
I also was reading a lot of interactions in in caps like and
21:28
I'm like, oh, OK, that's weird. And then I figured out one day I
21:32
was writing something and I forgot I have a caps look on and
21:34
I sent it to some AI till I was like, oh, OK, now I get it.
21:40
So, you know, that's also something we want to pick up on
21:42
the frustration signals. So it's something that you, you
21:46
know, is it, for example, given the example of recommendation
21:51
with somebody ask you all the time more, give me more, give me
21:55
more. It means like we failed three
21:57
times in a row. So you also have to, you know,
22:01
just kind of step into somebody's brain and think of
22:04
why and how this person could get annoyed and how can I catch
22:07
it. So yeah, it's not already
22:11
obvious. So that's why it's really
22:13
important to try to understand and step into students.
22:15
We also have some kind of help from, you know, more like groups
22:21
from, you know, we have that kind of group of lawyer users in
22:25
terms of that. So they sometimes test out
22:27
things before we actually put it in the the whole market.
22:31
That's that's really good. But yeah, it's not obvious and
22:35
that's why I don't believe that there are a certain set of
22:37
metrics that can help. You silver bullets because I
22:40
thought another silver bullet is if there's curse words in it.
22:43
As soon as I start saying you fucking it, I would figure that
22:47
is a very high frustration margin.
22:50
I don't think there's anybody happy redirection saying that
22:53
so. Somebody might just be angry
22:55
because they had a bad. Day I wonder like you know if
22:57
LLM is just in the receiver of those frustrations you know and
23:00
then I'm like oh if the people real life are talking to users
23:03
like that I'm just sorry you know it's.
23:05
Not I think they do. Yeah, I.
23:08
Am a little bit more reserved when I talk to a real human, but
23:13
if I know it's a bot on the other end, I'll be and I I think
23:17
of it, I'm like I'm doing God's work here.
23:18
I'm helping the evals on trying to make it very easy for you on
23:23
the other side to know that this was a shitty interaction.
23:26
I wish, I wish that actually would be true for a lot of us,
23:29
for a lot of users. We had a conversation yesterday
23:31
because I created an educational form of traces, and I had a
23:36
comment that was very good comments in a way.
23:40
Where did you get this data from?
23:41
Those users are very informative.
23:43
Like they tell me so much about what they don't like, what they
23:46
like. And I'm like, oh, yeah, I had to
23:47
make it up for us to make it quicker.
23:50
Yeah. And I think a lot of users, and
23:52
that's also what evils, why evils matter, is they just drop
23:56
off. There's not a lot of users that
23:58
are going to tell you, I don't like that.
24:01
So these are the, I think the biggest loss loss that you can
24:05
have is the user that are, you know, talking to you and
24:08
suddenly drop off because they're not satisfied.
24:10
But it can be a success too, right?
24:12
Because they find what they need.
24:14
Yeah. But yeah, it's also usually
24:18
it's, we have, when I think about the analytics part, we had
24:22
a metric that was called user satisfaction and was trying to
24:25
mimic more or less what user satisfaction means, coming back
24:29
to what is definition of good, because nobody knew what the
24:32
satisfied user is. So we came up with it.
24:35
So you have, I'll say 15% of users that are satisfied, then
24:42
you have 15% of users that are not satisfied.
24:45
And then you have big chunk of users that are partially
24:48
satisfied. What it means because those are
24:51
the user that dropped off. They didn't tell you anything.
24:54
They didn't end up with conversion, or they did, but in
24:58
a very mysterious way. Like this conversation shouldn't
25:01
have ended with conversion. Like this is not the way we
25:04
should work. So this is very mysterious.
25:07
So then you try to, you know, figure this out.
25:10
And yeah, that's actually fun part about evils.
25:14
And I don't know, I love this part because then you, you know,
25:18
yeah, I'm a detective. I'm like, OK, imagine I'm a user
25:21
that comes to this platform and I want to, you know, literally
25:24
step into someone's brain. I think of, oh, OK, that would
25:27
be better. Or, oh, I guess that was it.
25:30
And then this way you come up with the metric that actually
25:32
matter for your, for your platform, for your app.
25:35
But then how do you reincorporate all this eval
25:38
data? Yeah, back into the product to
25:41
make it better. It really matters on a team and
25:44
how well you structured it because usually we had, you
25:48
know, we make reports, we have weekly meetings and in a
25:50
corporate setup it usually works like this.
25:52
So you have some kind of outcome or users are, you know, dropping
25:56
off after minimum order barrier or, you know, have a lot of
26:00
pricing discrepancies are getting lost.
26:02
And it usually is just like iteration, Sprint iteration.
26:08
And I'm not really a huge fan of that in a way that you have to
26:11
wait. Oh, yeah, we have to focus on
26:13
that. But we're going to pick it up in
26:15
the next Sprint. And like, OK, so for the next
26:18
few weeks, somebody's going to, you know, bump into the wall.
26:21
There are things that are more prioritized, obviously, that in
26:25
a corporate setting. It's not like we try to work as
26:27
a startup for sure. But if you have, you know,
26:29
multiple projects, then it's hard to put a structure on
26:33
something without putting a structure on it.
26:35
So everything, you know, you know what I mean.
26:37
It's just it's a tricky fine line between those two.
26:41
But what do you actually do? You get the data and then you
26:44
increase. You change the prompt.
26:46
Oh yeah, that's OK. It's really depends because when
26:49
you take the data and you look for failure modes, you look for
26:54
them and then you find, you know some of them might be, we call
26:58
it specification issue. So issue that is caused by us
27:03
more or less. So it's a comprehension between
27:05
you know how the agent work and how the developer thought it
27:08
would. So some issues are caused by by
27:12
us, but not really prompting the agent correctly.
27:15
Or giving it the wrong context in the wrong time.
27:17
Yeah, or not really specifying things.
27:19
So, you know, when we specify an agent be helpful in the
27:21
situation, what does it mean? You know, it also depends on the
27:25
context. In the context of customer
27:27
service, hopeful is different in the context of, you know, food
27:29
ordering or you have to give it the context.
27:33
So the LLM knows what what helpful means in your case,
27:36
sometimes it's just a simple prompt change that can, you
27:40
know, decrease that failure mode for us.
27:44
And of course, that's also an evaluation play part when you
27:47
can, you know, detect the regression between different
27:49
prompts. So you have to have, you know,
27:52
trustworthy evaluators for that. Don't you feel like you're
27:55
playing whack A mole, though? You change the prompt a little
27:58
bit and oh cool. It's evaluations are better over
28:01
here, but they're worse over here.
28:03
Oh yeah, that's, But that's also why you want evaluations, right?
28:08
You want to know what failed? Why?
28:10
And to be honest is I think it's a little bit of dance, right?
28:14
You're dancing around trying to change the prompt.
28:17
And either you do it without, you know, choreography, without
28:20
music, you know, completely blind, or you do it with evos
28:24
when you try to, you know, track the regressions between the
28:27
changes that you make. And of course, there are also
28:29
different prioritization that you're going to, you know,
28:31
correct this issue by switching the prompt.
28:34
They're going to create a new one.
28:35
But which one is more prominent for a use case?
28:38
Is it better that the age hallucinates in the minor cases?
28:40
Or is it better that they recommend something different
28:42
or, you know, it's, it's an art that you have to, you know, have
28:48
for every, each of the use case that you have as a as a, you
28:51
know, developer or product owner.
28:54
Have you resigned to just knowing that you're never going
28:57
to get 100%? That is not going to happen.
29:00
I hope I won't. You hope you won't?
29:03
Yeah, because something's broken if you're hitting 100.
29:06
Percent there's always something broken and tell something tells
29:08
me that's 100% correct I'm like yeah, this bullshit so I hope I
29:12
won't because I'm I'm not you know that's that's a trustworthy
29:15
metric in me it's 95% I'm also like OK, that's really high yeah
29:20
something. Fishy with this data.
29:23
The key, I guess when you're looking at it is just where
29:27
you're having the issues. It's not necessarily can we have
29:33
100% success or 95% success. It's really like out of all of
29:38
the big flows or the uses that we want, we want to make sure
29:42
that we're the strongest in these certain ones because they
29:46
add to conversion or they increase the likelihood of the
29:49
conversion. Is that kind of how you look at
29:51
it? Yeah, I think so.
29:53
You know, we of course, everybody's aiming 200%.
29:57
So the higher we get, the better.
30:00
It's also like after a while when you're measuring something
30:03
and it's, you know, reaching almost 100% all the time, maybe
30:08
try to focus on something else. And to be honest, it's it's a
30:13
neverending story in my opinion. That's where the fascinating
30:17
part of this for me is how to, you know, make it more automated
30:19
and interesting for for the team to do.
30:23
And it should be part of the DNA.
30:24
So it should be a constant dance of like trying to figure out
30:28
which metric, which evaluator is better for you, what you're
30:31
trying to aim. That's why, you know, also
30:34
setting at twenty types of different evaluators that are
30:37
not connected to your goal and what's hiding inside of them.
30:42
It's kind of a receive, the kind of, I would say instruction to
30:46
fail because then you measure 20 different things not related to
30:50
what you want to achieve to any different things within inside
30:54
of them. A lot of things are fading that
30:55
you're nowhere not aware of. Then you're aiming to hit 10%,
30:59
but you know, a lot of them are sometimes overlapping.
31:03
So you measure different evaluators that are overlapping
31:07
themselves and then like, OK, it's confusing.
31:10
You know, that's why you know, it's, I think a lot of people
31:14
talk about it within agents, right?
31:16
You start simple. First you had multi agents.
31:20
We jumped on that like a multi agent, so crazy.
31:23
You're building something so nice.
31:24
And then we said, I'm like, oh, it doesn't work.
31:27
So with evos is a bit the same. You start slow, you start, you
31:30
know, to get as far as you can with what you have and then you
31:33
build up on top of that. And you know, you always make
31:37
sure that your core is the most reliable that you can have.
31:39
So your core evaluators that are tied to your business metrics
31:43
are tied to your behavior. You know, it's going to be
31:46
always a bit different sets, right?
31:48
So as mentioned before, it's a constant search.
31:51
So but they're always going to be like you're a group of like,
31:54
you know, your puppies that you're going to always want to
31:56
take care of and you see that they're very much correlated or
32:00
related to your business metrics.
32:03
And then on on top of that, you can build something more
32:06
sophisticated AB testing suite, you know, just check on the
32:09
regressions. You know, we also started to
32:12
experiment a bit with DSPI, so trying to, you know, improve the
32:16
prompts if you showed examples. So there's a lot of even
32:20
simulation of of the agent that we talked about.
32:23
I want to know about eval tooling and all the the good,
32:28
the bad and the ugly. Yeah, I I don't use them that
32:32
much beside observability so far I think.
32:36
Observability like Datadog observability.
32:38
Yeah, it's just, you know, basic metrics or just having a tool
32:42
that is showing your conversation and a nice way.
32:45
So you are you're allowing your team to see the conversation
32:49
turn by turn. And yeah, that's my main thing
32:53
purpose because I think there's a lot of, as you said, there's a
32:55
lot of them out there. And as I mentioned before, it
32:58
links very well to what we talked about.
33:01
Every use case is different to so name a platform that is able
33:04
to, you know, match your use case.
33:07
And all those evaluators are so outdated sometimes, like they're
33:11
breaking basic rules. That one that you can also
33:13
create and observe in the tools and themselves.
33:17
Like if I'm going to see again that is evaluated, that is on a
33:19
scale of from zero to 1, like I'm not going to trust in any
33:22
other thing in this app platform because I, I hate this.
33:25
Like, why would you? We talked about it during the
33:28
workshop. Like if you have a .6, you know,
33:31
on a scale about the hallucination.
33:35
OK, so you wake up on Monday morning, my evaluator is .6.
33:38
And I'm like, what does it mean? Give me pass or fail, right?
33:43
Give me the the real actionable insight.
33:45
So coming back to the tools, I do have my favorites in a way.
33:50
I call them friendly tools because they're not really
33:53
useful yet for me, but they're friendly because they help me
33:56
with my work. So they enable a quick export of
33:59
the data. So you know, there's my pipeline
34:01
is working without impediments and there are a lot of tools out
34:05
there that are, there are huge impediments on that.
34:08
If I export more than 1000 traces, it suddenly slow down.
34:12
I have to do things in batches and it takes hours.
34:15
So imagine you want to run EVO and then you wait hours for for
34:19
your data set. They also don't enable sampling
34:22
as far as I'm concerned. And there's another thing, why
34:24
would you run Evo's in every single trace of yours if you
34:27
have 100,000 traces? And imagine you have 6
34:31
evaluators, 100,000 traces, then you want to do it on turn level
34:35
or multi turn level, so conversation and then you know,
34:40
you have to pick your strategy for that.
34:44
You don't want to spend thousands of dollars just to run
34:47
evaluation to tell you that five of them were wrong.
34:49
Yeah, So that's what I think. Not a lot of them enable multi
34:54
turn conversations. If they do, I don't think in a
34:57
very good way as far as I'm concerned.
35:00
That's why I look at them how well they enable me the export
35:05
of data. Not only for UI, but I'm talking
35:07
mostly of about APIs, how they, how will they work?
35:11
I will still look how well the dashboard looks like, because I
35:15
don't want to interest another tool to make only dashboards.
35:17
I don't want to, you know, have my observability tool with all
35:20
the traces, do my custom pipeline and then on top of that
35:24
also introduce another tool that I'm going to put my dashboards
35:27
in. And then I'm like, OK, that's
35:29
it's trivial, but it's also not really well yet.
35:33
Like it's not looking good. But for the team matter, if I
35:36
want to look at the error rates, I want to look, you know, how
35:39
what's the changes? What are the regressions?
35:41
It's also important. So custom friendly too is
35:44
something that I would look for and it's not a lot of them out
35:48
there. I work with Arise very often.
35:51
I do see Phoenix or Arise the enterprise version.
35:57
There are some issues still with this, so I'm not like 100%
36:01
happy. I'm not going to go for UI
36:03
solution inside of it to create my evaluators, although I tried
36:07
and funny enough, it's way harder to create an evaluator
36:10
within the platform that if I would just write a code for it
36:13
because you have to, you know, every data set is different
36:16
every, each of the tool, whatever you choose.
36:19
And then you have to match them try to do with a multi turn
36:21
level. I'm not sure if that even
36:22
enabled. I tried, I was, you know,
36:25
tweaking it, you know, I spent the whole day and I was annoyed
36:28
because I lost it. So yeah, I don't know if that's
36:32
a hot take. I just don't like them.
36:35
So I just, you know, try to make sure that they're as friendly as
36:38
they can be to enable my work to be as quick as it can be.
36:42
And then, you know, I'm looking forward to maybe explore
36:45
something more if there's going to be more options.
36:47
I'm sure there's going to be plenty more that come out.
36:49
Yeah. Is there something that you wish
36:51
you had besides the custom ability?
36:55
Yeah, I do, but actually built it.
36:57
So tell me more. Yeah, I was thinking about
37:01
everybody is, you know, enabling you to push out your evaluators,
37:05
all these experimental things, AB testing suite.
37:09
And I'm like, OK, but where is the basic?
37:11
So how do I train my evaluator? How do I, you know, approve, go
37:16
through actual coding without opening Excel?
37:18
Because that sounds really funny to me that even in the very
37:22
famous courses that are promoted on LinkedIn, we do ASO coding
37:25
and open coding in Excel. I'm like, OK, when I seen it,
37:28
I'm like, OK, this is not normal.
37:30
So we're the tools that are enabling that to go through the
37:33
whole process of discovering your failure modes, sampling
37:36
your data, which is not really there yet in the in the
37:38
observability tool. Then, you know, create a good
37:42
interface to go through the traces without, you know, losing
37:46
yourself. And then, you know, split the
37:48
data training, validation. So the basics actually about the
37:52
evaluator and then pushing into production.
37:55
So I would say a thing that would enable me creation,
38:00
creation of evaluators that are reliable and I don't see it yet,
38:06
but yeah, we solved it internally and the.
38:10
Thing is just so easy to build now.
38:12
Yeah. But especially for things that
38:15
are internal for your team's use case, it's like, do you need to
38:19
go out on the market and then have a company go through
38:22
procurement and make sure that all the data sharing privacy
38:26
laws are right if you can just build it?
38:28
Yeah, yeah, it's, it's tricky and I think a lot of people just
38:33
fall for it and I don't want to fall for it.
38:38
And I very much always, you know, discuss that and go in
38:41
detail. What's your methodology and
38:43
training the evaluators? What do you think are your
38:45
starting points? Another thing, outsourcing the
38:48
labeling, No, that shouldn't happen.
38:52
Like why would I, you know, why would I leave the labeling part?
38:55
First, That's the greatest opportunity for you to learn
38:59
what's happening. Second of all, why would you
39:02
trust anybody else if it's your product and you know how you
39:06
want to assess it? So basic thing, why would I do
39:11
it? So a lot of those companies are
39:13
still trying to sell you something that you're going to
39:15
as source of labeling. This is the quickest way you're
39:18
going to create evaluator about your house nation.
39:22
Good. Thanks.
39:23
Yeah, yeah. And you're very opinionated
39:26
about how you want your evals to be, and you know from experience
39:30
what has been working and not working, so it makes sense that
39:33
you would want more of a custom solution.
39:35
To be honest, mature product teams are in my opinion, are
39:39
going to usually go for custom solution because evils
39:45
themselves are not very hard. Like if you think about the
39:48
process to grasp the complexity is maybe something hard but
39:53
simple task. There are simple tasks that you
39:55
have to do, but you have to structure them in a good way.
39:58
You have to be consistent, you have to iterate and then you
40:04
have to have your own strategy. And I think the alignment with
40:07
being team members and alignment on what it, what matters for
40:11
your product is the part that is the trickiest part.
40:14
And the execution part is like, yeah, we finally want to see
40:16
numbers part. Is the last step in it.
40:21
So and to be honest, I think a lot of solutions are trying to
40:25
sell you something that is, you know, trying to replace the
40:29
first crucial part of the evaluation.
40:32
So the alignment on it. And I don't think it's
40:35
replaceable. I don't think it's something
40:38
that we can, you know, ultimate very quickly.
40:41
And yeah, that's why it's failing.