Latest / Agentic Conversations (formally mlops.community) / Making AI Reliable is the Greatest Challenge of the 2020s // Alon Bochman // #312

Transcript

Making AI Reliable is the Greatest Challenge of the 2020s // Alon Bochman // #312 · Agentic Conversations (formally mlops.community)

0:00
So it's Alan Bachman. I'm the CEO of Rag Metrics.
0:05
Our website is Rag Metrics dot AI and I don't drink coffee.
0:10
Demetrius I I'm sugar is my is my vice.
0:15
We talk deeply about how to bring the subject matter experts
0:19
into this eval life cycle, if you can call it that.
0:25
We also talk deeply about evals. As you could guess since we were
0:30
on the topic, he is doing some great stuff at Rag Metrics on
0:34
LLM as a judge. Let's get into it and a huge
0:37
shout out to the Rag Metrics team for sponsoring this
0:40
episode. Dude, I just want to jump right
0:49
into it because you said something when we were just
0:52
talking before I hit record and it was all around testing each
0:57
piece of the entire system. And I am always fascinated by
1:03
that because there are so many different pieces that you can be
1:09
testing and you can be looking at.
1:11
And I know that a lot of folks are trying to go about this.
1:15
You had mentioned a, do you want to try and use big chunks or
1:19
smaller chunks? Do you want to try and use one
1:21
agent or 10 agents? And so there's all this
1:25
complexity that comes into the field and especially when we
1:29
start having more and more agents involved, how are you
1:33
looking at that? How do you see that and how do
1:37
you think about testing all of that?
1:41
Yeah, absolutely. So first of all, it's let me
1:45
just talk about the approach a little bit because I think that
1:48
a lot of engineers out there are, you know, we're all new to
1:55
this thing. And so it's natural when you're
1:57
new to something to just kind of take somebody's word for it.
2:00
So we read what, you know, maybe what the big labs have to say
2:04
about how to prompt our models, or we read what the, what the
2:08
vector database vendors have to say about how to set up, you
2:12
know, vectors. Or we read, you know, and then
2:14
there's maybe like a YouTube influencer that has like an
2:18
idea. So, you know, if they're
2:19
popular, if they're charismatic, then we listen to what they have
2:22
to say. And that's all fine.
2:24
Like you, you know, we should listen to everybody.
2:26
But you probably didn't get into engineering just to like take
2:31
other people's word for it. You probably get into it because
2:33
you enjoy building, you want to see it for yourself.
2:36
And this is an amazing opportunity to do that.
2:38
Like amazing. Because let me tell you,
2:41
Demetrius, nobody knows what's going to work for your task.
2:44
Like definitely not me, but definitely not like open AI
2:47
Anthropic. They don't know, you know, we
2:50
V8, they don't know. Like nobody knows.
2:52
We're inventing this as we go. And that is awesome because if
2:55
you get into the mindset of I'm going to follow what the data
3:00
tells me, then you will come up with some amazing solutions.
3:05
And that is how like every magical product that we hear
3:10
about that I know of when you know, I, I haven't spoken with
3:13
all the teams that are building magical products.
3:15
But like every, every time there's a magical product, there
3:19
is a data-driven process behind it where the team that's
3:22
building it is not taking anybody's word for it.
3:24
They're actually like trying out the different models, trying out
3:27
different prompts, trying out like different configurations of
3:30
an agent. And sure like you don't, you
3:34
can't try everything you definitely had.
3:36
You know, you can't just shut the world out.
3:39
You you have to learn, you have to build on the shoulders of
3:42
giants, but you have to build like, and it's it's awesome.
3:46
It's an awesome privilege to build like it's you'd be super
3:48
boring if we figured it all out. We probably all move on to some
3:52
other place. So so this is like this is the
3:55
good part. Yeah, there would be a lot less
3:58
hype influencers if it was all figured out already.
4:00
That's right. You know, I think that let let's
4:04
take like AI don't know, take a ragpipe, a typical rag
4:07
application that you're building.
4:08
So if you are following sort of this like empirical philosophy,
4:15
it you'd probably set up something that is pretty common,
4:19
right? Maybe you'd set up the easiest
4:20
one. So you have you take the most
4:22
popular model, take the most popular vector database.
4:25
Maybe you have like 1 prompt, 1 task.
4:28
And then you think through sort of what, what do you want out of
4:32
that step in the process? Like what outcome are you
4:36
looking for? Regardless of what technology
4:38
you use, think about like, what does it mean to generate value
4:42
from this? What are your users like?
4:44
What are your requirements? And then in my opinion, you set
4:49
up an evaluation. All an evaluation is is, you
4:52
know, you set up a couple of examples.
4:54
If a user asks me this, this is what I want out, this is what I
4:57
don't want out. And you come up with like it
5:00
doesn't have to be a massive amount, could be like 30-40 of
5:03
them. Like it's totally possible to
5:04
write out an e-mail in a couple of hours.
5:07
It's not like a major project, but it is a major sort of
5:13
journey of self discovery because you're thinking through
5:15
like what you want. You're thinking through what you
5:17
want. It's a way of taking what you
5:20
want from your head and putting it on a piece of paper.
5:23
It's not unlike writing a requirements document, but it's
5:25
like requirements for AI. And, and then you have this,
5:29
this Greek, Now you have a benchmark and you can actually
5:32
benchmark like different models can give you really different
5:36
results. You'd be surprised and opening
5:39
Eye and Anthropic and Mistral, they have no clue which of their
5:42
models is going to be the best one for your.
5:44
I assure you they don't know. Nobody will know until until you
5:49
run this benchmark on your task. And the same thing with like all
5:52
the other pieces of the of the pipeline.
5:55
Yeah, the the funny piece is you have that trial and error and
6:01
you have to test it out and see in a way, like how few examples
6:09
can you give to get a bit of a signal and at least know
6:14
directionally we're going the right way.
6:19
But then later to get that last 10%, it is just mayhem.
6:25
And that's why I think it's fascinating tracking all of the
6:29
changes across the whole life cycle.
6:32
And it reminds me a little bit of Jupiter notebooks and how you
6:39
are, you know, you have final dot whatever I pop and and here
6:45
it's in a way that exploration is happening just like you were
6:52
doing this data exploration back in the day.
6:55
And so now you're trying to track it and you want to know
6:58
each little thing that's changed and you want to see against the
7:02
eval set. How are we doing here?
7:06
Right. I think it's not unlike building
7:09
code, right? You, you start with you just
7:11
start with a happy path. Probably.
7:13
Maybe you're starting with a demo and it's like famously easy
7:16
to make a demo with AI, right? Because it says some crazy
7:19
stuff. You try enough things, it's
7:21
going to say something amazing. If you just show that like you
7:24
don't show any of the other stuff, you're going to look
7:26
fantastic. It's not hard to do, right?
7:28
So you make a demo, maybe you get a little bit of buying, you
7:31
get a little bit of support, maybe some resources, and then
7:35
you have the next step, which, you know, maybe you know, some
7:38
people call it hill climbing, but it's just like, you know,
7:41
instead of just one happy path, maybe you come up with like 10
7:44
or 20 happy path. So whatever it is that your chap
7:48
out or copilot or AI step is supposed to handle, just think
7:52
of a couple different examples. So it's not just one path.
7:54
And that's usually also fairly easy.
7:57
It's not as easy as the demo, but it's really not hard.
7:59
You're just thinking through a couple of different, you know,
8:01
examples and then like you said, like it gets bigger.
8:06
Hopefully, you know, maybe you, you have a second, then we have
8:09
a bit more buying. Eventually you let users in.
8:13
When you let users in, it's just it's going to hit the fan.
8:16
It's going to hit the fan in the most amazing, beautiful way
8:20
because they will put stuff in there that you just would not
8:24
imagine in a million years. Like I, you know, you, you
8:26
always know like the guy who was like when testing software
8:30
became like a popular thing, there was this guy was like, I
8:32
don't even know automated testing tools.
8:34
You just bang his head, bang his fingers on the keyboard, all
8:37
like 7 keys at once and it would break like a lot of software
8:40
that way. I'm talking about like 20-30
8:43
years ago, like that was a really good way to test
8:45
software. You just bang the keyboard.
8:47
Well, that's what users do. Like they will really bang up
8:49
your bot in a way that you just like never imagined.
8:53
And it gives you the scope to just like make it a lot better.
8:57
And it's like you said, you know, the challenge is that it's
9:01
very easy for, for an engineer to focus on the, the edge case
9:08
that you're fixing and like break a whole bunch of other
9:11
ones because you don't, you don't know that you broke them.
9:13
You just change the prompt. The prompt used to work for 20
9:16
cases. You're focused on case number
9:19
21. You're working really, really
9:20
hard on it. It's not easy to get like
9:22
shoehorn AI to do what you want on case 21 and you finally get
9:26
case 21 working. Yay.
9:28
Oh my God, cases one through 5 just broke.
9:30
You have no idea until the next user comes in tries that.
9:34
So like to me, the solution for that is to build up that, build
9:41
up that eval. So every time that you run an
9:43
eval, don't just run it on case number 21, run it on all the
9:47
cases. And then if you broke something,
9:49
you will at least know right away.
9:51
That's kind of like the first step to fixing the problem is
9:55
knowing the problem. How many evals are enough?
10:03
That is a bit like how long is a piece of string.
10:05
It depends on, you know, it depends on what you're building.
10:08
I mean, if you're building a, you know, I don't know, an
10:13
entertainment application poetry generator, probably fewer than
10:19
if you're building like, I don't know, a government service.
10:23
Depends on the stakes, depends on the breadth of like
10:26
functionality in general. I would say like the maybe the
10:29
short answer is it's, it's a dialogue with your users, right?
10:35
So it you want as few evals as necessary to keep your users
10:40
happy. And the longer the thing works,
10:43
like the more feedback you get from users.
10:45
So it tends to expand. It's like scope creep, right?
10:48
It tends to users ask for more stuff.
10:50
So it expands the functionality. And as you get more functions,
10:54
you, you know, you add your evals.
10:56
But another way to think about it, in my opinion, sorry, long
10:59
answer for a short question, but like, OK, let's let's say that
11:03
you're, let's say that you're comfortable with automating
11:07
testing for regular software. Let's let's park AI for the
11:10
moment. And somebody were to ask you
11:12
like, how many tests are, how many tests do you need for to,
11:16
to like in your test harness? You probably say, well, I have
11:20
some intuition because I have some idea about how complicated
11:23
the code is, how many different code paths it needs to go
11:26
through. But I'm also not so like literal
11:30
and and I'm not such a novice that I have to get 100%
11:36
coverage. Like it's not about checking the
11:38
box. There are some paths that are
11:40
more important than others. And so as a good software
11:43
engineer, I'm going to make the trade off.
11:45
I'm going to try to figure out the, I have an idea of what's
11:48
most likely to break. What's the critical path that is
11:52
like right on the edge of working.
11:54
It's not so easy with something. If I mess up a little bit, it's
11:57
going to break. And I have like, they're kind of
11:59
like veins going through the body.
12:01
Like I, I can trace them because I, I built the thing.
12:04
And if I, if I can only write like 34510 tests, I know where,
12:09
which one, which are the critical cases that I need to
12:11
cover. I think it's the same kind of
12:13
intuition when you're building evals for for an AI system.
12:18
Until you give it to the users and they teach you.
12:20
Until that's right and they realize I go, no, like, what if
12:24
I what if I ask it to, you know, play chopsticks?
12:28
So yeah, well, I guess another way of phrasing the question is
12:33
can you ever have too many evals?
12:38
I think you can. So first of all, I think that
12:40
evals can be redundant in the same way that automated tests of
12:44
non EI systems can be redundant. You can have this false sense of
12:49
security if you've got, you know, 1000 tests, but they're
12:53
all like really testing the same like 10 paths to your code.
12:56
First of all, you're spending a lot of extra time that you don't
12:58
need to to get through a test harness.
13:01
And just because you have that high count should not make you
13:04
extra confident because you're not actually testing breath.
13:07
So in the same way you can have redundant evals that are not
13:11
actually going deep enough and testing the edge cases that your
13:14
users care about. So that's one way of of too
13:18
many. Another way of too many is you
13:22
think of emails are kind of more interesting than regular tests
13:26
in my opinion, because they, you know, in the same way that you
13:29
you use rag because you can't jam everything into the prompt,
13:33
right? If we can jam everything into
13:35
the prompt, we wouldn't need RAG, but it's expensive and
13:37
needle in a stag and all that stuff.
13:39
So we use Rag to kind of expand the memory or the mind of the
13:43
model in the same way I think evals are expanding the
13:47
knowledge base or the area of responsibility of the model.
13:52
And just like with any kind of learning process, you want to
13:56
have a healthy intake that takes in like new cases and you also
14:01
want to be able to forget things that are like you need a clean
14:05
up. So like, maybe some evals are no
14:07
longer relevant. The facts have changed, the law
14:10
has changed, your users have changed, the applications
14:13
change, right? It's a new year, crazy things
14:15
are happening. Some things that we believed are
14:18
no longer true. We need to update our view of
14:20
the world. So to the extent that evals are
14:24
a shorthand for your view of the world, you need to have like,
14:28
just a, like a healthy digestion, digestion system,
14:31
stuff comes in, stuff comes out. Is that just every quarter or
14:38
year or half year you're going through the evals and you're
14:42
checking them? So you definitely need a cleanup
14:46
stage and the frequency of that stage, you know, it depends on
14:50
how fast moving your industry is, right?
14:52
So if you're doing constitutional law, right, it's
14:57
probably going to be really slow moving.
14:59
If you're doing, I don't know, like news, then you know, yeah,
15:03
you're going to, you're going to need to clean it up a little, a
15:06
little faster. And you don't want to evaluate
15:10
current events based on stuff that happened like 10 years ago
15:14
unnecessarily. So yeah, it's basically like a,
15:19
it's a like an update speed. And it depends on how volatile
15:22
and fresh your knowledge needs to be in my opinion.
15:25
I like that visual too of the trash collector, the garbage
15:28
collector you come through and you also need to remember to
15:32
have the trash collector for your evals.
15:35
Otherwise you could potentially be spending money where you
15:38
don't need to. Here's another way to think
15:40
about it. Think about if you're, you know,
15:42
if you're planning out a product you're writing out, you know,
15:45
let's say you, you want to communicate with a team that's
15:47
going to build this product for you.
15:49
You want to write a requirements document.
15:52
Now you could think of, there's many ways to go wrong when
15:57
you're writing a requirements document.
15:58
You could go to detailed like if you write a requirements
16:01
document that is super, super detailed.
16:03
First of all, it's kind of obnoxious for the engineers that
16:06
are working on it because you're specifying things that are
16:08
really like probably they know better than you.
16:11
They could probably, if you empower them, they could
16:14
probably make better decisions than if you just like spell out
16:18
like old and how the coach be organized and stuff like that.
16:21
You can also go like not the but most of the problem.
16:24
That's usually not the problem. Usually the problem is it's not
16:27
detailed enough. And then there's a lot of
16:28
misunderstanding. An engineer thought that you
16:31
meant something, you meant something else, used shorthand,
16:34
they misinterpreted it and they build something that doesn't
16:37
work the way the customer wants. Like that kind of error, the non
16:40
detailed enough error in my in my career has been like a lot
16:44
more frequent than the too detailed error.
16:46
And I think that when you're communicating with AI through
16:48
evals, the overwhelming risk is not enough.
16:52
Like I think it is possible to have too many, but you know, if
16:56
we look at 8020, you know more, more than that, like 99.1 are
17:01
not enough. 99% of the teams that I know don't have enough
17:05
evals. Let's talk about handcrafted
17:09
evals versus the evals that you just ask in LLM to generate for
17:16
you. How do you view those and the
17:18
difference is there? So first of all, like it's a
17:20
matter of cost, right? So it's.
17:22
So let's, let's put that in the context of what stage you're in,
17:26
in the project. So maybe, and we talked about
17:31
some of those stages earlier, right?
17:32
In the beginning, you just have the one happy path, you've got a
17:34
demo and then you've got maybe like a little plant.
17:38
It's, I don't know, it's like a couple of weeks old, got a
17:41
couple of leaves and then you got this whole like tree and it
17:44
gets bigger and bigger, right? So in the beginning, when it
17:47
you're just working on the happy path, you probably don't need a
17:51
ton of evals and it's probably fine.
17:53
You know, when you're starting your evals, it's probably fine
17:55
to have them synthetically generated.
17:57
You want, you just want to move quickly, like you don't want to
18:00
wait for other people. And it's not so, so critical
18:03
that each one of them be letter perfect.
18:05
You're just trying to grow the surface area quickly so that you
18:10
get feedback faster so that you can adjust faster.
18:13
It's like a learning exercise. So the faster the better.
18:17
Over time, as you get exposed to more users, synthetic in my
18:22
opinion loses some value because you have it.
18:26
It becomes more important to cover the edge cases and there's
18:29
more of them, and you have access to user data.
18:33
And when you have access to user data, it's usually better than
18:36
synthetic. Now you could still get some
18:38
help in my opinion, like just processing the user data into
18:42
something useful. It's just a slightly different,
18:45
I don't know what the right label for is.
18:47
You're not asking a model to generate from scratch, which is
18:50
what I think of as synthetic evals, But you're, you could be
18:53
asking the model to say, OK, help me scan through this like
18:57
massive ocean of stuff that the model said to people.
19:01
And I want to, I want to boil it down into the ingredients, like,
19:07
you know, take the soup and I want to get the ingredients out
19:09
of the soup. I just want like, give me the
19:11
carrot, give me the celery, give me the Brussels sprout.
19:14
And I'm meaning that I want like a few examples that are
19:18
different from each other. They're all orthogonal and
19:20
together they collectively will give me the same flavor as the
19:24
soup. And then you end up with so AI
19:27
can actually help you with that can help you boil down this like
19:30
massive log file into themes, groups and and then like kind of
19:36
reduce them so that you have like a a few examples to work
19:40
with, but it's different than generating it synthetically.
19:43
Yeah. And and probably also, I've
19:46
never tried this, but it makes me think about it as you're
19:49
saying it, recognizing where we have less evals, where we're a
19:56
little skimp on the evals. And can you just help us augment
20:01
that from because we have a few users that have gone down this
20:04
path, but we don't have a very robust eval set in that regard.
20:09
So let's augment it with a few examples from the LLM.
20:14
Yeah, absolutely. So AI can be like a sparring
20:17
partner when it comes to growing your evals.
20:19
Exactly like you said, maybe maybe you noticed a couple of
20:23
failures in a particular area that your eval is supposed to
20:27
do, but there's only been a few user inputs.
20:29
But you're anxious about it because you know how The thing
20:33
is built and you feel like there's going to be more and I'm
20:37
just not ready. And so AI could be a really nice
20:40
safe a sparring partner that could just elaborate on those
20:44
examples, come up with Harvard ones.
20:47
And it's set up, it's, it can set up this sort of dialectic,
20:50
right answer, you know, help me ask some questions that are
20:54
going to be hard for me to answer.
20:57
Well, it it brings up this topic that we were also talking about
21:01
before we hit record, which was how can you loop in the subject
21:05
matter experts, especially the non-technical stakeholders to
21:11
craft better evals to create a much more robust system.
21:16
Totally. So let me just maybe set up,
21:20
this is my understanding of the status quo.
21:22
This is how I think things work today.
21:25
Most teams don't test in my, in my experience, they just don't
21:28
test. So like if they even just even
21:31
tune into this conversation, listen to like 2 minutes of it.
21:34
Thank you. Just thank you for that.
21:36
Appreciate it. Hopefully they get a ton of
21:38
value from just just set up the first eval with 30 examples.
21:42
Amazing. That would be a win.
21:43
But like the teams that do test, they get into these ruts where
21:49
on the one hand, you know, maybe they generate the data
21:53
synthetically. Usually the people that know the
21:56
synthetic tools are the engineers.
21:57
They don't have the domain expertise.
21:59
And so if they show the synthetic data to the domain
22:01
experts, the domain expert looks at that and it looks like a
22:05
fifth grader did that. Like they don't to them, it
22:08
looks like you don't even know what you're talking.
22:10
I can't understand what this is like.
22:12
You're asking me questions about the, the footer and like what?
22:16
And so, so it, it makes engagement hard for the domain
22:21
experts. On the other hand, you know, if
22:24
an engineering team asks, the domain experts say, hey, can you
22:26
help us with an eval? Of course, she's like, what the
22:28
heck's an eval? Like that is not the word that
22:31
they use. So that hurts engagement.
22:34
If you are able to have an hour with them and you show them what
22:36
an eval is and you show them the spreadsheet, then it's a blank
22:39
page problem because like nobody wants to start with a blank
22:42
page. The columns are a little
22:44
unclear. It's very difficult to get
22:46
domain experts to start like ready writing these evals.
22:51
So the way that the way that we think about it is it's more
22:55
about creating a learning loop, a feedback loop.
23:00
And the feedback loop should be that you have an LM judge that
23:06
evaluates what the application does.
23:10
That's that's the typical LM judge.
23:12
But then you also have a human being that potentially evaluates
23:16
what the application does. And that human being can either
23:18
be the user or it could be the domain expert.
23:22
It's up to them like how much they engage.
23:25
But it's really, really important to have the LM judge
23:28
scores alongside the human scores.
23:31
So you have, let's say the, you know, the user asked X, the
23:36
model said Y, and then, and then the user said, you know, Y is
23:39
not correct. I actually want Y prime.
23:42
And the domain expert said Y is not correct.
23:45
I actually want Y double prime. So it's really important to have
23:49
Y prime and Y double prime like on the same screen, so you can
23:52
compare what the model said, what the user wanted, what the
23:56
domain expert wanted, and they can review each other.
24:00
That's what makes, in my opinion, the feedback loop
24:03
really powerful. What does it mean to review each
24:05
other? So the domain expert sees what
24:08
the LLM judge said and gives feedback to the LLM judge to
24:13
make the LLM judge better. So, you know, hey LM judge, you
24:17
keep focusing on the footer, but really you should be focusing on
24:20
what's between the header and the footer because that's like
24:23
the part of the document that we lawyers care about.
24:25
Or, you know, hey LM judge, you keep picking on my spelling, but
24:29
just ignore my spelling and focus on the math, whatever it
24:33
is, whatever is the right thing for your application.
24:36
And so the feedback from the domain expert to the LM judge
24:40
can make the LM judge a better stand in for the domain expert
24:45
the next time around. And that kind of standing
24:47
quality, there's a statistical definition for that.
24:50
So you basically can, you can measure the correlation between
24:53
LLM judge decisions and domain expert decisions.
24:57
And the higher that correlation goes, the happier everybody,
25:00
everybody gets who domain expert trusts the LM judge more.
25:03
The LM judge is a lot more scalable and good things happen
25:07
at the same time. Domain experts need the LM judge
25:10
because sometimes the LM judge can notice things that just as
25:15
human beings would just don't have like the bam, because like
25:18
maybe there is an error in the footer.
25:20
Like do you actually look at the footer?
25:22
Like maybe it's got the wrong ear in there?
25:24
I don't know. Like you have all these websites
25:26
that say copyright like 2022. It looks terrible because nobody
25:29
looks at the footer. So my point is like, sometimes
25:32
the LM judge can notice things that as human beings, like, Oh
25:35
my God, there's just like, I just don't have, I don't have
25:38
the, there's not enough years until the death of the universe
25:42
when we look at that. So, so the LM judge actually
25:48
makes the domain expert better at their job too, because
25:50
they're noticing that. And they say like, you know
25:52
what, yeah, you should keep that.
25:54
So that feedback loop makes the LM judge better and makes the
25:59
domain expert better. And the fact that they review
26:01
each other improves the chance that your domain expert will
26:05
actually engage because they're getting feedback now instead of
26:08
a blank page, they they give a thumbs up, thumbs down, they
26:11
explain why. And the net effect, in my
26:14
opinion, is that this is really like, it's a way of sucking in
26:20
domain expertise into your application.
26:23
And there's not a lot of other ways for it to get there.
26:26
If you think about it like all of these, like the AI labs are
26:30
solving for, for general intelligence, right?
26:33
They want to make the best model.
26:35
So they use general benchmark benchmarks like, you know, MMLU
26:39
and theoretical math and physics.
26:41
And you know, when we're building stuff for customers
26:44
like who care less about how well it does and theoretical
26:47
does, it just doesn't matter. So you need to know like, I
26:50
don't know the last out that you built, you know, you need to
26:54
know how it's, you know, whether it's extracting the right bits
26:58
from an earnings release or whether it is being rude to a
27:01
customer or not, or whether it's using the up to
27:16
So when you, when you're building one of those, the value
27:22
that you build over time is going to be proportional to how
27:26
much domain expertise you can bring into your, into your bot.
27:30
And the domain expertise is going to come from your domain
27:32
experts, your application, your knowledge, your inside, inside
27:37
out view. And it's going to be different
27:39
from mine, even if we're in the same niche.
27:42
Like we might both be writing financial parsers, but you've
27:46
got a different view on, you've got a contrarian finance view
27:50
and that, you know, that's why you have alpha.
27:53
So when I'm building mine, like I can't just copy yours.
27:57
First of all, I don't know yours.
27:58
It's private to you. Secondly, like my customers are
28:03
looking for something different. Like there's a, there's a reason
28:05
why your company exists and why my company exists.
28:08
And we're, we have like different business philosophies.
28:10
That's why we're different companies.
28:12
And so it's a beautiful thing. So this spurs a whole load of
28:18
questions. I want to start with, yeah.
28:19
Are you just when the expert is having that exchange with the
28:25
LLM judge, are you updating the prompt every time?
28:30
How are you making sure that the judge then is able to solidify
28:34
that and do it every time? Yeah, there's a couple different
28:38
update mechanisms. Let's let me go through the the
28:42
easiest one, the easiest 1 is just with fine tuning.
28:45
It's really it's like that simple.
28:47
So every time that and domain expert reviews what an LM judge
28:53
said and says, actually you're wrong about this part, it should
28:57
be that part, you can construct a better LM judge response based
29:02
on taking the domain expert view and the original and combining
29:06
them into like an improved like what what I wish the LM judge
29:11
response would have been, now that I know what the domain
29:13
expert thought about it. And so that becomes a question
29:17
an input output pair. And if you've collected, let's
29:21
say even like 50 to 100 of those and put up with pairs, you can
29:24
fine tune the model that you use for the Ellen Judge and it gets
29:29
better, like it gets better and better and better.
29:31
That's the simplest update mechanism.
29:34
You can of course also update the prompt of the Ellen Judge.
29:38
I think that it's, it's a little bit quicker maybe than fine
29:41
tuning, but it can get bristly, it can get, it can get brittle
29:46
over time because, you know, the bigger the prompt gets, the
29:49
slower the judge gets. And eventually you end up with
29:51
like this sea of edge cases and nobody, everybody gets scared to
29:55
update that prompt. So, you know, I think it's
29:57
really good for the early stages when you just have a few use
30:01
cases, but over time you probably want to switch to fine
30:05
tuning. Fascinating.
30:07
Now the other question that came up right away was when you have
30:13
different subject matter experts, how do you normalize
30:18
all of the feedback? Oh, that's amazing.
30:20
That's I'm so glad about I had this problem.
30:22
So I was I was head of AI for this company called FactSet.
30:26
And let me just give you this example.
30:28
So FactSet is it sells financial data like Bloomberg.
30:32
You've heard of Bloomberg. Imagine you're a financial
30:35
analyst, you're building Excel spreadsheet and it's a financial
30:38
model of a company. So you're, you want, you don't
30:40
want to hard code the sales and the income, you just want to
30:43
pull in like the last five quarters.
30:46
And this way you could just hit refresh and it pulls that.
30:48
So, so there's like an Excel formula you'll put into some
30:51
cell to pull in the revenue and a different one to pull in the
30:54
EPS and a different one to put in the expenses.
30:57
And after building this Excel language for 40 years, you end
31:02
up with half a million formulas, half a million formulas.
31:05
And like absolutely nobody knows them, nobody can even remember
31:11
them. Nobody can remember what they're
31:12
called. So OK, so then users start
31:15
calling and say, like, what's the formula for EPS?
31:18
The manual is giant and out of date and they can't.
31:20
And like, and more and more users call.
31:25
So like 2/3 of the calls are from users just asking like,
31:28
what's the formula for X? It's a super annoying, you know,
31:31
thing for you to answer because it's a it's a really, it's just
31:34
like a look up answer, but there's just a huge volume of
31:37
users asking because you know, nobody.
31:40
The manuals can't keep up with half a million formulas.
31:44
So then it's not so easy to answer because there are nuances
31:48
like if I ask you, what's the formula for cash, I might mean
31:53
cash in the bank, or I might mean cash and marketable
31:57
securities like a short term debt.
32:00
And it's arguable which one. I mean, cash is kind of like a
32:04
Gray area word. Some analysts interpret it one
32:07
way, some interpret the other way.
32:09
So when we wanted to build a copilot for this, we were like
32:13
really, really excited about, about it.
32:16
And the domain experts told us like, you know nothing about
32:19
finance. And so how are you going to
32:20
build a Covilot or like, we got this.
32:23
So, so we built it and we, we very quickly got up to like 70%
32:29
accuracy and we were feeling like Rockstars, Demetrius
32:32
Rockstars. And then but, but the domain
32:36
expert basically said, look, we're not going to let your bot
32:39
talk to actual people because it could mess up.
32:41
Like we cannot have the Russell, we're going to have a human in
32:44
the loop and the humans going to look at what your bot says and
32:46
they're either going to accept it or reject and you'll know.
32:49
And they kept rejecting the cash question.
32:51
Like it kept, you know, we kept getting it wrong.
32:54
And then when we looked at the data, we realized that it was
32:57
like, it was like an edit war in Wikipedia where you have like 2
33:01
people that are not talking to each other, but they're sure
33:05
that they're right about what cash means.
33:07
And they keep overriding each other's edit.
33:09
So no matter how we update the prompt, it's never right.
33:13
And it just keeps like the bit keeps flipping.
33:15
Oh, cash means this. Oh, no, cash means that, oh, you
33:18
know, my, my, my mother, my sister, my mother and sister.
33:21
So it's like, you know, it's like you cannot, This is why I
33:25
like, we cannot get above 70% accuracy.
33:27
And I realized that 70% is the threshold at which people agree
33:32
with each other. And 30% is actually the area
33:36
that's actually the Gray area in the system.
33:39
And this is amazing opportunity because these people, the agents
33:42
that are actually answering these questions for people, they
33:44
didn't know that there was a 30% area of it, that they thought
33:47
that it was just all black and white.
33:50
It's like in Excel formula, there's a right one and there's
33:52
a wrong one. But it turns out that we have a
33:55
unique ability to organized this knowledge base better than it
34:00
was before the copilot started. So to come back to like machine
34:04
learning terms and like the nitty gritty, we did a cluster
34:08
analysis where we embedded the questions and we identified
34:13
clusters of similar questions, semantically similar that had
34:17
opposite answers. And that's how we were able to
34:19
find this like all of these people are asking, you know,
34:23
where's where's cash? What's the formula for cash?
34:26
Tell me cash, I want cash or not that you not.
34:29
Or sometimes they don't use the word cash, but they use a close
34:32
synonym for cash. And all of those questions got
34:34
grouped together purely through semantic clustering through the
34:38
embedding model. And then we basically say, OK,
34:41
the clusters that have consistent answers, I don't care
34:44
about those. Those are good soft check.
34:46
I want the clusters where it's a cluster fuck, right?
34:50
I want like, I want the clusters where it's like all different
34:53
answer. They're all over the, the
34:54
questions seem identical and the answers seem all over the place,
34:57
right? How could that be?
34:58
And then, you know, so we, we, but the beauty part is we could
35:02
identify those automatically. We didn't know the right answer
35:05
automatically, but we could identify disagreements between
35:09
domain experts automatically. And this was hugely value added
35:12
because they didn't have a process for finding those
35:15
before. And then it's just a matter of
35:18
you, like, basically getting the two sides engaged.
35:22
Like, how do you solve the Wikipedia edit war?
35:24
You get the two people in the room and they start to talk to
35:27
each other and like, hopefully one convinces the other.
35:29
And then, you know, then the edit war settled.
35:32
So we kind of did that. There's like a little bit of
35:34
plumbing. There's a little bit of
35:35
organizational design. There's a little bit of, you
35:38
know, ML programming and the knowledge base got much, much
35:41
better. And we were able to push that
35:43
Gray area from 30 to 25 to 20 to 15 to 5:00.
35:48
And that was not just great for the for the humans that we're
35:52
dealing with all these user questions because now, you know,
35:55
now they can have consistent answers, but it was great for
35:58
the copilot because now the copilot clear up that knowledge
36:02
base as well. But there are going to be
36:05
scenarios where you do not have a black and white answer, right?
36:09
And so are those I've heard folks say steer clear of those
36:12
with your AI. It really depends.
36:16
I look, I think AI is not going to be any more black and white
36:19
than people are, right? So like, we can't, we can't
36:23
expect AI to give us black and white answers to questions that
36:26
are just like part of the human condition And like there's no
36:30
right answer. Like we can't, just because
36:31
there's a new technology doesn't mean that suddenly there's a
36:34
right answer where there wasn't before.
36:36
But we can generate a boatload of value if we, if we go to the
36:44
areas of human knowledge that are where like most people agree
36:49
and there's value to giving the answer that most people agree
36:52
on. And that value, like if it's
36:54
very expensive to get to that answer today, because you need
36:57
very experienced, you know, expensive people to do it.
37:00
And you're able to get the 8020 with, you know, like 80% of the,
37:06
of the accuracy value area of agreement with like 20% of the
37:10
cost, 10% of the cost. That's a, that's a huge value.
37:13
We don't have to all agree on what poetry we like.
37:15
We could just like focus on, you know, whatever, extracting data
37:21
from financials, you know, so, so like all the, all the use
37:26
cases that we know of, you know, for, for echo pilots.
37:29
And I think like it's you, you, you would think like how, how
37:35
does a company compete in this kind of in this kind of area?
37:38
Like how does an engineer compete?
37:39
But also how does an organization compete?
37:41
So there's going to be, there's usually some secret sauce to
37:44
every business. And right, you know, in the
37:47
beginning, that secret sauce is in the head of the founder, in
37:50
the head of the partners, the senior, let's say 5 to 3% of the
37:54
people. And if there's a way to scale
37:56
that, basically to apply that, not to publish that secret
37:59
sauce, but to apply it at scale so that the bot applies the same
38:05
level of nuance and the same level of special sauce that the
38:09
senior partner does. That's a huge value unlock for
38:13
the organization, right? Like in my example, you know,
38:16
there's probably like there were a couple of people that had a
38:19
really good nuanced understanding of cash.
38:22
It's just that they that that value that was in their heads
38:26
was not communicated to all the callers.
38:28
So you basically hit the lottery.
38:30
Like if you get somebody who thought it was one thing or the
38:33
other, you wouldn't even know. Like you wouldn't know that
38:35
you're getting part of the value.
38:36
But as a company, we all wanted to figure out what is the best
38:41
answer and give it to everybody, even if the best answer includes
38:43
some options, right? And this kind of process is a
38:47
way not just to make the AI more accurate, but to help domain
38:51
experts, like kind of settle that value pyramid.
38:56
Then when you talk about the nuances in someone's head and
39:00
then being able to translate that to reality, for lack of a
39:06
better term, the thing that instantly came to my mind was
39:09
wow, I could do that with folks that I get documents from.
39:16
And sometimes one thing that I do not like doing is getting on
39:22
calls with folks. If it could have been an e-mail
39:25
or if it could have been in document, I would prefer for it
39:28
to be a document 1st and then I'm happy to get on a call
39:31
afterwards once we have established what it is that we
39:35
are talking about, right, and what the problems are, etcetera,
39:39
etcetera. What I find though is sometimes
39:43
you can be working with someone and the document is amazing and
39:48
it's like, wow, this is incredible.
39:51
There's so much here. It's very clear, it's really
39:55
well put together, it flows and you can just comment on it and
39:59
whatever you can't figure out in the comments, that's when you
40:02
get on the call. Other times I'm like, holy crap,
40:06
this document is a mess. And it's not a mess because I
40:09
make a lot of documents that are just brain dumps and those are a
40:13
mess. But it's a mess because it is so
40:18
verbose or it is so AI generated that it hurts my eyes to even
40:22
read it. And then there it's not clear
40:24
what are we? What is this document for?
40:26
What are we even doing here? Where's the meat and potatoes of
40:30
this whole document? And so I was thinking about, I
40:34
wonder how I can start getting the ways that I enjoy, like
40:41
documents that I really appreciate using those as my
40:45
eval set and then saying, all right, anytime anyone brings me
40:48
a document, just as run it through an LLM and say here, I
40:55
want to make sure how can I make this better so that it aligns
40:58
with these documents or so that it's more in Demetrius style?
41:03
100%, You know, I mean, Demetrius, you and Sofia built
41:06
an amazing community. There is a special sauce to
41:09
building a community. Like I don't know how to do it.
41:12
So right now, in order to get that special sauce out there,
41:18
you've got to put your own personal touch on it.
41:21
And I can tell your style because I read some of your
41:25
LinkedIn posts and I've been to some of your conferences.
41:27
Like, it's distinctive, but you and your community would benefit
41:33
a lot if in addition to when you're able to be there in
41:39
person, you could figure out a way to scale your style to like
41:43
all the touch points that are just like you.
41:45
You only have 24 hours a day. You got to sleep like I do.
41:48
So if there was a way to scale your style, your community would
41:52
grow that much faster, get that much more value.
41:54
It doesn't. It's not about replacing you.
41:56
It's about like just taking the the thing that's unique, the
42:00
thing that made you guys grow and making it accessible to
42:02
people, even if like whatever, you know, even if they can't be
42:07
at the show. Yeah.
42:09
There's something that I wanted to mention before we move on to
42:12
around the whole subject matter expert piece because you had
42:17
said this before and it's worth repeating how traditionally when
42:23
engineers are building SAS products, it's OK if they are
42:28
not the subject matter expert and they don't bring in the
42:31
subject matter expert. Because at the end of the day,
42:35
the users will suffer through it.
42:37
And if it kind of helps and it kind of gets the job done,
42:40
that's great. It's useful.
42:42
However, now if you're trying to create a copilot and you're
42:47
doing it without the subject matter expert, you're probably
42:51
not going to get very far because it's not going to be
42:54
that valuable. So the crucial piece here is
42:59
again, like hammering on this, how you can bring in that
43:01
subject matter expert and make it easy for them so they don't
43:05
have the blank page problem where they don't have to learn
43:07
how to code and they're not firing off Python scripts or, or
43:11
having to really update Proms to make it better.
43:15
So, so that is something that I, I really wanted to hit on
43:19
because I, it resonates with me a ton.
43:22
Yeah, thanks. Thanks, Dimitris.
43:24
I think it's yeah. Like if you imagine, you know,
43:27
you're building, I don't know, building SAP or you're building
43:30
Salesforce.com. Maybe it's an inventory
43:32
management system or ERP or whatever it is.
43:34
It's basically a couple forms and all you really have to do is
43:39
the engineer. You have to make sure that the
43:41
data goes where it needs to go. Like it's so it's pretty black
43:44
and white. You just have to like not lose
43:46
the data and the form has to render, you know, there's, it's
43:50
pretty clear, like when you've messed up, it's pretty clear.
43:53
And you will on purpose avoid any kind of situations where I
43:59
don't know, you need to render you, you need domain expertise.
44:04
You'll avoid those because you're going to try to stay to
44:06
the, you know, the objective piece.
44:09
And in the AI world, it's kind of flipped on its head because
44:13
the value is in delivering domain expertise more broadly,
44:17
cheaply, scalably. So if that's the goal, like if
44:21
you're trying to, and whether that domain expertise is
44:24
finance, legal, architecture, whatever it is like AI is
44:27
reaching into all these places. That's really the new part.
44:30
The new part is not being able to draw a GUI from scratch.
44:34
Like, yeah, it's kind of exciting when you talk to AII
44:37
could draw a GUI from scratch, but we know what it's like to
44:40
draw GUI's. Like we've seen that before.
44:41
The exciting part, the economic value, the new part, the unlock
44:45
is you know, you deliver legal expertise that used to cost 500
44:51
bucks an hour. You deliver it for 5 bucks an
44:53
hour or architectural expertise or medical or whatever.
44:57
It is like stuff that people really, really need and used to
45:01
only be accessible to the few and now it's way cheaper
45:04
accessible to the many. That's the unlock.
45:07
And in order to get that unlock, you need a different skill set.
45:11
Like that's, that's our challenge.
45:12
We're so new. And it's the same challenge with
45:14
every new technology. Like we, we got here because
45:17
we're into the AI, like we're both you and me, we're into the
45:20
gears. Like I love to see those gears
45:23
going. I just want to like jam my hand
45:25
in there and like fix it. And you know, I love playing
45:27
with it, like getting my hands dirty.
45:29
But the value from those gears is the main expertise that we
45:34
don't really have. Like, so in a sense, the most
45:39
successful features, the most successful products are going to
45:42
be the ones where we can make the gears invisible.
45:45
Everything that we're doing will be invisible.
45:48
And the user will just have the experience of talking to an
45:52
experienced lawyer or, or you know, or an experienced engineer
45:57
or an experienced podcaster, right?
45:59
Or, or community builder Like we're, we're, that's the magic.
46:03
And in order to get that magic, we have to get these domain
46:07
experts like right in there with us.
46:11
And my suggestion, my humble suggestion is that evals are the
46:17
most friendly medium to get domain experts into your
46:22
workflow. They're much more friendly than
46:23
asking them to write stuff from scratch to edit, to edit
46:28
prompts. They because they're they're
46:30
very flexible, they can grow to your needs.
46:33
It's so funny you say that because the last podcast that I
46:38
did an hour ago, before we were on here, the guy was talking
46:42
about how in his product he put, he went and he had this idea of
46:48
some fancy AI that he was going to add to his product.
46:51
And after talking to all of his current user base, he realized,
46:55
wow, people just want like to save some money on their law
46:59
fees or their legal fees. So maybe we should try and
47:02
figure out if we can use AI for that.
47:05
And they ended up doing it and implementing it and it's been a
47:08
huge success. And so you, I really like that,
47:12
like taking something that you're used to paying a whole
47:15
lot of money for and then can you, maybe it's not 100%, but it
47:21
is at least trying to figure out a way to get expertise for much
47:26
cheaper and figuring out how you can fit that into your product
47:32
with the help of the subject matter experts.
47:34
So now I want to jump on the topic of the LM as a judge
47:40
because we kind of danced around it with the subject matter
47:44
experts and how they can help the judge.
47:46
What I am fascinated by is how many different ways you can do
47:53
this LLM as a judge. And I've seen some papers that
47:56
come out and say, no, just one LLM isn't enough, you need all
48:00
12 of them. So it's like a jury or you need
48:04
to be doing it where I I thought it was hilarious.
48:07
It's like some researchers are hanging around in the dorm room
48:11
and having their pizza and beers and saying, well, what if we did
48:15
not one LLM judge call? It was two LLM judge calls.
48:21
And that's a problem. He else is like, what if we did
48:23
three of them? And so you just keep more LLM
48:27
judge calls on top of it. And you see, is there a place
48:31
where that will kind of pop out? So I know that you you've been
48:37
doing a lot and give me the download on it.
48:40
Yeah, that's an ICLR paper for sure, right.
48:42
Just N + 1 LLM judges like you know, get get me the seat.
48:47
OK, so before we talk about all the different ways of doing it,
48:53
the how I propose a metric for deciding and the metric I
49:00
propose is the human agreement rate.
49:03
Because like, OK, we're talking about, we're swapping tips.
49:07
Like we, you've read a paper and we have like, and there's lots
49:11
of ways to do it and we're, we're, we're swamping
49:13
techniques. And any time that you're
49:15
debating like is technique A better or technique B better?
49:19
Like should I have two or three or five?
49:21
What's when is it going to any questions that you have about
49:24
which one is better as an LM judge?
49:27
Like there has to be a way to answer those questions.
49:28
That's got to be like more than whichever one of us is like has
49:32
the bigger audience or you, you like yelling, yelling more or,
49:35
you know, as higher paid or whatever.
49:37
They just should be a metric. So the metric I propose is
49:41
whichever LLM judge approach reaches the highest human
49:45
agreement rate. And that's just like not a human
49:47
off the street, but like the humans of the matter for you,
49:50
maybe it's your users, maybe it's your domain experts, We can
49:52
define them in advance. And we're basically trying to
49:55
optimize for that. And you know, we're ML people,
49:59
we optimize, we understand what optimize means.
50:02
So OK, so we start with the same approach, right?
50:06
We start with like a simple LM judge, probably pick a cheap
50:09
model, pick a simple prompt, and we measure the baseline human
50:14
agreement rate. So in we have our eval set, we
50:18
evaluate it with people, we evaluate it with the LM, we
50:20
measure the agreement and it's whatever, it's 70%.
50:24
And then we could try things we could try like the jury could be
50:27
good because different models of different strengths.
50:29
So for like for use cases where let's say some of the questions
50:33
are heavy math and some of the questions are heavy creative
50:36
writing, well, you know, maybe Claude is better at creative
50:39
writing and maybe, I don't know, DeepSeek is better at math.
50:43
And so if we, if we jury them together, maybe we can have
50:46
higher human agreement rate, or maybe they'll just like fight
50:49
with each other and we'll have less consistency and the human
50:52
agreement rate will be less. My point is the empirical
50:55
approach over the theoretical and like, try it with your data
50:59
and with your eval instead of like taking my word for your
51:02
word for the OR the paper. Like the guy who wrote the paper
51:05
does not know what's going to give the highest human agreement
51:08
rate for your task. And if you can unlock eighty
51:12
9095% human agreement rate with your domain experts, you'd be
51:16
amazed how happy they would get because it's a lot of work off
51:21
their plate. They don't have to look at any
51:24
of the like whenever the LLM judge expresses confidence which
51:28
is going to be 8090% of the cases.
51:30
If they have high human agreement, then the domain
51:33
experts can be like, I don't want to look at any of those,
51:35
just give me the exceptions. So immediately, like their work
51:39
drops down a couple of orders of magnitude.
51:41
That's awesome then. So then would you say you have
51:46
to always have those human curated eval sets before?
51:54
Like you can't just raw dog with some LLM as a judge right from
51:58
the. Yeah, you bring up a good point.
52:01
There's a, there's a bit of a chicken and egg, right?
52:03
Because like, like we said earlier, if you just, if you
52:06
don't have anything and you, you start by coming to your human,
52:11
human domain experts and you say like, here's a blank spreadsheet
52:15
and show me all the things that teach me law.
52:19
Basically, that's what you're saying.
52:21
And they're going to be like, get the hell out of here.
52:23
Like I, you know, I went to, I, I spent and I, I have clients
52:28
and like, I don't have, I can't teach them.
52:30
Too important for this. Right.
52:31
But so I think like in the beginning, yes, you should
52:35
probably like you come up with the E value yourself and you are
52:39
the human that the LM judge should be agreeing with.
52:42
Because if the LM judge does not agree with you on the first,
52:46
let's say 20-30 examples in your, in your LM set, that's
52:51
really, really easy to fix. It's a very tight loop.
52:53
It's like you yourself and me, myself and I and I, I test a
52:57
couple things. If the LM is doing things that I
52:59
totally don't expect, I can just assert like, OK, maybe I don't
53:03
need to be the world's greatest domain expert to deal with these
53:07
like really simple few cases. I'm going to start there.
53:11
And then once you have it, once the LOL judge behaves in a way
53:14
that you trust on a small group of cases, which should not take
53:19
long, we're talking about like days of work, maybe not, maybe
53:23
not hours, maybe days of work less than a week.
53:25
So if you have like some agreement on a very narrow base
53:29
of knowledge, you could say, hey, domain expert, I basically
53:34
got it as far as I can get it. Like I, I'm not a law expert,
53:38
but it doesn't say totally crazy things that, that I don't
53:42
understand. So I trust it and I could show
53:45
you some things that it says. If you trust it, we could just
53:48
let it go. If you want to get in the loop,
53:50
you can. And I think they will want to
53:54
like if it, if you've cleared the bar that you can clear
53:57
yourself, they're much more likely to engage at the next
54:01
level in my opinion. I like how you bring up this
54:06
idea of we need a baseline of what we're comparing it to.
54:11
So if we're using an LLM as a judge, what are we comparing it
54:14
to? Let's start with something from
54:16
me and then we'll bring in the subject matter expert when it's
54:20
needed. But we'll get the basic stuff
54:23
out of the way so that I'm not wasting some famous or some
54:27
highly. Yeah.
54:28
And there's a couple more hacks there, like you're trying to
54:30
sort of accelerate learning, right?
54:32
So imagine imagine that you're really like trying to teach
54:35
somebody long without taking them through law school.
54:37
So OK, you could start with just common sense.
54:40
There's also probably, you know, there's probably some data sets
54:44
out there that are available where you're just teaching them
54:47
like basics. And then there might be a couple
54:50
of examples where, you know, maybe your law firm has a
54:53
different view on things. Like maybe your accounting firm
54:56
has an idea that it's OK to take a chance in this particular and
55:01
be aggressive in this like 1 niche because they have a
55:03
variant view. And because you know that you
55:06
work in the firm, you could just like build it in as a little as
55:10
an, as an example, that knowledge is plastic and it can
55:12
fit to what the firm thinks and not just what the standard
55:17
benchmark thinks. And you could just build one or
55:19
two of those to show your domain expert that what can be
55:23
accomplished and they will put in more.
55:26
When you're grading the LM judge responses, I imagine there's
55:33
different vectors that you can grade it on.
55:35
How do you look at that? Right, we call that criteria in
55:39
our product and there are hundreds and just like every
55:42
organization has a secret sauce, every evaluation will have
55:47
different criteria that what it means, what success means.
55:51
So, so I think it's a really juicy challenging area for a lot
55:58
of people. I think there's a lot of value
56:00
to first of all, having a library of criteria that's
56:03
available off the shelf. And I don't think that it's
56:06
enough to have like 5 or 10. Like I think you need a couple
56:09
100 to cover all the different use cases.
56:11
There's really 'cause they could be very, very different.
56:14
And then probably beyond the standard library, there's, you
56:19
know, you will want to create some that are specially for you,
56:23
like special for your task. So not just creating the the
56:27
eval, let's say the input and the output, but also creating
56:30
the criteria, the prompt for the criteria and the definitions for
56:34
the criteria are going to evolve as your system evolves.
56:37
There's a lot of value to breaking down.
56:39
Usually people will start with really broad criteria like
56:43
accuracy. Accuracy is kind of a catch all,
56:45
but it means different things to different people.
56:48
And the way I've seen it evolve, maybe maybe you've seen
56:51
something different, but the way I've seen it evolve is, you
56:53
know, typically an engineer or engineering team will just
56:57
define accuracy and they'll use like a non domain specific way
57:02
to define accuracy and that the main experts will shit all over
57:06
it be like, well, this got a five in accuracy, but it's
57:10
totally missing that this guy's going to get sued or it's got a
57:13
5 and you know, it's got a one in accuracy.
57:15
But there's all these good things about it.
57:17
And you should, you should, you should get partial credit for
57:20
that. They'll shit all over it.
57:22
And that's great because they're engaged, you have feedback and
57:26
the opportunity there is to break down the one accuracy
57:29
criteria into probably like 3 or 4 that you can observe from the
57:33
pattern of feedback that you get is very valuable activity.
57:36
So like maybe they care about, I don't know, naming things
57:40
correctly. That's one way of accuracy.
57:42
And maybe they care about the length.
57:44
That's a different, totally different axis of accuracy at
57:47
the level of detail. Maybe they care about the
57:49
refusal rate, Maybe they care about the how persistent you
57:54
are, like how many times you've tried.
57:56
So these are like all different dimensions that I find that the
58:00
best way to elicit those is to show people obviously wrong
58:05
things and then they they jump in and correct it.
58:08
And that feedback loop is a really, really good.
58:11
It's a good flywheel to get people to think more about what
58:13
they want. Yeah, it reminds me a lot of a
58:18
talk that Linus Lee gave at one of the conferences that we had
58:22
last year. And he was really banging on
58:26
about how in music you can see sound waves in different ways.
58:33
And you have like if you're a producer, you have an equalizer
58:37
that you can play with in the production or the the DAW and in
58:45
filmmaking or in Lightroom for photographers, you have these
58:52
histograms and they're so rich and you can change the photos
58:57
output by playing around with the histograms.
59:00
And his whole thing was, how can we bring that to AI output?
59:07
Is there a way that we can now start creating more of a visual
59:11
aspect of this output? And it makes me think about what
59:16
you're saying here. There's all these different
59:18
vectors and you have all these different criteria as you call
59:21
them. And so is there a way that we
59:24
can visualize this in like different ways so that it is
59:29
more engaging for folks to look at?
59:32
Yeah, 100%. You know, I think I think of it
59:35
like an evolutionary process. We start with really big blocks
59:39
that are really easy to put in place and we get more refined
59:42
and more sophisticated over time and it kind of never stops.
59:45
So, you know, it starts with accuracy and then it evolves
59:48
into maybe 3 or 4 criteria. It can eventually evolve into a
59:52
checklist. Checklists are super useful in a
59:55
lot of medical scenarios, a lot of high risk scenarios where you
1:00:00
know that you need to basically meet all these different
1:00:02
criteria to be successful. It's hard to remember them all.
1:00:05
It's, you know, it's hard for the best domain experts to
1:00:07
remember the most perfect, perfect job for an LLM judge
1:00:10
because they're tireless. They'll just be like super rigid
1:00:14
and check all the things that you ask them to check, even if
1:00:18
they're, you know, even if they've been doing it for a day.
1:00:21
And humans are not like that. So, so it's a, it's a great way
1:00:25
to scale a rubric, which is a set of criteria and scaling.
1:00:32
It makes things fair and it makes things cheaper and makes
1:00:35
things valuable. And as an organization, you want
1:00:38
to have a feedback mechanism about the rubric, not just about
1:00:42
the eval for, for the same like very same things.
1:00:45
And we're, you know, we're finding that, you know, social
1:00:52
media companies are revising their views about what is
1:00:56
appropriate moderation. So like there used to be a
1:00:59
rubric of how to, you know, how to grade a social media post and
1:01:04
it was implemented one way. And for a long time every post
1:01:08
was graded on this rubric. There were people grading it.
1:01:11
And now we're in a different era and like it's been graded in a
1:01:14
different way. The rubric changes.
1:01:15
Yeah. So my point is that for for
1:01:18
commercial application, LM application, AI application you
1:01:21
want to have that flexibility as well your rubric change.