Latest / Agentic Conversations (formally mlops.community) / Holistic Evaluation of Generative AI Systems // Jineet Doshi // #280

Transcript

Holistic Evaluation of Generative AI Systems // Jineet Doshi // #280 · Agentic Conversations (formally mlops.community)

0:00
So my name is Janet Doshi, I'm a staff AI scientist at Intuit and
0:09
I'm caffeine free so I do not drink.
0:12
I neither like tea or coffee. Try to keep it natural.
0:21
Welcome back to the End Alarms Community podcast.
0:23
I am your host Dimitrios, and today this is the holistic view
0:28
of evaluation. We break down three different
0:32
ways that you can evaluate your Gen AI systems and talk about
0:40
LMS as a judge. Some best practises, some
0:43
challenges, different ways that you can do them.
0:48
Oh. I felt like I've had a lot of
0:52
conversations on evaluation in the past two years.
0:55
This really took all of those conversations that I've had.
0:59
You know, maybe there was a sliver over here, a nice little
1:02
chunk over there and it wrapped him up, put a little bow tie on
1:07
top of it and served it as a present.
1:11
So Christmas came early for y'all.
1:13
Hope that some of the stuff we're saying in this makes you
1:18
have better AI systems. Let's jump into the
1:23
conversation. And as always, if you enjoyed
1:27
it, share it with just one friend.
1:32
And if you are sticking around on Spotify, I've got just the
1:36
song for you today because I want to give your recommender
1:43
system something new. And as always, we're sharing a
1:49
song to cruise into this podcast conversation with.
1:55
This one today is called Sunsets, and it's by Living
2:02
room. Yes, sunsets by living room.
2:10
And we'll drop the link into the description or just go ahead and
2:15
Shazam it right now. So good.
3:04
This conversation came about because I was given a talk two
3:08
years ago now, not this last paid in an AI summit, but the
3:12
one before that around LMS and AIM production.
3:18
And you were at that talk and afterwards, or actually even
3:21
before I went on, we got to chatting and you were saying,
3:26
you know, one thing that is very important that I'm not seeing
3:29
enough people talking about is evaluating.
3:33
But evaluation on a systems level.
3:36
And you had ample experience because of your work at Intuit
3:40
and building out, what is it, Intuit Assist that's powering
3:44
over 100 million customers, which is insane, just in
3:47
general, insane scale. And so that planted the seed in
3:51
my head. And here we are two years later,
3:54
finally getting around to having the evaluation conversation.
3:57
So it's great to have you here, man.
3:59
Yeah. Thank you, Dimitris, for having
4:01
me. Yeah, absolutely.
4:04
This is like a very important topic because it kind of goes
4:08
like right to the core of like building trustworthy ML systems,
4:14
which is like extremely important.
4:17
So yeah, happy to talk about this.
4:19
And yeah, I also do want everyone to know like the views
4:25
that I express are my own and yeah, it should not be
4:30
associated with any organisation or entity.
4:33
There we go, Just as a disclaimer and we'll put that
4:36
one front and centre so nobody gets twisted.
4:41
But there is a lot of cool stuff that's happening at into it.
4:45
You guys are powering a lot of models.
4:49
What was it? Hundreds of thousands or 10s of
4:51
thousands? Right.
4:53
So when I joined into it like 6 1/2 years ago, at that time like
5:00
basically we had like single digit models in production to
5:04
now like Fast forward to now where we really like invested a
5:09
lot in AI and we are at the point where we have like
5:14
thousands of models in production making 58 billion
5:18
predictions a day. Like that's the scale at which
5:22
we are at now. 58 billion, that is such a large amount I, my
5:29
mind can't even fathom how many zeros that is.
5:32
And I can assume that they're not all LLMS. You have a mix of
5:37
traditional and Gen AI in there. But what you've been noticing,
5:42
right, that when it comes to the Gen AI type of products that
5:49
you're putting out, you need to really think wisely about
5:54
evaluation and how you are going about recognising if the system
6:01
in general is performing. And so I think when we kicked
6:05
off this conversation way back when, you were saying, yeah, you
6:09
have to look at the model output, but you have to look at
6:12
the model output on many different vectors.
6:15
And you also have to look at the system.
6:19
So retrieval evaluation can be one thing.
6:22
So how are you chunking things and looking at that?
6:25
There's and even looking at the how are you embedding and the
6:30
embedding model? And that is becoming more and
6:34
more mainstream. But I'm wondering if you want to
6:38
give us a lay of the land on why you feel vows are challenging
6:43
and why it still feels like it's an open problem even two years
6:47
later. Yeah, yeah.
6:49
Absolutely, Yeah. So I mean, evaluating generative
6:52
AI systems is it is quite challenging.
6:56
And I think there's like no right or wrong.
7:00
There's it's all like dependent on the use case at the end of
7:04
the day. So like when we look back and
7:07
see like, OK, why is fundamentally evaluating LLMS
7:10
challenging, right. So if we go back to like the
7:14
classical ML space essentially like it feels like a long time
7:19
ago, but it wasn't that long ago.
7:22
So with the classical ML models, essentially what we would do is
7:28
we would train like your usual tree based models or SVMS on a
7:32
very specific task. Like those were all very task
7:36
specific models every like the problem was very well defined
7:40
and the output was like very limited, right?
7:43
Like the model would essentially output the probability of
7:47
classification or like some number if it was a regression
7:51
problem. And the metrics as well were
7:53
pretty well defined. So essentially it was like these
7:58
models were trained on very niche tasks and the outputs were
8:03
very limited, well defined, and the metrics were also very well
8:07
defined. So if it's classification, you
8:09
look at accuracy, precision, recaller, you see like what have
8:13
you. Same with regression.
8:16
It had it's own set of metrics. But then when these LLMS kind of
8:21
came about, it completely flipped the script, right?
8:25
Like everything I felt like went upside down in a way because now
8:30
suddenly these LLMS are producing open-ended outputs.
8:35
Like you can ask it to like write a poem about Pytorch and
8:39
it'll do that. And there's like no one right
8:44
way of doing that. There's like millions of
8:46
possible right answers. And the other pieces, the
8:51
breadth as well, right? Like earlier we had very task
8:54
specific models, but now like one single model is able to do
8:58
so many different tasks, right? It can write a poem, it can
9:02
summarise, it can do Q&A. There's like so many different
9:08
things that these LLMS can do, which again begs the question,
9:12
like how do you measure intelligence like on a broader
9:16
scale? And I feel this is something we
9:20
haven't really solved even for humans.
9:23
Like we have things like IQ tests, but again, those are far
9:26
from perfect. So that's why I feel like it's,
9:30
it's a very essential and challenging topic.
9:36
You know, one thing that I have thought about so much when it
9:40
comes to evals is if there will be a product to solve the
9:49
evaluation problems and if that product is an eval product or if
9:55
it's part of a greater platform that you're doing AI development
10:01
on. And I know that you've, you've
10:06
had some experience already. Do you feel like it's going to
10:09
be a product? I'm not sure about the product
10:16
piece, but it is absolutely essential like to have evals at
10:21
I would say the entire ML life cycle.
10:23
So like right when you choose your data to like when you're
10:28
building or like training or fine tuning the model offline to
10:33
then OK, even once it's in production, we still need evals
10:36
to constantly monitor and make sure like there's no drift and
10:41
everything's like working as expected.
10:43
So I feel like evals at the end of the day are essential across
10:48
the entire like ML life cycle. Again, whether that's classical
10:51
ML or Genii, I think that doesn't matter.
10:55
Yeah. What are some different
10:57
approaches that you have in mind when you are doing evals?
11:01
Have you experimented with ways that have deemed to be more
11:07
successful or less successful? Sure.
11:11
Yeah. So I, I'm guessing you're
11:12
referring specifically to LLMS engineering systems.
11:15
Yeah. So yeah, So in that space again,
11:20
there's I feel there's no right or wrong.
11:22
There's like different techniques that are out there
11:26
and each one comes with it's own trade off at the end of the day.
11:29
So we have to like be aware of the pros and cons of each of
11:34
them and make those decisions based on the use case.
11:39
But in general, I feel like we can think of the different
11:44
evaluation techniques for gener systems into like 3 higher level
11:49
categories. So the first category is I, what
11:53
I would say like some more traditional NLP techniques.
11:58
So again, when LLMS came about and people realised, OK, like if
12:02
everything is a big challenge, like how do we approach this?
12:06
The first thought process was NLP as a field has existed for
12:11
multiple decades, right? Like that's not me.
12:13
So can we like borrow all of the great work that people have done
12:19
already in that domain and see like how, how we can reuse some
12:24
of that. So if we like dive a little
12:27
deeper into that. So one approach is essentially
12:32
that LLMS again usually create have open-ended outputs, but
12:37
essentially can we like tweak our evaluation test set to
12:43
essentially have multiple choice questions and like almost
12:48
artificially force the LLM to pick options and force a close
12:53
like a close ended output versus an open-ended output.
12:56
And then by doing that, like all of the traditional evaluation
13:02
metrics suddenly become applicable.
13:05
Of course, like it's easy to compute and like everything is
13:10
well defined. But of course, the corn here is
13:13
we cannot really evaluate on a lot of the open-ended tasks.
13:17
It's it's more helpful when it's like knowledge based or
13:21
reasoning based tasks that we can evaluate on an extension of
13:26
that was also like in the like the tech similarity space, where
13:32
again, if you do have like long form output from the LLN and
13:37
let's say you have some reference answer, the idea is
13:40
that you can convert both of them into like an embedding
13:44
space or like into like an N gramme space.
13:47
And then like let's like do the comparison in that space.
13:52
So again, there's like pretty well studied metrics like board
13:56
score, bot score, cosine similarity at the distance that
14:01
exist in that space, which again have been used like across the
14:07
board by different teams. So one of the advantages of
14:10
doing that is basically with in the embedding space, your
14:17
answers can be like language agnostic in a way, right?
14:21
Like if your LLM output is in English, let's say, but your
14:25
reference answer is in Spanish, The idea is start in the
14:29
embedding space at least as long as the embeddings are good
14:33
quality, they capture the semantics and the similarity
14:38
scores would would be like higher there.
14:44
But then of course that's also that also leads to a con of that
14:48
approach. Like something to keep in mind
14:50
is it is heavily dependent on the choice of the embedding
14:54
models. So if you're looking at like
14:56
domains, specific evaluations where it's like you have the
15:00
legal language or the medical domain where the language tends
15:05
to have like quirks and nuances of its own, you need to pick an
15:10
embedding model that kind of captures that.
15:14
Because if it doesn't, then. Invite you in the ass.
15:17
Things failure and also some of these metrics there have been
15:21
studies that have been shown that they do not always like
15:25
align with human judgement. So that's something to keep and.
15:29
To underpin this is the traditional NLP type of
15:33
evaluation. So that's what you are.
15:36
That's like the first bucket that you see as one way to do
15:40
it. Yes, absolutely there's some
15:42
more techniques there as well like benchmarks is another
15:47
pretty I would say popular widely used technique.
15:51
Again, there's thousands of benchmarks out there increasing
15:54
like everyday things like you know, MMLU, super glue squad,
15:59
like these are like very well known benchmarks so that.
16:04
Just a quick shout out to the Pro LLM benchmarks that our
16:08
friends at Process Setup. And the reason is because they
16:13
don't give any of that data out publicly and they're constantly
16:18
creating new data sets. So it feels like it's high
16:23
quality benchmarks, you know, and I know there's a lot of
16:26
discussion about how benchmarks are essentially bullshit since
16:31
you can over fit to them if the data is public.
16:35
And so the Pro LLM benchmarks feel like, hey, this is a good
16:40
way to do it. And it's got 10 different ways
16:44
to evaluate the models. So it's not just one type of
16:48
benchmark. You get different benchmarks
16:50
like there's a Stack Overflow benchmarks, there's function
16:54
calling benchmarks for agents, etcetera, etcetera.
16:58
But I don't mean to derail it, keep going.
17:01
What's the other buckets? Yeah, sure.
17:04
So, yeah, since we're on benchmarks, like as you
17:06
mentioned, there's a lot of open source benchmarks as well that
17:10
are out there. And like these benchmarks are
17:13
really like across the board, right?
17:15
Like benchmarks for testing physics or knowledge or
17:19
reasoning or even for things like hallucinations or toxicity.
17:25
Like all of these different vectors we kind of want to test
17:30
before putting production, right?
17:32
So that's why they're pretty popular.
17:33
Like you can combine, of course, multiple benchmarks together and
17:38
see across the board across the different factors like how your
17:42
model is performing, which is helpful.
17:45
But yes, at the same time, again, benchmarks have their own
17:47
limitations as well. Like lot of them are again,
17:50
fairly multiple choice type of questions.
17:54
So again, like we cannot really evaluate like the long form or
17:59
open-ended type of tasks. Are you grabbing benchmarks and
18:04
adding them to your test sets just to get like a sanity check
18:09
on your models? And if there is something that
18:13
feels like it's not really working, like toxicity score for
18:17
some reason is really high on this model, then you can go and
18:21
you can curate your own data set and try to dig into it a little
18:25
bit more. Is that something that you've
18:28
experimented with? Yeah.
18:31
So benchmarks, I would say it's very use case specific at the
18:36
end of the day. Like yes, there's a lot of like
18:39
open source benchmarks out there, but at the end of the
18:42
day, you have to like look at what exact use case your models
18:45
trying to solve and if whatever benchmark you pick is applicable
18:51
for that use case. Well, in some cases, yes, off
18:54
the shelf benchmarks are good enough.
18:57
But again, depending on like how niche your use cases, there have
19:02
been times where again, you almost have to create your own
19:06
benchmark as well. Yeah.
19:09
And then like use that because again, you don't have like an
19:15
open benchmark for your specific needs.
19:18
One more point on the benchmark piece though, while we're at it.
19:22
I like you raised an important point about essentially like
19:27
training data leakage at the end of the day, because again, these
19:30
LLMS are trained on like Internet skilled data.
19:35
If the benchmarks become part of their training data, then of
19:38
course that leads to like inflated numbers in terms of
19:44
performance or something to keep in mind.
19:47
There have been a lot of benchmarks that I've had to be
19:49
updated because of this problem. I think Lu was one of them.
19:56
MMLU also I think had to be updated to like we do because of
20:00
this problem. So, yeah, again, one of the
20:03
challenges I would say in this space.
20:06
And what are the other buckets that you see?
20:10
Sure. So the other two I would say
20:13
higher level buckets, 1 was like the traditional NLP kind of
20:18
approaches. The second bucket I would
20:21
probably put under is like human based approaches so where you
20:26
have essentially manual labelers so.
20:31
Expensive. Yeah.
20:33
And like that is one of the disadvantages of that approach
20:39
for sure. But yeah, it's still at the end
20:43
of the day, it's still like considered to be like golden in
20:49
terms of like lot of different use cases.
20:52
So with like like whenever like manual labelling is involved,
20:57
like from experience, what like I've seen as it's really
21:02
important to have like a well defined criteria for the
21:05
labelers. Because at the end of the day,
21:08
again, when subjective things get involved, it's like, OK, if
21:13
this is a three out of five answer for you, it could be like
21:17
a 5 out of five for me, like based on like subjective factor.
21:20
So it's really important to like have a good like training and
21:28
like a good definition of of like all these different
21:31
criterias to ensure consistency across like human labels.
21:35
And even within, I think human labelling, there's some
21:40
interesting approaches. So again, it doesn't always have
21:43
to be absolute scoring. There are there is this idea of
21:47
arenas as well where essentially you have like different models
21:50
are kind of facing off against each other.
21:53
So the idea that is sometimes it's easier to just give
21:56
preferences rather than absolute scores.
21:59
Like you can say, OK, this answer is better than this
22:02
versus OK, this is like a three out of five or this is like a
22:05
four out of five. So there's like chat bot arena
22:10
is like quite popular. There's like a bunch of such
22:13
arenas out there as well. And then ultimately you can
22:15
aggregate these preference scores and create a leaderboard
22:20
from from that one. I would say like extreme version
22:25
of this is like just hiring your own red team.
22:29
There's a lot of companies that are doing that now where, again,
22:34
like the the idea is, has been around for quite some time in
22:39
the cyber security space. So now that's kind of being
22:42
applied here, which is pretty interesting to me, where you
22:45
have these specialists who are like trained on essentially like
22:50
trying to break these models and trying to evaluate them before
22:54
like you expose them to like millions of your customers.
22:57
Yeah. I've even seen red teams, so red
23:00
teams as a service, but also I've seen companies that are
23:04
creating agents or a SAS platform that is basically
23:09
agents as red teams. And so it's not humans that have
23:14
been trained to break models, it is models that have been trained
23:20
to break models. Yes, that is a fairly common
23:25
approach and it's increasing in popularity now.
23:28
So that was actually going to be my third bucket which is model
23:33
based evaluations essentially. But before we go there, I think
23:38
while we're at the human evaluations piece, like just a
23:43
quick, I think pros and cons there is like the advantage is
23:47
of course that like with manual labelling, you can evaluate
23:53
across like a wide array of different spectrum.
23:57
Like you can really do holistic evaluations, whether it's not
24:01
just for quality, but also for like toxicity bias, like things
24:05
like that. It is again still considered
24:09
golden for lot of different use cases.
24:12
But of course, the disadvantage is that it can't scale that
24:15
easily. And then for domain specific
24:20
evaluations like again, if you are in the legal space or the
24:23
medical space or the finance space, you have to ensure that
24:28
your evaluators are experts in that domain.
24:32
They have domain knowledge essentially, which again can be
24:35
difficult many times and expensive.
24:39
And. Expensive.
24:40
So yeah. And that's why which brings me
24:42
to the third bucket which has really picked up I think off
24:46
late, which is model based evaluation.
24:49
So essentially using like LLM as a judge kind of approach.
24:55
So that high level hypothesis there is like if you use a more
25:00
capable model, essentially as a judge, can it provide the same
25:05
type of benefits as that of a human evaluator but without some
25:10
of the bombs? And again, there have been
25:13
studies that have shown promising results along that
25:17
front, but still it's not, I would say it's still not a
25:20
solved problem because like even with LLM as a judge based
25:24
approaches, there are challenges to keep in mind.
25:29
Like again, there's been a lot of studies that I've shown that
25:33
essentially the performance of LLM as a judge at the end of the
25:36
day boils down to your choice of judge.
25:39
And like things like prompt formatting, like how do you
25:42
format the prompt of the judge? And there is like a by.
25:45
Choice of judge? You mean the model?
25:47
Yes. Judging.
25:49
Yeah. Which model do you pick as a
25:51
judge? How do you format the Proms?
25:55
There's like some sensitivity to these parameters.
26:00
And also there have been a lot of studies that have shown that
26:04
it's important to keep in mind that sometimes, again, these
26:07
LLMS to have biases. So like verbosity bias is one of
26:12
them, where essentially LLMS sometimes tend to prefer answers
26:17
that are longer versus shorter ones, even though the shorter
26:20
ones might be of like similar quality.
26:23
There's also positional bias to keep in mind, which is some
26:27
somehow LLMS tend to prefer answers at the first position
26:33
somehow versus the others. So again, things to keep in
26:38
mind. There is something fascinating
26:40
when it comes to LLMS as a judge because so much research is
26:44
going on in this area. I love reading about how folks
26:50
are doing it and the one that struck me as a really cool idea
26:59
is LMS as a jury. So instead of saying, because as
27:03
you were mentioning you, if you just have one LLM as a judge,
27:08
then you really got to know the prompt.
27:11
Here is the best prompt that I can get and the model is the
27:15
best model that I should be using.
27:17
But I've seen others who say, well, let's just grab a bunch of
27:21
LLMS and different prompts, different models, and then we
27:26
can take the average of all of those and see if it is higher or
27:31
lower than what we are ready to deal with.
27:36
Yes, absolutely. And that's another very
27:39
interesting like idea that has come about.
27:42
I have seen like a lot of studies around that as well.
27:46
And with regards to like bias mitigation strategy, that is
27:50
definitely something which does provide promising results.
27:56
But then again, there's trade-offs there, which is like
27:59
when you have a jury now again, your Evas become more expensive
28:04
and like time consuming. So again, there's like some
28:08
trade off decisions to be made. And then again, when you have a
28:11
jury, it's like, OK, how do you, how do you consider like all the
28:16
different outputs from different judges?
28:18
Like, you know, how do you weight them separately, do
28:20
weight them equally? Like then you go into like those
28:24
kind of details. There's that detail and there's
28:28
another way that I've seen it done, which is you have one LLM
28:32
as a judge, but then you have another LLM judging the judge.
28:36
And so it's saying, hey, let's see if this matches up the the
28:42
LLM as a judge said this and you give both of the answer and what
28:47
the LLM as a judge said to another LLM and say, does this
28:51
make sense? And so then it will judge the
28:54
judge. And So what are some LLM as a
28:57
judge best practises that you've been seeing?
29:01
Yeah. So this technique has definitely
29:05
picked up quite a bit of late because essentially what we've
29:09
seen is it provides the flexibility that human
29:13
evaluators provide, but of course it's much more scalable
29:16
if done right. So there has been a lot of work
29:20
that's gone in I think over the last couple of years in the
29:23
space and they have been some now well defined set of like
29:29
best practises while using LLM as a judge.
29:34
So the first thing is like, how do you validate the validator,
29:38
right? We spoke about that a bit.
29:40
So essentially one idea that has worked for a lot of people is
29:45
like to have a golden data set first, which is like created by
29:50
domain experts by manual labelling which you have
29:54
complete confidence on, and then use that to calibrate the judge.
29:58
So you would pick your, your judge model, you would finalise
30:03
the prompt and all of that stuff based on that to ensure that
30:08
there is a high correlation with that data set.
30:12
So at least you have some confidence.
30:15
Again, like any ML model, drifts do happen.
30:19
So the judge is expected to drift overtime.
30:22
So again, it's very important to like monitor the judge and
30:26
recalibrate as needed overtime. The judge is usually again, it
30:32
can be used to do either like absolute scoring or pair wise
30:36
comparisons. For more subjective type of use
30:42
cases, pair wise comparisons are usually better than absolute
30:47
scores. But again, if you're do using
30:50
absolute scoring, another really good practise, and again, a lot
30:56
of studies have been published around this, is to have the
31:00
judge use as low of a precision scale as possible.
31:06
So like for. So what I mean by that is
31:08
essentially having the judge preferably output A0 or A1
31:12
versus it outputting like a score between zero to 100.
31:17
Again, just like humans, they can be a lot of confusion for
31:21
the LLM as a judge as well, which is like, you know, how do
31:24
I differentiate between a 60 out of 100 and an 80 out of 100
31:28
answer? So like essentially it's really
31:31
good to keep it simple. It's also helpful to like add
31:37
few short examples to the prompts.
31:40
Again, you're kind of giving that reference to the judge.
31:43
So that has shown to be like pretty helpful, asking the judge
31:50
to output reasons with every evaluation.
31:54
That's another very like important point that has helped
31:58
especially with debugging. So when you ask the judge to
32:01
really reason like why it thinks something is not good or
32:06
something is good, you can check it's reasoning and like use that
32:11
to iterate. And then there's some really
32:14
interesting work that has gone on recently where there's this
32:18
idea of grading notes that was introduced like I think just
32:22
couple of months ago, which has also shown to really improve the
32:26
performance of LLM as a church, especially on domain specific
32:30
evaluations. So essentially the idea of these
32:33
grading notes is for your evaluation set for each row, you
32:39
would kind of add like a more detailed like version of what to
32:45
look for. So if it's, let's say, like a
32:47
code execution task, you would say, OK, like, please check if
32:53
basically from the input file, like are all the columns right?
32:58
Like is the formatting of all of the inputs right?
33:02
Like check for essentially like any null type errors or like
33:09
those type of exceptions. So essentially like adding these
33:12
kind of granular notes to the church really helps, especially
33:18
in in like the domain specific evaluations where the judge
33:24
might not have all the knowledge.
33:27
I after reading a few research papers, I'm like, man, I swear
33:35
some of these researchers got to be so high and they're eating
33:38
pizza in their dorm room or they're just enjoying life and
33:43
they're thinking, you know, it would be awesome if we just had
33:46
more LLM calls. That's what we need the to
33:50
remedy the problem here, we'll just have more LLMS. Yeah, I
33:57
think you heard on a very important challenge with this is
34:02
at the end of the day, like who validates the validator or like
34:06
how do you validate the validator given the validator
34:09
itself is another AI in this case, right.
34:12
So one of the like the best practises like I've seen like
34:17
different people use in the industry is essentially you kind
34:23
of have like a golden test set, which is again very trustworthy.
34:28
So let's say, again, you have like domain experts who would
34:33
essentially create that golden data set, like do those Evas for
34:38
you, you have that golden data set.
34:40
And then you use that to kind of calibrate the judge in the 1st
34:45
place. And you first like establish
34:48
some form of correlation with that golden data set where then
34:52
you can say, OK, the judges like whatever, 80 percent, 90%
34:56
correlated with my golden data set.
34:58
So then I have a fair amount of confidence that OK, this judges
35:04
is going to do like fairly well. But again we are dealing with
35:09
challenges of overfitting there as well, the classic bias versus
35:13
variance trade off in ML systems.
35:16
And then another thing to keep in mind with that is of course
35:19
that like every ML system there is going to be drift with time.
35:24
So it's like you constantly have to monitor the judge again, like
35:28
you know, is there drift in the judges judgments overtime?
35:32
Like how frequently do we need to retrain and things like that?
35:37
And. You're bringing up a great point
35:39
that I just literally finished writing A blog post on, which is
35:45
the costs associated with this type of stuff and it came about.
35:49
My blog post came about because I, I saw yet another person on
35:54
the Internet posting about how the cost per token is going
35:57
down. And I, I wrote, yeah, cost per
36:01
token is going down, but cost per answer is going up.
36:06
And I didn't really explain myself good enough I guess with
36:10
that comment. And the person replied like no,
36:14
any given model anything, the best models, the tokens cost are
36:18
all just plummeting. And I'm like, yeah, I didn't say
36:21
cost per response is going up. I said cost per answer.
36:27
And what I meant by that is if you look at the whole system and
36:30
you look at everything that has to do with going into getting
36:34
you a quality answer, you as that end user getting that
36:38
quality answer. All of this that we're talking
36:40
about LLM calls like up the wazoo because we've got a judge
36:45
and a jury and the judger that's judging the jury.
36:48
And also all of the human resources that are going into
36:54
building out the system. If you're using agents, like
36:58
don't even talk about how many more LLM calls you need because
37:02
of the all of the agents. And that's not even factoring in
37:07
this golden data set that you need to pay humans to curate.
37:12
And now there's a tonne of other factors that I go into in depth
37:17
in the blog post, but that's like the gist of it.
37:20
The cost per answer. No one can say that that is
37:24
going up or that is going down. That is 100% going up.
37:30
Yeah. I think that kind of boils down
37:32
to, I think the systems that we are now designing like we are
37:38
taking on more complex tasks. And as you rightly mentioned,
37:42
when you look into like RAG or agentic workflows, like some of
37:46
these systems are extremely complex involving like multiple
37:50
LLM calls. Sometimes there is a lot going
37:53
on behind the scenes. So, yeah, as like these models
37:58
get better, I feel we are also trying to tackle bigger tasks,
38:02
bigger challenges which like creates very complex systems
38:07
which has an impact on on the cost.
38:10
But there have been again, some interesting, there has been some
38:14
interesting work in that space as well where I have seen a lot
38:19
of folks again try to use like small language models rather
38:23
than large language models in again.
38:27
That's so funny you say that because there's a whole part of
38:30
the blog that I wrote where I said, all right, this is where
38:34
folks are going to come in and saying, but what about open
38:37
source, small domain specific models?
38:41
And I said, OK, that's kind of arguing my point though, because
38:45
let's think about that. Who is gathering the data to
38:49
fine tune those models? If they are fine-tuned?
38:53
What are you using the GPU cost to fine tune the models?
38:57
The people that actually know how to fine tune the models, how
39:00
much do they cost? Are you hosting those models and
39:04
are you renting GPU's to host those models on your own?
39:07
How much is that? And you better have some good
39:09
alerting set up. So there's all this extra cost
39:12
that comes into it that I do see is not part of the LLM call
39:17
itself. But if you again are looking at
39:20
it in this systematic way, in a holistic way, you recognise that
39:25
the cost per answer is going up. Yeah.
39:30
Again, that's a valid point that even with small language models,
39:35
we have to factor in like all of these different costs.
39:39
At the end of the day, I think it boils down to the use case.
39:42
Like again, is your use case like specific enough that a
39:47
small language model could work versus having like a more
39:51
capable like large language model?
39:54
So I think a lot of these design and create off decisions have to
39:57
be made kind of like specific to the use case.
40:00
And you, you said it perfectly with the complexity, we, we
40:04
expect so much more from AI these days and their
40:08
capabilities that inevitably that just translates to more
40:13
complexity in the systems that we design.
40:16
And so since we expect more, we're not just going and
40:22
creating a system that hits an API once and then comes back
40:28
with an answer, no, we've got to set up a whole rag system and or
40:32
we've got agents that we want to actually go out and do things or
40:36
plan and reason around these things and then go and execute.
40:39
So it, it's interesting to think about that when you're looking
40:44
at these different systems and I know that you have thought about
40:48
evaluating RAG and evaluating agents in different ways.
40:54
And so let's like jump into that.
40:56
What are some things that you keep top of mind when you are
40:59
evaluating RAG system? Yeah.
41:03
So that's a very good question. So as we like develop more
41:08
complex systems along the way, again evaluate evaluation
41:13
remains like a challenge. It is very important to keep in
41:18
mind like with these complex systems that we need to evaluate
41:22
the parts and kind of the sum of the parts ultimately.
41:26
So in like the more traditional software sense, it's like you
41:29
need to do unit testing and integration test at the end of
41:32
the day for the whole system end to end, right.
41:35
So when we talk about, it's really important to like
41:39
evaluate different parts of the rack system.
41:42
Like you want to know how well is the retrieval doing?
41:46
Like is the retrieval actually fetching relevant like chunks
41:52
from the vector database for you?
41:54
So there again, there has been a lot of work again done in the
41:58
space because that's essentially a search problem.
42:00
So there are more traditional metrics like your MRR and BCG
42:06
reciprocal. I mean, recall at K, precision
42:09
at K. So these metrics have existed
42:12
for quite some time, again, because the search problem has
42:15
existed for quite some time, like multiple decades.
42:18
So I do see like reuse of some of that work within like
42:24
retrieval systems as well. But then of course, there's
42:28
scope for newer techniques as well, like LLM as a judge.
42:31
Again, I've seen like lot of open source packages and tools
42:36
out there using LLM as a judge just to like evaluate the
42:41
retrieval system. And that's just the retrieval
42:44
system piece, right? Then we go on to the answer
42:47
generation system, like, OK, once you retrieve relevant
42:50
chunks, you want to make sure that the LLM that's answering
42:56
eventually is, is generating good answers.
42:58
And then you want to test the whole system end to end, right?
43:01
Like right from where you have a user query to all the way into
43:07
like generating the final answer.
43:08
Like, how good is that system? How consistent is that system?
43:13
Like is there any like, bias in there?
43:17
Is it producing like toxic outputs?
43:19
Like, yeah, there's, I think there's absolutely a need to
43:24
like, think of it holistically. Yeah.
43:28
So when you are doing unit tests, you're thinking about the
43:33
different pieces where you can add that evaluation in.
43:37
And you mentioned one, which is how's it retrieving?
43:41
Are you also doing it on the on the embedding model or are you
43:50
doing it on like where are all the places that you're putting
43:53
in unit tests, I guess as you set up your system and your rag?
43:57
Right. So again, that's a very good
44:00
question with a system like RAG, which again there's so many
44:05
bells and whistles that she can pull like each of those things
44:11
kind of requires it's own set of evals, right.
44:13
Like OK, what kind of chunking strategy do you want to use?
44:17
What kind of embedding model do you want to use again to make a
44:22
lot of those decisions if you'll evals and Even so, yeah, if you
44:31
want to ensure like success of the complete bigger access them,
44:37
I feel like you need all these evals like at the individual
44:40
level and then at the system level as well.
44:45
And the, yeah. And then the integration tests,
44:47
what do those look like? Sure.
44:49
So the integration test in this case would be you're just
44:52
testing the whole system like end to end, So right from when
44:55
you get the user query interact system to your final output,
45:00
right. And again, so a lot of the
45:02
foundational techniques that we discussed earlier come into play
45:06
here. So like you can have your own
45:09
benchmark, you can create your own benchmark or you can reuse
45:13
an existing benchmark from the open source domain.
45:17
We can have like again, human evaluators, we can have a red
45:21
team as well try to evaluate the system or again, we can always
45:27
have like LLM as a judge based approaches there.
45:31
So. And so when I traditionally
45:36
think about tests in software development, it is a very
45:42
instantaneous thing that you know, if my tests failed or
45:46
passed, right. But with this, as soon as you
45:50
start bringing humans into the mix or if you start bringing LLM
45:55
calls into the mix, are you still getting that instantaneous
46:00
feedback to know if your tests fail or pass?
46:03
And are you doing these types of extended tests?
46:08
I guess what does, what does it look like?
46:11
Is it a monthly thing that you're bringing a human in and
46:15
asking it to look and looking at the drift and really having deep
46:21
examination there? Or is it more of we do it once
46:26
and then it's more of like a yearly maintenance?
46:29
How do you see that? Right.
46:33
So yeah, evaluations are again very important, I feel in the
46:37
entire life cycle of the project.
46:40
So like I think what we discussed and now was kind of
46:43
offline evals. So like all the evals should do
46:46
before you launch the model. And then there's this again,
46:50
completely different world out there of like online evals or
46:53
production evals as well, right. So once you launch the system,
46:57
you constantly want to monitor and make sure it's producing
47:01
like the right quality of outputs.
47:03
So even for that, yes, there are different ways of doing that.
47:08
So like establishing feedback loops with your customers, like
47:12
again, the classic thumbs up comes down.
47:15
And again, so not everyone likes to provide those kind of inputs.
47:20
So sometimes you also have to look at like more implicit kind
47:24
of signals like are they trying to search for help somewhere
47:27
else rather than like, you know, are they abandoning your system
47:31
like more frequently? So there's like lot of some of
47:35
these implicit systems that can be used in this signals.
47:38
And then again, I've seen approaches where again, when you
47:42
use LLM as a judge like essentially deploying the same
47:46
thing online as well. And you're having the LLM
47:50
judgements kind of like on a daily basis or on an hourly
47:54
basis as well in some use cases. But on the cadence piece, I do
48:00
feel like it is very important to have evals like as frequent
48:06
as possible so you can monitor for like drift and quality and
48:11
things like that. Why do you feel like maintaining
48:15
production ML systems is such a challenge and under appreciated?
48:24
Yeah, it is. I think it is one of those
48:27
things which I feel like people don't think of that as like, oh,
48:33
this is like something cool to do, but it is extremely
48:37
essential to ensure like a good experience for customers
48:41
eventually in production, right. So with software, again,
48:45
traditional software systems, when we speak about maintenance,
48:49
it was more around like ensuring that we take care of technical
48:53
debt, we ensure that all the dependencies, all the different
48:57
libraries that are up to date. But with ML systems, there is
49:01
like an added layer of complexity to that whole thing,
49:04
right? So you have all of the software
49:08
based maintenance that you still need to do.
49:11
And on top of that, now there is all this data level stuff that
49:16
you need to look into as well, right?
49:17
Like is your data drifting in production versus what to train
49:21
the model on? Are there any issues with your
49:24
features or is there any drift in the model itself?
49:27
Like is the models prediction kind of drifting over time and
49:31
trying to debug, OK, why that is?
49:34
And then again, constantly having to retrain the systems,
49:37
the models to make sure that they are up to date, like with
49:42
what's going on in your product and with customers expectations.
49:46
And again, at the rate of rate at which I feel like the whole
49:51
AI space is moving again, just essentially keeping up with all
49:55
of that progress, right? Ensuring that we have like
50:01
again, good quality models, if there are better models out
50:04
there, if there are better techniques out there, again, how
50:07
do we again incorporate that into the system?
50:12
Again, there's new and new better tools that are coming out
50:15
everyday. So again, it's like, OK, now how
50:19
do we incorporate that into our system?
50:22
So as you can see, it is, there's basically, it ends up
50:26
being like a very huge task, which I feel is like sometimes
50:33
overlooked because like people want to go and build things from
50:38
scratch without like realising, like even the current systems in
50:43
production, like maintaining them.
50:46
Is is is like an absolutely like a big, I would say like resource
50:53
requirement for sure. Yeah, it's a beast, especially
50:57
as you start to add more models into production and you start to
51:02
try to recognise which models are creating value.
51:06
How can we get more of those types of models, right.
51:10
And so Speaking of along that line, how do you look at
51:18
evaluating and identifying the right use cases for different AI
51:25
and ML applications? At the end of the day, I think
51:30
like you just have to be very like surgical about like where
51:35
to where you really want to deploy like MLAI modules when
51:41
there's like a lot of like background work.
51:44
I feel that goes into that where you would be talking essentially
51:48
with like all the product people design with like engineers, like
51:54
basically the whole cross functional team and really
51:57
trying to like identify like what is a hypothesis at the end
52:02
of the day. Like if we had AAI model and ML
52:06
model here in this space, like what's a hypothesis?
52:09
Like how can it move the needle versus like, let's say a rules
52:14
based approach or like something like that?
52:17
So and then essentially once you have that hypothesis, then it's
52:22
all about like, OK, how can we go and validate that hypothesis
52:26
as quickly as possible? So can we like launch like
52:31
essentially MVP, like a minimal viable product in there, do some
52:36
quick testing, see if like basically essentially validate
52:40
that hypothesis and then OK, if we see like good promising
52:45
results, then OK, now let's scale this to like a full blown
52:49
system. So I think that's usually the
52:52
approach I've seen right being taken.
52:55
What are some things that you haven't been able to figure out
52:59
recently? A.
53:01
Lot of lot of things I think remain like big challenges.
53:06
I think evaluation is definitely like evaluation of generative AI
53:10
systems particularly is, is definitely like very important
53:17
and still remains like a challenge.
53:20
Again, I think we spoke about RAG, but now again with the
53:24
advancements in like agentic workflows, I think again, we are
53:28
essentially entering a space where the with the systems, the
53:33
evaluations also getting more complex.
53:35
So with agents, again, depending on like how many steps are there
53:41
in your agentic flow, Again, I think the same concept applies
53:45
where you need to evaluate every step output and then you again
53:49
need to evaluate the whole end to an output.
53:52
Also, depending on how many tools your agent is calling or
53:56
has access to, the evaluation complexity like increases quite
54:01
a bit because we're tool calling again, we are essentially
54:04
looking at two things, right at the end of the day, like is the
54:09
LLM picking the right tool for the task?
54:11
And then B are the right parameters being passed to the
54:16
to the tool that's picked. So there's two factors that
54:20
you're evaluating for each tool. And then as again, the agent has
54:24
access to like more and more tools, that adds, again,
54:27
significant complexity. Yeah.
54:31
But at the same time, I do see a lot of interesting ideas, lot of
54:35
interesting approaches coming up.
54:38
So I know one of the ideas I was basically thinking about is like
54:44
with cloud launching, like essentially computer use, right?
54:48
Like it's like, what if you could simulate users now with
54:55
cloud where it's like literally you have cloud play like
55:00
different personas and then go and kind of do the testing of
55:06
these agentic systems or, or like rag world flows.
55:10
I feel it's pretty fascinating right now.
55:12
It's like, yeah, we have more complex systems and we're trying
55:17
to come up with solutions for them.
55:19
And now we do have more capable tools and more capable models.
55:24
Again, I think when we think about multimodal LLMS, that's a
55:28
completely different beast all together.
55:31
I feel like we were barely figuring out the text output and
55:35
now we also are now in domains where we have to look at OK
55:38
image outputs and speech and audio and like what have you.
55:42
So yeah, I personally find it pretty fascinating.
55:49
I'd like this idea of COD computer use going and stress
55:56
testing or validating your AI workflow that you've built or
56:02
your RAG pipeline and you are able to see if you have
56:09
different ways of synthetic data generation in a way trying to
56:15
use your product and use it with different persona prompts.
56:22
Yeah, absolutely. It's a fascinating idea.
56:25
Some of it is already used, I think in the self driving space
56:29
where you have companies like Waymo, Cruise, Tesla, where like
56:37
they simulate lot of these cases like in in the virtual world and
56:43
they use like RL essentially to create like this huge model from
56:49
all that feedback. So I feel like again, with
56:54
open-ended output where again, there's infinite different
56:57
possibilities out there, I do see that idea like being pretty
57:03
interesting. So I think it's fairly new
57:06
again, I would probably love to see that more studies and like
57:09
results from that. I know there's a bunch of
57:12
startups as well, like working in that space, trying that idea
57:17
out. So I think it's going to be
57:19
pretty interesting.