Latest / Agentic Conversations (formally mlops.community) / LLM Evaluation with Arize AI's Aparna Dhinakaran // #210

Transcript

LLM Evaluation with Arize AI's Aparna Dhinakaran // #210 · Agentic Conversations (formally mlops.community)

0:00
Hold up. Before we get into this next
0:02
episode, I want to tell you about our virtual conference
0:06
that's coming up on February 15th and February 22nd.
0:10
We did it two Thursdays in a row this year because we wanted to
0:15
make sure that the maximum amount of people could come for
0:19
each day since the lineup is just looking absolutely
0:24
incredible. As you know, we do.
0:27
Let me name a few of the guests that we've got coming because it
0:31
is worth talking about. If you got Jason Louie, we've
0:34
got Shreya Shankar, we've got Drove who is product Applied AI
0:39
at Uber. We've got Cameron Rook Wolf
0:42
who's got an incredible podcast and he's Director of AI at Rebuy
0:45
Engine. We've got Lauren Lockridge who
0:49
is working at Google also doing some product stuff.
0:53
Ohh, why is there so many product people here?
0:55
Funny you should ask that because we've got a whole AI
0:59
product owners track along with an engineering track.
1:03
And then as we like to, we've got some hands on workshops too.
1:07
Let me just tell you some of these other names just for a
1:10
moment, you know, because we've got them coming and it is really
1:14
cool. I haven't named any of the
1:15
keynotes yet either, by the way. Go and check them out on your
1:19
own if you want. Just go to home dot ML OPS dot
1:22
community and you'll see what we've got.
1:24
Tungi, who's the lead researcher on the deep speed project at
1:28
Microsoft. We've got Holden, who is the
1:31
open source engineer at Netflix. We've got Chai, who's leading
1:36
the AI platform at Uber. You may have heard of it.
1:40
It's called Michelangelo. Ohh my gosh.
1:42
We've got Phazon, whose product manager at LinkedIn, Jerry
1:46
Lewis, who created good old Lama Index.
1:50
Well, he's coming. We've got Matt Sharp, friend of
1:52
the pod, Shreya Rajpal, the creator and CEO of Guardrails.
1:58
Oh my gosh, the list goes on. They're 70 plus people.
2:02
That will be with us at this conference, so I hope to see you
2:08
there. And now, let's get into this
2:10
podcast. Hey everyone, my name is Aparna.
2:13
I'm one of the cofounders of Arize AI and I recently stopped
2:20
drinking coffee. So I take I I've started on
2:23
matcha lattes instead. Hello and welcome back to the
2:27
Mill OPS Community Podcast. As always, I'm your host Dimitri
2:31
Osam. We're coming at you with another
2:34
Fire episode. This one was with my good and
2:39
old friend Aparna, and she has been doing some really cool
2:43
stuff in the evaluation space, most specifically the LLM
2:48
evaluation space. We talked all about how they are
2:54
looking at evaluating the whole LM systems and of course she
3:00
comes from the observability space.
3:02
And for those that don't know, she's cofounder of a Rise and
3:06
Rises doing lots of great stuff in the observability space.
3:09
They've been doing it since the traditional ML OPS stays and now
3:14
they've got this open source package, Phoenix that is for the
3:19
new LLM days. And you can just tell that she
3:23
has been diving in head first. She's Chief Product Officer and
3:29
she has really been thinking deeply about how to create a
3:31
product that will help people along their journey when it
3:35
comes to using LLM's and really making sure that your LM is
3:41
useful and not outputting just absolute garbage.
3:44
O. We talked at length about
3:48
evaluating rags, not only the rag, the part that is the
3:53
output, but also the retrieval piece.
3:55
She also mentioned and she was very bullish on something that a
3:59
lot of you have probably heard about, which is LM's as a judge.
4:04
So I really appreciated her take on how you can use LM's to
4:09
evaluate your systems and evaluate the output.
4:12
But then at the very end we got into her hot takes then.
4:16
So definitely stick around for that because she thinks very
4:20
much on the same lines as I do. I don't want to give it away,
4:23
but she came up with some really good stuff when it comes to fine
4:27
tuning and traditional mill and how traditional ML engineers
4:32
might jump to fine tuning. But that is all.
4:35
No spoilers here. We're going to get right into
4:37
the conversation and I'll let you hear it straight from
4:39
Aparna. Before we do though, huge shout
4:42
out to the Arise team for being a sponsor of the ML OPS
4:46
community since 2020. They've been huge supporters and
4:51
I've got to thank them for it. A partner was one of the first
4:54
people we had on a virtual meet up back when everything was
4:58
closed in the COVID era and she came into the community.
5:03
Slack was super useful in those early days when we were all
5:06
trying to figure out how to even think about observability when
5:10
it relates to ML. And so I've got to say huge
5:16
thanks. Huge shout out to the Arise
5:18
team. Check out all the links below if
5:20
you want to see any of the stuff that we talked about concerning
5:24
all of the LM observability or just ML observability tools.
5:29
And before we get into the conversation, would love it if
5:31
you share this piece with just one person so that we can keep
5:36
the good old ML OPS vibes rolling.
5:40
All right, let's get into it. OK, so you wanted Beast story
5:49
about how I ended U in Germany. Here it is.
5:52
Here's the TLDR version. I was living in Spain, so I
5:55
moved to Spain in 2010. And I moved there because I met
5:59
a girl in India and she was in Bilbao, Spain doing her masters.
6:03
She wasn't from and she wasn't from India or Spain.
6:05
She was from Portugal. But I was like, oh, I want to be
6:08
closer to her. And I also want to like, live in
6:11
Spain because I enjoyed it. I had lived in Spain.
6:14
I spoke a little bit of Spanish and Poquito and then I was like,
6:18
alright, cool, let's go over to Bilbao.
6:21
I've heard good things about the city and the food and the
6:23
people, so I moved there. As soon as I got there, this
6:27
girl was like, I want nothing to do with you.
6:29
And so I was sitting there like heartbroken on the coastline of
6:36
the Basque Country and it took me probably like a month to
6:40
realise, well, there's there's much worse places I could be
6:43
stuck. And so I enjoyed it.
6:45
And I had the time of my life that year in Bilbao, and then I
6:50
met my wife at the end of that year at this big music festival.
6:54
And so we were living in Spain. We ended up getting married like
6:59
five years later, had our first daughter like 8 years later.
7:02
And then we were living there until 2020 when COVID hit.
7:05
And when COVID hit, the lockdown was really hard.
7:08
And we were in this small apartment in Bilbao and we were
7:12
like, let's get out of here, let's go to the countryside.
7:15
And we had been coming to the German countryside because
7:18
there's like this meditation retreat centre that we go to
7:21
quite a bit. And so we thought, you know
7:24
what, let's go there. Let's see, like if there's any
7:27
places available and we can hang out on the countryside, not see
7:32
anybody. The lockdowns weren't as strict.
7:35
I mean, there were lockdowns and stuff, but when you're in the
7:37
countryside, nobody's really enforcing it.
7:39
So we did that and we ended up in the middle of nowhere.
7:44
Germany was, you know, 100 cows and maybe like 50 people in the
7:47
village that were in. So that's the short story of it.
7:52
Wow. Well, that's an interesting
7:55
intro. There you go.
7:58
I mean we were just talking and I will mention this to the
8:01
listeners because we were talking about how you moved from
8:04
California to New York and you are freezing right now because
8:08
it is currently winter there and Germany isn't known for its
8:14
incredible weather, but it's definitely not like New York,
8:17
that is for sure. It's East Coast winter out here,
8:20
so. I wanted to jump in to the
8:25
evaluation space because I know you've been knee deep in that
8:28
for like the last year you've been working with all kinds of
8:32
people and maybe you can just set the scene for us because
8:37
you're currently for those who do not know you.
8:39
I probably said it in the intro already, but I will say it
8:41
again, You're the head product or Chief Product Officer I think
8:45
is the official title at Arise and you have been working in the
8:50
observability space for ages. Before you started to rise, you
8:54
were at Uber and working on good old Michelangelo with that crew
8:59
that is got very famous from the paper.
9:02
And then you've been talking a tonne to people about how
9:08
they're doing observability in the quote unquote, traditional
9:12
ML space. But then when Elms came out, you
9:16
also started talking to people about, OK, well, how do we do
9:18
observability? What's important with
9:20
observability in the LM space? And so I'd love to hear you set
9:26
the scene for us. What does it look like these
9:28
days? I know it's hard out there when
9:30
it comes to evaluating LLM's. Give us the breakdown.
9:35
Let let's jump in. So I.
9:38
So First off, we're seeing a tonne of people trying to deploy
9:43
LM applications like the last year Dimitrios has just been.
9:46
It's been super exciting. People are right.
9:50
You know, I'm not just saying you know the the fast moving
9:52
startups. I'm saying there is like you
9:55
know older companies, companies that you're like wow, they're
9:57
deploying LMS that have like a you know skunks work team who
10:01
are out there trying to deploy these LM applications into
10:04
production. And in the last year, what I
10:06
think we've seen is that the there's a big difference between
10:12
a Twitter demo and a real LM application that's deployed.
10:17
And yeah, and the fact that what we're seeing is that, you know,
10:23
unlike traditional malware, people have deployed these
10:26
applications. And there's a lot of people who
10:27
have that kind of experience with LMS. It's still relatively
10:31
a small group or few people who are who have deployed
10:34
successfully these applications and would you know the hardest
10:39
parts about this still ends up being evaluating the outcomes.
10:43
You know, in the new era in traditional ML you had, you know
10:48
one of the things that still matters is you want your
10:50
application to do well. In traditional mail you had
10:53
these common metrics, right? You had classification metrics
10:55
and regression metrics and your ranking metrics in the new
10:59
element era. You can't just put, you know,
11:03
these metrics, you know, you know, what I'm saying is like we
11:06
saw in the beginning, some people were doing things like
11:08
Rouge and blue score. Oh, it's translation task.
11:11
Oh, it's summarization task, but there's a lot more that we could
11:17
do to evaluate if it's working. And the biggest one that we're
11:20
seeing kind of take off is you've probably been hearing it
11:24
is LLM as a judge. And so it's a little meta.
11:29
It's where you're asking an LM to evaluate the outcome of an L
11:36
and IT, I mean, we're finding that across deployments, across
11:41
what people are actually putting in production.
11:43
It's one of the teams. It's one of the things that
11:45
teams actually really want to get working, and I don't know,
11:49
it's not that crazy as you start to think about it.
11:51
Humans evaluate each other all the time.
11:53
We interview each other. We greet each other seeing our
11:55
teachers grade students papers and it's it's a very you know
12:00
you know it's not that far of a jump to think AI evaluating AI
12:06
but that's that's kind of this novel new thing in the LM space.
12:12
So you don't have to wait for the ground truth necessarily.
12:16
You can actually just generate an eval to figure out was this a
12:20
good outcome or not. And the thing that my mind
12:25
immediately jumps to are all of these cases where you have
12:29
people that have deployed LMS or chat bots on their website a
12:34
little bit too soon and you see the horror stories because
12:38
people go in, they gaolbreak it and it's like, ohh man, that is
12:44
not good what this chat bot is saying on your website.
12:49
All of a sudden I saw one screenshot where people were
12:54
saying, you know, I can't remember what the website was,
12:56
but people were talking about how, Well, I don't even need
12:59
Open AI, I can just use the chat bot on this website.
13:02
It's obviously ChatGPT or GPT 4. Like you can ask it anything and
13:09
it will give you any kind of response and you can play with
13:13
it just like you would play with Open AI LM or GT4.
13:20
Like is this, you know, asking questions about the product and
13:25
saying, but the product isn't that good, is it?
13:27
And then leading it into the chat bot then saying, yes, this
13:31
product is actually horrible, you shouldn't buy it.
13:33
And it's like you can't say that on your own website about your
13:36
product. This is really bad.
13:38
So does like do the LM's as a judge and stop that from
13:43
happening? I think is the really
13:45
interesting piece, no? I mean it's absolutely a
13:48
component of it. So you're absolutely right.
13:50
People don't want, you know, the common applications we're seeing
13:54
right now is like I think you hit one of them which is like
13:56
chat bot on your docs or chatbot on your product.
13:59
So give me some kind of like a replacement for like the
14:03
customer support bots we had. And then there's kind of some
14:07
more you know, interesting ones which I've been calling like a
14:11
chat to purchase type of application where a lot of these
14:16
companies that are doing like used to do recommendation models
14:20
or et cetera or are you know, maybe selling you trips or
14:26
selling you some kind of products now have a chatbot
14:30
where you can go in and actually explicitly ask, hey, I'm doing
14:34
XYZ, I'm looking for this. And then it recommends a set of
14:38
products. And sometimes these applications
14:40
use both ML and LM's together in that shop.
14:44
But like the LM's doing the chat component, it's doing the
14:48
structured extraction. But then the actual
14:50
recommendation you might call out to an internal
14:53
recommendation model. So it's not one or the other,
14:55
but sometimes you have both of them working together in a
14:57
single application and you're absolutely right, they don't
15:00
want stuff like it saying stuff it shouldn't say giving.
15:06
We had one interesting case where someone was like, I don't
15:09
wanna say we support something in this policy because I'm
15:12
reliable to it if someone asks the question.
15:16
And so there's all sorts of, especially if you're putting an
15:20
external facing, there's all sorts of more rigour that it
15:24
goes through to make sure it's it's working.
15:26
And what Elm isn't judge can do is what we see people checking
15:30
various things like, well, did it hallucinate?
15:33
And the answer is it making up something.
15:36
Like is it making up something that wasn't in the policy?
15:39
Is it toxic in its response? Is it negative about, you know,
15:43
like in the one where it's kind of shit talking its own product?
15:46
Is it, is it negative in in its own response?
15:50
Is it correctness, factuality. So all of these are things that
15:54
you can actually generate a prompt to, you know, generating,
15:58
you know, basically an eval template to go in and say, well,
16:02
here's what the user asked, here's what all the relevant
16:04
information we pulled was, and then here's the final response
16:07
that the Ellen came back with. Does the response actually
16:10
answer the question that the user asked?
16:12
And two, does it actually, is that answer based on something
16:17
factual, AKA the stuff that it was pulled on the retrieval
16:21
component? And you can go in and score that
16:24
and you know what we end up seeing a lot is if the response
16:28
isn't actually based on the retrieval, then it's
16:31
hallucinating and they actually don't want to show those types
16:34
of responses back to the user. And so this is, this is just a
16:39
very specific, I see the hallucination eval, the
16:43
correctness eval, summarization, all of these are very common
16:49
kind of LLM tasks, evals that we're seeing out in in kind of
16:53
the wild right now. I should mention it's very, very
16:56
different than what the model evals are, which is a whole
17:00
nother category of evals you might be seeing like if you go
17:04
on like Hugging Face for instance has a whole open source
17:07
LM leaderboard, I'm sure you've all seen it.
17:09
It's like changes every couple of hours and they have all these
17:14
metrics, right? MMU and hella swag, which is all
17:19
different ways of measuring how good the LCM is across a wide
17:25
variety of tasks. But for the average kind of, you
17:30
know AI engineer who's building an application on top of an LM,
17:36
they kind of picked their LM and then they're not really looking
17:41
at how well does the model do across all sorts of
17:44
generalizable multimodal kind of tasks they care about.
17:49
Specifically, I'm evaluating how good is this LLM and the prompt
17:55
template and the are you know, the structure that I built doing
17:59
on this one specific task that I'm asking it to do.
18:03
And so it's a very, does that make sense like that delineation
18:07
between the the model evals versus the task evals here?
18:11
Yeah, 100%. And you did say something else
18:15
like how does. I guess what my mind goes to
18:20
here is how you are able to restrict the output from getting
18:28
to the end user. Is it that if that LLM is a
18:30
judge gives a certain confidence score than anything lower than
18:36
whatever A5 of confidence that this is the right answer, It
18:41
doesn't go out to the end user and it has to regenerate it?
18:44
Or what does that look like? Like that last Mile piece?
18:47
Yeah, I think it depends back on the application owner.
18:51
So there are some people who will generate that eval and then
18:54
decide not to show that response because the eval was, um, was
18:59
pretty poor. But there's some folks where you
19:05
know, if it's a lower risk type of application, then they still
19:08
show it, but then they can come back and you can use the ones
19:11
where it did poorly on to come back and figure out how to
19:14
improve the application. So I think there's a, there's
19:20
kind of like a, you know, a trade off that the application
19:23
owner has to make between do you want to block kind of you know,
19:28
you got to wait for the LMP evaluate and then you're going
19:30
to have for sure impact speed of the application experience.
19:34
And So what we see a lot of people doing is these evals, you
19:40
don't just have to do them in production.
19:42
So similar to how you know in the traditional mill world you'd
19:45
have a, you know a performance like an offline performance as
19:50
you're building the model and then you're kind of monitoring
19:53
an online erformance. Well here.
19:56
Similarly as you're building an eval, you're kind of you're
20:00
seeing folks kind of evaluate the LLM offline, see how you
20:05
know build some confidence essentially around you know how
20:10
how well is the application doing before they actually go on
20:14
to to pushing it out to an online monitoring.
20:17
So there's there's a lot of these similar kind of paradigms
20:19
that that still apply in in this space.
20:23
Yeah, I would imagine you try and do a little red teaming.
20:25
You try and make it hallucinate, see how much you can break it
20:30
before you put it out there and and set it live if you're doing
20:34
it the right way and not just rushing it out the door.
20:37
Ideally the other piece on this that I think is is important is
20:42
a little bit more upstream because you were talking about
20:44
how the LM is a judge. It's evaluating if the answer
20:50
that the other LM gave with respect to the context that it
20:55
was giving or that I was given was actually the right answer or
21:01
evaluating that answer based on the context.
21:04
But then one step above that is actually getting the right
21:09
context I would imagine. And so making sure that are we
21:14
getting the context that is relevant to the question that
21:18
was asked. And I know that's a whole
21:20
another can of worms. And if you've been seeing a
21:23
bunch of that and maybe do you also use LM as a judge there?
21:27
Ohh yeah. OK, so this is, this is a great
21:30
actually so great to talk about some research we've been, we've
21:34
been dropping lately. So yes Elim is a judge can
21:40
totally be used to also evaluate the performance of retrieval.
21:43
So for folks who are listening to this or what is you know,
21:47
what is retrieval, how do you measure the performance of it?
21:50
Well, basically you know in retrieval you're retrieving some
21:54
sort of context. So if someone asked a question
21:56
about some kind of, let's say product, you're retrieving
22:00
relevant information about, let's just say a chat on your
22:02
docs type of applications, very common.
22:05
Someone's asking your, you know, product support documentation or
22:08
your customer support documentation questions.
22:11
And what happens is it retrieves relevant information from your,
22:17
you know, document corpus, and then it pulls, you know, just
22:22
relevant chunks and then uses those relevant chunks in the
22:25
context window to actually answer the question.
22:29
The most important thing there is that it's in this type of
22:33
retrieval. By the way, is super important
22:34
because it helps Ellms connect to private data.
22:39
Remember, LM's were not trained on every single company is
22:42
individual private documents. And so if you actually wanted to
22:45
answer questions on your own private data, then using
22:49
retrieval is kind of one of the best ways to do that.
22:52
Yeah. And so here what really ends up
22:55
being important is that did it retrieve the right document to
22:59
answer the question And that ends up being a a whole, you
23:05
know, there's a whole set of metrics that we can use to
23:08
actually evaluate that retrieval.
23:10
And recently actually we've been running a tonne of tests
23:14
measuring different LLM providers.
23:19
So we evaluated GPT 4, we did Anthropic, we charted some
23:23
results on Gemini, we did Mistral and we actually showed,
23:29
yeah, if you guys have been following Greg Cameron actually
23:32
dropped the first of these like needle in a haystack type of
23:36
test, you find that. Lost in the middle?
23:38
Yeah. And you know for those of you
23:42
haven't seen it it's basically an awesome way to think about it
23:45
which is you know it essentially cheques you know on one axis you
23:50
have how long is the context O the context can be 1K tokens all
23:57
the way to you know some of the smaller models it's like 32 Ki
24:01
think for some of the bigger ones we tested pretty pretty
24:06
significantly and we we double check exactly what 100.
24:09
And 20 Ki would imagine. I think that it feels like
24:13
anthropics goes all the way up to that.
24:15
Or maybe it's even more these days it's like 240.
24:18
They just said fuck it, we'll double it.
24:20
Yeah. So some of them we checked,
24:23
yeah, like definitely close to, yeah, 120 K and what what we did
24:28
was basically, so that's that's on one axis which is basically
24:32
just the context length and then on the other axis is basically
24:36
where in the context you put the information.
24:40
So you know, because some there's all these theories out
24:43
there like if you put it early on doesn't forget.
24:46
If you put it later down, does it not use it And so kind of
24:50
placement of the context within that context window to see it,
24:55
you can you actually find the needle in the haystack.
25:00
And the question we did for this was to a little context was the
25:05
question we asked the BLM was what's that?
25:09
So we did kind of like a key value pair.
25:11
The key was the city and the values and number.
25:15
So he said something like what's Rome special magic number and so
25:20
and then inside the context we put something like Rome special
25:23
magic number is like some 7 digit number like 4-2.
25:29
Yeah, like 1 to the roughly 6-7. And so that was a Rome special
25:32
magic number. And then later we asked we put
25:35
that somewhere. So we tested kind of all of the
25:38
dimensions of putting it at the very beginning of the document
25:41
for a very short context, putting it at the very end for a
25:45
very long context window, and all the combinations in between.
25:48
And then we asked it what was your own special magic number.
25:52
And so it would have to go through, look at the entire
25:54
context window and then answer the question.
25:56
And sometimes it couldn't find it and it's that unanswerable or
26:03
etcetera and sometimes it, you know it answered the question.
26:07
And what we did was we just ranked a lot of these L and
26:11
providers out there on how good was it at retrieval and GT4 was.
26:19
By and large, you're definitely the best out there.
26:24
Think of the small model providers.
26:26
We were definitely impressed with Mistral.
26:29
Yeah. Like the 32 K content it was.
26:31
It was pretty impressive, but there was some were you know, I
26:37
think we realised some of them were very, very if you change
26:43
the prompt a little bit then the results totally varied.
26:47
So you got totally different, varied responses based on just
26:52
like adding a sentence or adding two sentences.
26:55
And so as a user, you know as you're as we're coming back and
26:58
we're evaluating this. If you're using some of those
27:01
Elms where you have to be really careful about how you prompt it,
27:07
those prompt iterations can have a big difference in the actual
27:10
outcome of of that retrieval. So I'll share the links with
27:15
Dimitrios, maybe you can link it with the the podcast interview.
27:17
But definitely, I think going back to your original question
27:22
of like can you evaluate retrieval?
27:25
Absolutely. I think it's it's it's really
27:29
important to make sure that if you wanted to answer well and
27:33
not hallucinate on your private data, it's got to do well with
27:36
that retrieval part. Yeah.
27:38
So it's almost like you're evaluating in these two
27:41
different locations, right? You're evaluating the retrieval
27:44
when it comes out and if that is relevant to the question.
27:47
And then you're evaluating the output of the LLM once it's been
27:52
given that context, and if that output is relevant to the
27:56
question also. Exactly, exactly, exactly.
28:00
And if that outputs based on the context that was retrieved.
28:05
So first, there's that. So, yeah, yeah, totally so. 1st
28:10
is like is the retrieval that was retrieved even relevant to
28:14
the question asked? Then was the output based on the
28:17
retrieve tax? And then yeah it was the output
28:19
itself. Answering, you know,
28:22
correctness. I guess is is looking at.
28:24
Is it correct based on the information that was retrieved?
28:26
So it's there's levels to this. So I'm a little bit like I just
28:36
had a stroke of inspiration right now and I'm going to let
28:40
you have it because I love talking with you and I love the
28:44
way that you create products. But the next product that you
28:49
create in this space, I think I found the perfect name for it.
28:54
And. What is it?
28:55
Golden retriever, you can do so many things with that and it is
29:03
so perfect. It's golden, of course, you
29:05
know, like it's the golden metrics, it's the golden
29:09
retriever. And so you've got your, if you
29:13
do, if you create a product around that, I'll just, you
29:16
know, we'll talk later about the way that we can figure out this
29:21
patent and the Golden Retriever. I love that.
29:25
But I mean jokes aside, I know that you have been creating
29:28
products in this space. I saw a Phoenix and I would love
29:31
to know a little bit more about Phoenix and also as I mentioned
29:36
before we hit record one of my favourite things with all of the
29:42
products that you've been putting out from the get go.
29:45
I think when we talked in like 2021 of the first things that I
29:48
noticed with the Arise product was how well put together the UI
29:54
and the UX was for observing what was happening under the
29:59
hood. And it feels like Phoenix took
30:01
that ethos and went a little bit further.
30:04
You being a product person, can you like, break down how you
30:08
think about that and how you were able to get inside of what
30:11
metrics are useful and how can we present the metrics that are
30:14
useful in a way that people can really grab onto it?
30:19
Yeah, Well, first of all, thanks me chess that that's really
30:22
kind. And yeah, I'm super excited
30:25
about Phoenix. I think you got to give a big
30:28
shout out to the Phoenix team within within our eyes actually.
30:32
So Phoenix is actually for those of you don't know, it's our OS
30:37
product, it's got a tonne of support for LM evaluations and
30:43
LM observability. So if any of you guys are
30:46
looking to just try something, not have to send your data
30:49
outside, have it be lightweight, it's you know it's open source,
30:53
so do do do what you want with it.
30:57
I think the intention find Phoenix really was so there's a
31:00
couple different components in Phoenix that I think folks who
31:03
are trying to get observability on LMAO like.
31:07
This is. Skyler I Lead Machine learning
31:09
at Health Rhythms. If you.
31:11
Want to stay on top of everything happening in ML OPS?
31:14
Subscribe to this podcast Now, now now, now, now.
31:24
Now now first. Is if you are.
31:30
One of the things we just noticed very early on was that
31:34
these applications, many of them, have not just one call
31:39
they're making. There's many calls under the
31:41
hood. Like even in a simple chat bot
31:44
with retrieval, there's first the user's question, then you
31:48
have the you generate an embedding, then there's the
31:50
retriever, then there's the actual synthesis of the context,
31:53
and then there's a response generation.
31:55
So there's five or six different steps that have happened in
31:59
something that feels like 1 interaction between the product
32:03
and the user. And so the first thing was, well
32:07
if you have all these sub steps, some if something goes wrong,
32:10
something goes wrong within those you know five or six
32:14
different steps that's happened. Then being able to pinpoint
32:17
exactly what are the calls that are happening under the hood and
32:20
how do I get visibility is important.
32:22
And so with Phoenix, one of the most popular components of it is
32:27
you can see your full, you know you can see the full traces and
32:30
spans of your application. So kind of like the full stack
32:34
trace is how you can think about it.
32:35
So umm, you'll see the breakdown of each calls, and then which
32:40
calls took longer, which calls use the most tokens, and then
32:43
you can also evaluate at each step in the calls.
32:47
So kind of like we were just talking about where at the end
32:50
of the application, at the very end when he generated a
32:54
response, you can have a score of how well was the response.
32:58
But then if the response, let's say it was hallucinated or was
33:01
incorrect, then there's a step above.
33:04
You can go in and look at the individual span level evals,
33:08
look at well, how well did it retrieve.
33:11
And then within the retriever, you know how let's evaluate each
33:15
document that it retrieved and see if it was relevant or not.
33:18
So there's kind of a lot of thought put into first, how do I
33:23
breakdown the entire application stack and then see, you know,
33:27
evaluate and evaluate each step of that outcome.
33:30
And then the other part that's been, I'd say really a lot of
33:34
thought in is Phoenix does come with an evals library.
33:38
It's task evals first or you know, Elim application evals.
33:45
So it's it's definitely useful for folks who are actually
33:48
building the application and then we've just seen a bunch of
33:53
people kind of build these evals in production.
33:56
So it comes with kind of a lot of these best practises baked in
34:00
one of them that actually just just went viral on Twitter last
34:05
week. We dropped a big research start
34:07
on this, which is should you use score evals or classification
34:12
evals? I don't know if you caught
34:15
caught that one, but I. Saw your post blew up.
34:18
I definitely did, but I didn't. I don't know what the difference
34:21
is between the two, which yeah, might make me look like.
34:24
No, no, no, this, this. Let me just trace the changing
34:27
so fast early like there's I think we're all just trying to
34:30
learn and soak up as much as we can, but so score you gals
34:35
versus classification evals, score evals is basically you can
34:40
think about it as the output to a numeric value.
34:43
So let's just say we were asking LM to evaluate.
34:48
How frustrated is this response and, you know, rank it between
34:53
1:00 to 10:00? One being someone's really not
34:55
frustrated. 10 being someone is super frustrated.
34:59
Well, what would you expect? You would expect that, OK, if it
35:03
said one, it's super frustrated. If it's ten, it's not
35:05
frustrated, But then kind of somewhere, if it's something
35:08
like a five, it's kind of like, OK, maybe it's best regressive.
35:12
It sounds super frustrating, but it's like you know, like you you
35:14
kind of expect it to like kind of the numbers in the middle to
35:19
make sense. And you what we just realised as
35:24
we did this research was that the score value actually had no
35:31
clear connection to the actual, you know, you know, frustration
35:38
that this person like if it gave a number that basically wasn't
35:41
one or ten, then that score value really had no, no real
35:46
connection. Like another example which was
35:48
actually the one that we posted was we we gave it a paragraph
35:52
and we said, you know rank, how many spelling, how many words
35:57
have a spelling error in this document between 1:00 to 10:00.
36:00
If every word had a spelling error, give it A10.
36:03
If no words have a spelling error, give it a zero.
36:06
And then if it's kind of somewhere in the middle, like
36:09
20% of the word Tevis spelling error, you know, give it a
36:11
tooth. It's 80% give it an 8, you know.
36:14
Yeah. And what we saw was that in many
36:17
cases it would give a spelling, you know, score of like 10, but
36:21
in some cases only 80%. You know only 11% of the words
36:25
had a spelling your like. All of the words were off like
36:29
this. Might as well be in Dutch.
36:32
That value that it came back with meant nothing to it.
36:37
And so and and the reason this is important is like what we're
36:40
seeing is like there's a lot of these LM eval cookbooks that are
36:44
out there where people are recommending, you know,
36:46
basically set it up as a score. And we've actually been seeing
36:50
is don't do that, just do it as a class.
36:53
Just do binary stuff, you know. Or you can do multiclass, you
36:57
know, But like, tell it explicitly like frustrated, not
37:00
frustrated. Because if you try to assign a
37:02
score, that score just doesn't. It doesn't actually have any
37:09
meaning to to the corruption level.
37:12
So basically it's saying, hey, it's it has to be very clear.
37:16
It's frustrated, not frustrated or a little frustrated.
37:21
And you have to make it. It's not a sliding scale.
37:25
There's no Elms do not understand spectrums.
37:28
It's like a you know from all the tests we've done it is is
37:33
not going to give you a meaningful value on a spectrum
37:37
and then if you're. Basically, yeah.
37:38
If you're basing stuff downstream off of that score,
37:41
you're screwed. Exactly.
37:44
And so and this has been something that like has been a
37:47
lot of people. I I think it's kind of like as
37:51
we've been putting it out, people have been like I've seen
37:53
that like I'm you know like I was using score and then I was
37:56
like these values meant nothing to me.
37:58
So then I switched to classification and so there's a
38:03
whole set of research around this.
38:05
Probably still to be deep dived into here, but yeah, this is,
38:11
you know this is the kind of stuff that's in the Phoenix
38:14
Evals library is just best practises based off of what
38:17
we're seeing, what we're putting in production, what we're
38:20
helping folks actually launch. And so you get kind of these
38:24
best, you know, best in class I'd say templates that are
38:28
pretested around things like how to test for hallucination, how
38:30
to test for toxicity, how to test for correctness.
38:34
And then you can kind of go and we have people who then go off
38:38
and make their own custom evals. But it's a great place to kind
38:41
of have a framework that runs both in a notebook and also in a
38:46
pipeline very efficiently and is meant for for kind of you know,
38:53
so you can swap in and out of offline and online very easily.
38:58
Yeah, because the other piece that I was thinking about, all
39:01
these these evals that you give it, then it feels like you're
39:08
not going to get very far if you're not doing custom evals.
39:11
Have you seen that also? Totally.
39:14
I think there's a lot of folks who are building, you know,
39:17
maybe they start with something, but then they end up kind of
39:19
building their own what makes sense for their application
39:23
adding on to it. So I I definitely think at the
39:25
end of the day the nuance here is it's probably different than
39:29
the mental space is that that customization of that eval ends
39:33
up being really important to measuring what's what's
39:37
important to your A. So I I don't know that that's
39:40
one of the things I predict is we're gonna see a lot, a lot
39:43
more of this. And do you feel because one
39:47
thing that I've seen is like how when you put out these different
39:54
evaluation test sets, the next model is just trained on top of
39:58
them and so then they're obsolete?
40:01
And so it's going to be this game of cat and mouse in a way,
40:05
because the models are going to be able to eat up the test sets
40:08
as soon as they get put out for anyone to see.
40:12
Or is it going to be alright? I've got my evaluation test set
40:16
and I just keep it in house. I'm not going to let anybody
40:20
else see that so that I don't paint the model.
40:24
Yeah, I it's actually something I wonder about a lot too, is as
40:30
these new albums come out, are they really blind to the test
40:34
sets that they're actually then evaluating them on?
40:36
I think like the Gemini paper I thought did a really good job of
40:40
calling out. They actually built their own
40:43
data set that was blind and then tested on that data set and they
40:48
called that out explicitly, which I thought was really
40:51
important because you know, as as people are sharing the
40:55
results of like the next bestseller, etcetera.
40:57
Yeah, I think we're all wondering like did you just, you
41:01
know, like you know, did did have access to that training
41:04
data set. So I I wondered that all the
41:07
time too well. It's pretty clear these days
41:10
that as I did not coin this term, but I like it and I will
41:14
say it a lot. Benchmarks are bullshit.
41:18
And so all these benchmarks on hugging face or on Twitter that
41:23
you'll see like, oh this is so this just came out.
41:26
It blew everything else out of the water by whatever 10 times
41:29
or you make up a number there. I don't even consider that to be
41:34
valuable anymore. It's it's really like what you
41:37
were saying where these things. I know you actually went and you
41:42
did a rigorous study on it, but it's so funny because we are,
41:48
the rest of us are just going off of vibes and we're seeing
41:51
ohh yeah, this is not really working.
41:54
This is not doing what I thought it was going to do.
41:56
And so if I use a different model, does it.
41:59
And then you try that and you go, Oh yeah, OK cool, This is
42:01
better. This is easier to prompt or this
42:03
prompt is much easier to control, whatever it may be.
42:07
And so I appreciate that you did a whole rigorous study on it.
42:10
I also, I'm conscious of time. I want to make sure that, like
42:16
with that you all are doing like paper studies, right?
42:19
And you're meeting every once in a while.
42:20
I think that's awesome. I know that you've got a tonne
42:23
of smart people around and so I I can imagine you're getting all
42:26
kinds of cool ideas from gathering and doing these paper
42:29
studies. I would encourage others to go
42:31
and hang out and do those. We'll put a link to the next one
42:36
in the show notes so that in case anyone wants to join you
42:39
and be able to ping ideas off of you, that's great.
42:42
I still would love to hear how did you come up with the
42:45
visualisations. That's the coolest pieces.
42:48
I think you didn't get into that part and I wanna get get to it
42:52
before we go. I I mean, I just.
42:54
Gotta say the, the the team team's amazing.
42:59
So the and I think the, you know, trying to find a way to
43:04
bubble up, you know, at least in the LM space, one of the cool
43:08
things, you know, maybe you've seen some of the demos of it,
43:10
but like especially with the retrieval, there's a lot in the
43:14
embedding space that's helpful to visualise.
43:17
So how far away is the prompt that you have from the context
43:22
that was retrieved? And if you're missing any
43:25
context and it's super far away or it's, you know, it's like
43:29
reaching to find anything that's relevant to, you know, those are
43:33
all really cool visualisations that you could actually surface
43:37
and kind of help people see a little bit of, OK, here's my
43:40
data, here's what it thinks things are connected to.
43:44
So yeah, again, you know, check out Phoenix.
43:47
Love. Love all the notes on on the UI.
43:51
Yeah, actually. It it reminds me that I think
43:55
one of the first people I heard the word embeddings from was you
43:59
on like that first meet up that you came on back in like June
44:03
2020 around then because you were already thinking about
44:06
them. I think you were thinking about
44:08
them from recommender systems and that angle and then has it.
44:14
How has that changed now? No.
44:18
Great, Great question. Well, I think, I think
44:21
embeddings are just so powerful and yeah, I'm I'm so glad that
44:25
we're, you know, we're all talking about them and using
44:28
them and observability because there's it's it's super powerful
44:32
even in the LM space. I think in the past there is you
44:36
know folks used them like you mentioned recommendation
44:38
systems, image models. But the LM space, I mean the
44:41
core basis of retrieval is based off of those you know Word and
44:46
you know the the embeddings itself.
44:49
In doing that vector similarity search to fetch the nearest
44:52
embeddings. So I think the the use case is
44:57
really really strong in RAG for alms the you know and and and so
45:06
because it's such a core component it it's also something
45:10
that it is important to now you know and just like when we were
45:14
going back if the retrievals off then the response is just not
45:17
going to be good and so you do really good way to verify that
45:20
what was retrieved was relevant and if there's any shift in
45:25
again going back to all of this is now textual.
45:28
If the prompts are changing, what your users are asking are
45:31
different, or the response of the LMS are different, these are
45:34
all things that you can you can actually measure using
45:36
embeddings and embedding, drift and things like that.
45:40
So I I think there's just maybe more use cases now than ever to
45:45
to dig into embeddings. Yeah, it has to be.
45:48
Treated as a first class citizen 100% these days.
45:50
Exactly. That's a really good point.
45:52
And I do. I saw a recent paper speaking
45:55
papers here from Shreya Shankar. Did you see that?
46:00
Splat or split spade, I think is what it is, talking about the
46:05
prompt deltas and evaluating via the prompt delta.
46:09
Like, you have your prompt templates, but then you're
46:12
evaluating the prompt deltas and it's like, wow, there's so much
46:17
creativity in this space and the ability to look at how can we
46:23
evaluate things differently than we are right now and see if we
46:27
can get a better outcome. Yeah, I I I.
46:31
Think I still need to dig in to to spade specifically, but yeah.
46:36
The UM. I I mean I I think the the
46:42
amount that the space is moving is just so fast right now.
46:44
It's so exciting and it is very cool, the.
46:48
The one thing maybe? I'll just you know, there's
46:51
maybe two things I just wanted to like at least drop like my
46:54
quick hot takes or or notes on let's do it this is.
46:58
Great. This is what we're going to chop
46:59
and put at the beginning of the episodes.
47:01
Yeah, exactly Right. So.
47:03
I think there's, you know I always hear this from.
47:07
I don't know. I I just see it in the in in
47:10
discussions. But I see a lot of people
47:12
talking about fine tuning like really early on, like their
47:19
applications aren't even deployed and they're like Oh
47:21
well, our use cases. Eventually we're going to go
47:22
back and fine tune and umm, I you know, I get asked from folks
47:28
like Hibernate. Does that make sense?
47:31
Is like a step in troubleshooting an LLM
47:33
application. And I think one of the reasons I
47:37
get that question a lot is if you just think back to, you
47:41
know, a lot of the AI teams are now, yeah, they've worked on
47:45
traditional now and they're shifting now to lens, but that's
47:49
something we're all very used to and very familiar with.
47:52
We're used to training models on data and then deploying those
47:57
models and fine tuning is feels very familiar, right.
48:00
You grab data points that it doesn't work on, you fine tune
48:04
it and that's how you improve the performance of your model.
48:08
But in this space, fine tuning feels like you're jumping to
48:14
like level 100 when sometimes a lot of this could be, you know,
48:20
like I was telling you in the rag case, change the prompt a
48:22
bit and you get vastly different responses.
48:25
And so it's like almost the thing that we're like geared
48:29
towards to do, which is like, oh, that makes sense, We're
48:32
going to trainings now, fine tuning and we're all used to
48:35
that paradigm. But I I think in this space
48:40
let's. Start with like the lowest.
48:41
Hanging fruit and see how that improves.
48:44
Because I I I think you know and and Dredge Carpathia actually
48:51
drew those like really awesome image of like you know level of
48:55
effort versus the ROI. You know, kind of kind of of
48:57
that effort and prompt engineering rack like there's so
49:02
many things you could do to improve the LM performance
49:04
before you jump into fine tuning or like training your own umm.
49:08
So it's just I think it's important to like start with
49:12
something that could have the highest RY you are preaching to
49:16
the. Choir and I laugh because I was
49:18
like talking about how fine tuning to me feels like when all
49:23
else fails, you'll throw some fine tuning at it and it's like,
49:26
yeah, that's. What you need to you need.
49:28
To look at it as like the escape hatch, almost not as Step 2.
49:33
It should be what you go to when you can't get anything else to
49:37
work. And try rigorously to get
49:41
everything else to work because it is exactly like you said.
49:45
It is so much easier to just tweak the prompt then fine tune
49:51
it. And I didn't connect the dots on
49:54
how similar the two are. And like, if we're coming from
49:58
the traditional ML space, then it's easier to jump there and be
50:01
like, Oh, well, that's just because we need to fine tune it
50:03
and then it'll do what we want it to do.
50:07
Yeah, totally. I I think there's just something
50:11
very natural feeling about, OK, you know, training is now fine
50:14
tuning, but it's, you know, I think it's one of those changes
50:18
we all have to just just adapt with with the the space change.
50:22
Yeah, simulate. Yeah, simulation.
50:25
Excellent. And then my.
50:27
My other. Hot take?
50:28
I guess yes. That's totally a a hot take.
50:31
But I think sometimes I hear a lot of this, you know?
50:40
Yeah, maybe I hear it less now than than I was in the
50:42
beginning. So I I hear a lot of like well,
50:48
it's kind of continuation of the fine tuning.
50:50
Well, if I pick an open source model, I can go in and fine tune
50:53
it more, and I can you know, I can then go and you know, modify
50:59
it for my use case because I I know I have access to the model
51:04
weights and do that. Umm, I think that I hear a lot
51:10
of folks asking, well, does choosing an open source model
51:13
versus a private model end up slowing down product development
51:16
or like well, what kind of what's kind of the pros and the
51:19
cons of one versus the other? I think I was hearing a lot more
51:24
of like, you know, almost resistance for some of, you
51:29
know, just the private models of the beginning and a lot more of
51:31
the, you know, open source. I am so in this horse I gotta
51:38
say I'm yeah all for the open source community like I think
51:42
I'm I'm also all for you know whatever.
51:45
Well I'm just makes your application the most successful
51:48
it can be. So pick pick the one that like
51:53
gets you the performance and the outcomes that you need.
51:57
I don't think that like you know some people make a bet on the
52:02
open source because they're like ohh later I can go back and find
52:04
to you or it's better and etcetera.
52:06
But it's again are how many of those folks are really going to
52:10
actually fine tune? And for what I've been seeing
52:15
out out in the wild, starting with the opening eye or GPT 4
52:20
has just been helping most people get to kind of the
52:24
outcome that they need their application to get to.
52:27
Yeah. And so again, I think I just
52:30
come back to like there's, you know, all for the open source
52:33
community, all for our, you know, just getting your
52:37
application to actually work as as good as it needs to to work.
52:40
But start with start with like what do you need for the
52:43
application? And less of like I think the
52:46
How's this going to scale? Yeah, like I.
52:49
Had conversation back. In the day where you're like,
52:51
oh, we're going to need to use liberties for this and you're
52:54
like, wait a minute, if we have no users, Are you sure you're.
52:59
I know you're planning for the future and this is great for the
53:01
tech debt, but we might want to just get something up on stream
53:05
that before we do anything. Totally, totally, totally And I.
53:09
Think that that's like what I keep coming back to is like the
53:13
more of these, you know someone in the MLS space.
53:15
We want to get more of these deployed actually in the real
53:18
world, get the application to add value to the organisation,
53:23
show the ROI and I think that that's really important to to
53:29
the success of these albums and companies actually and the
53:34
other. Piece to this that I find
53:36
fascinating was something that Laszlo.
53:39
Said probably. Like two years ago, and Laszlo's
53:43
infamous person in the community, for those who do not
53:46
know, in the community slack. And he was talking about how you
53:49
need to get something in production as fast as possible.
53:53
Because then you'll find where all of the bottlenecks are,
53:58
you'll find where everything is messing up, and unless you get
54:03
into production, you don't necessarily know that.
54:05
So each day or each minute that you're not in production, you're
54:09
not finding all of these problems.
54:11
And if you can use a model to make your life easier and get
54:17
you into production faster, then you're going to start seeing, Oh
54:20
well, maybe it's the prompts or or maybe it's, you know,
54:23
whatever the case may be, where you're falling behind and you're
54:27
making mistakes or the system isn't designed properly.
54:32
Yeah, absolutely. So I think maybe as we wrap up
54:37
the podcast that that's really is get stuff out as fast as you
54:41
can, you know evaluate the outcomes.
54:43
I think that's you know LLM evals is something that I think
54:47
is pretty pretty got a lot of momentum around it in in the in
54:52
folks who are deploying in the community.
54:54
So evaluations is important. And then I think knowing how to
55:00
set up the right evals, knowing how to, you know, benchmark your
55:04
own evals, customise it, what types of eval score versus
55:09
classification, there's just so much nuance and that whole eval
55:12
space. And so as we continue to drop
55:14
more research or share more stuff we're learning, we'll
55:17
share with the community. Excellent partner.
55:19
It's. Been absolutely fascinating
55:21
having you on as always. I really appreciate it and look
55:24
forward to having you back. Thanks.
55:27
Dimitrios, thanks and thanks envelops community.
55:33
Yeah. Hey everyone.
55:34
My. Name is Aparna, founder of
55:35
Verizon, and the best way to stay up to date with Emma Lops
55:38
is by subscribing to this podcast.