Latest / Agentic Conversations (formally mlops.community) / Evaluating and Integrating ML Models // Morgan McGuire and Anish Shah // #213

Transcript

Evaluating and Integrating ML Models // Morgan McGuire and Anish Shah // #213 · Agentic Conversations (formally mlops.community)

0:00
Hold up. Before we get into this next
0:02
episode, I want to tell you about our virtual conference
0:06
that's coming up on February 15th and February 22nd.
0:10
We did it two Thursdays in a row this year because we wanted to
0:15
make sure that the maximum amount of people could come for
0:19
each day since the lineup is just looking absolutely
0:24
incredible. As you know, we do.
0:27
Let me name a few of the guests that we've got coming because it
0:32
is worth talking about. If you got Jason Louie, we've
0:34
got Shreya Shankar, we've got Drove who is product Applied AI
0:39
at Uber. We've got Cameron Rook Wolf
0:42
who's got an incredible podcast and he's Director of AI at Rebuy
0:45
Engine. We've got Lauren Lockridge who
0:49
is working at Google also doing some product stuff.
0:53
Ohh, why is there so many product people here?
0:55
Funny you should ask that because we've got a whole AI
0:59
product owners track along with an engineering track.
1:03
And then as we like to, we've got some hands on workshops too.
1:07
Let me just tell you some of these other names just for a
1:10
moment, you know, because we've got them coming and it is really
1:14
cool. I haven't named any of the
1:15
keynotes yet either, by the way. Go and check them out on your
1:20
own if you want. Just go to home dot ML OPS dot
1:22
community and you'll see what we've got.
1:24
Tungi, who's the lead researcher on the deep speed project at
1:28
Microsoft. We've got Holden, who is the
1:31
open source engineer at Netflix. We've got Chai, who's leading
1:36
the AI platform at Uber. You may have heard of it.
1:40
It's called Michelangelo. Ohh my gosh.
1:42
We've got Fazan, whose product manager at LinkedIn, Jerry
1:46
Lewis, who created good old Lama Index.
1:50
Well, he's coming. We've got Matt Sharp, friend of
1:52
the pod, Shreya Rajpal, the creator and CEO of Guardrails.
1:58
Oh my gosh, the list goes on. They're 70 plus people.
2:02
That will be with us at this conference, so I hope to see you
2:08
there. And now, let's get into this
2:10
podcast. I'm Morgan.
2:13
I lead the Growth ML team at Weights and Biases, and I drink
2:17
my coffee as an Americana with a little bit of cold milk.
2:22
Hi I'm Anish. I work as a growth machine
2:24
learning engineer at Weights and Biases with Morgan and I prefer
2:29
tea. And I like to drink tea such
2:31
like oolong or chamomile. Welcome back to the MLB's
2:36
Community Podcast. I am your host Dimitri Ose and
2:41
you are in for a treat. Today we've got Morgan and Anish
2:45
from Weights and Biases joining us to talk all about what
2:49
they've been up to, how they're doing things.
2:52
You know, their exact title is Around growth, but I think
2:58
they're more like Debril. We talked about so many
3:03
different pieces going from all the cool products that they've
3:07
been seeing being put out, all the different fraud objects that
3:12
people who are using weights and biases are putting out.
3:16
And as you will hear in this conversation, make sure if
3:21
you're using weights and biases to give them due credit, it's on
3:24
their OK, ours. So let's not let these folks
3:27
down. Come on now, cite them in your
3:30
papers. And at the end of the
3:34
conversation, we got into such a cool musing around evaluation as
3:42
a service round Hallucination mitigation as a service.
3:47
And hopefully you heard by now I've got a little bit of a song
3:55
that we dropped for the last AI in production conference.
4:01
This song is all about them. Prompt templates and some of the
4:06
lyrics are evaluate me, make the AI, make sure the AI it is said
4:13
no. What do I say that I don't even
4:15
know anymore. But we dropped this song.
4:17
I've been threatening for ages to actually create something and
4:22
put it on Spotify. I think it's been a good three
4:26
years now. Back in the day, I wanted to
4:28
create a song all about go in real time or go into real time,
4:32
but never got around to it. Today I happily can say that in
4:39
my time is best spent creating a funny song for you all, and I
4:45
hope it's one of those songs. It's a banger and you'll listen
4:49
to it whether or not you're into AI.
4:51
But wow, if you are a nerd like myself and you can chuckle at
4:59
the lyrics, being my prompts are my special sauce and this song
5:04
is for you. We'll have a little play right
5:06
now. I'll play it for you so that you
5:09
can hear it, You can give me your feedback and let's get into
5:13
the conversation with Anisha Morgan.
5:16
Huge shout out to weights and biases for continuing to sponsor
5:21
the ML OPS community. It is a blast having the
5:24
knowledge that they bring to this community basketball.
6:03
All right, let's listen to this song.
6:05
And of course, if you like it, you know what to do.
6:08
Just stream it on Spotify on repeat.
6:10
I get point OO O2 cents for every stream.
6:14
So I'm trying to fund my next startup with these Christians.
6:18
Let's go, dude. I gotta start with this, Morgan.
6:30
What's this breakfast radio show that she hosted in college?
6:37
Yeah, good question. So in yeah.
6:41
Turn third year university. I don't know maybe something a
6:43
friend we saw the you know radio.
6:49
College radio had this opening for breakfast radio show once a
6:53
week and we're like oh we could do that.
6:55
Let's try that. I mean we just signed up.
6:57
Uh and so there was like three of us.
7:00
Yeah. Can't be that hard.
7:01
He was he was the the talker. And then we pulled in a third
7:04
friend. I I was like I'm yeah,
7:06
definitely quieter. So I was kind of more on the the
7:08
soundboard and like hitting the play button.
7:11
Um. But yeah it was a lot of fun.
7:13
Did it for about 6 weeks in my university.
7:19
Does this like charity, like fundraising week where like a
7:21
lot of you know, the different societies do different events
7:24
and it's a lot of fun? Um, there's a fair bit of
7:27
drinking involved. So we had as a coming into a
7:30
breakfast show at like half six in the morning after a night
7:33
before, we had pretty loose show.
7:37
I would say like nothing, nothing bad.
7:39
But our, I think our, the editor at that point was kind of sick
7:41
of us. So that was the end of our radio
7:44
career. But umm, I still have the
7:45
recordings. I'm afraid to listen to them,
7:47
but I might listen to them someday.
7:49
And were you playing music or it was all talk show?
7:52
Uh, yeah. No music.
7:53
Like music. Yeah, yeah.
7:55
And. And as cheesy as we could make
7:58
it. Like all the the 90s, early
8:00
early 2000s. Like, yeah, cheesy hits.
8:03
What were some of the bangers you were playing?
8:06
There's definitely a lot of Spice Girls.
8:08
Boys own Britney. All, yeah, all the girl brands,
8:11
all the boy bands. Yeah, get to get you going in
8:13
the morning. And this was not in the 90s or
8:17
early 2000s, not when they. Were cold.
8:20
Yeah. 2007, I would say, yeah. All right after their prime.
8:26
Yes, definitely. And Anish, man, tell me a little
8:30
bit about chemistry and what happens you you were studying
8:34
that and then you got sucked into the world of machine
8:36
learning. Yeah, so initially when I
8:40
started doing chemistry stuff, like I love the topics and I
8:44
loved Peckham as a concept too. So I joined like a couple of Got
8:48
lab or two. And then I realised that the
8:51
actual physical process of doing the science wasn't actually that
8:55
fun, like pipetting things or like going in and like putting
8:59
in like lasers or like physical hardware kind of stuff.
9:03
Wasn't really my type of a science, I realise, but we use
9:07
this like really awful graphical programming language.
9:10
I forget the name of it at this point to go and like analyse all
9:13
these different like experiments that we were doing.
9:16
And I kind of really, really fell in love with that side of
9:18
the world of like connecting nodes that would then like give
9:22
you insight about like ohh like from these three different like
9:26
lines of of like sensors for instance.
9:30
It goes into this specific database and then you can look
9:32
up graphically apply it to like some pre-made algorithm that
9:36
they had like a symbol for. And so I was like, oh, like,
9:38
this is cool. I can actually understand what's
9:40
going on. And I realised, like, I this is
9:42
the side of the world I like a lot more.
9:44
And so I got sucked into a whole bunch of like data stuff and
9:49
programming around that and then naturally just was like OK time
9:53
to like give up chemistry. It's here to like do data
9:56
programming based things. And luckily it was at the same
9:59
time that my university started to like really dive into more
10:03
data intensive classes, so like data engineering stuff and
10:06
machine learning stuff. So it was like all the stars
10:09
were lining up and I was like, if I don't do this now, I'm like
10:12
miss an opportunity, I felt like.
10:14
Was meant to be and now you both for context are working at
10:20
weights and biases as machine learning engineers and you're
10:23
working with different customers that weights and biases has.
10:27
Can you break down exactly what is the job?
10:31
Because you're machine learning engineers, but it's a little bit
10:33
of internal and external facing, right?
10:37
Yeah, yeah, that's a good way to to put it.
10:40
So the the roots of the team and our growth team started with you
10:45
know, doing events like locally in SF at the time when it was a
10:48
one person team or UM leader, Lavania was running those.
10:53
But also writing content for the blog and building integrations
10:57
and you know and like examples. And that's really been kind of
11:02
like we've still kind of stuck to like those core pillars, um,
11:06
to drive people into weights and biases.
11:10
More recently I think, yeah, over a year and a half ago now
11:13
we started a courses programme which is like really taking off
11:16
and I would say like that's our, our 4th one now.
11:19
Um. And that is been super fun and
11:23
you know the the primary goal is to just to deliver high quality
11:27
education. You know, if it's your cases
11:30
waiting rises in the process like all well and good.
11:32
But I'm definitely trying to deliver you know the education
11:36
and like the value to to the folks first, Um.
11:40
But yeah, then like also internally um helping out across
11:44
the sales side of the business. And you know, if you need to
11:48
like jump on any user calls, if there's any like debugging um,
11:51
that our support team can't do, although I would say that's less
11:55
less of it now our support team are like legit and like grown,
11:59
you know, a lot in the last year or two.
12:01
Um, yeah. And then just kind of like I
12:04
think we're here because you know we're we're you know
12:08
constantly focused on user growth and like understanding
12:11
where where the field is going, um, we're quite active and like
12:14
aware of what's going on, on you know Twitter and keeping on top
12:18
of like new models, new architectures.
12:19
So bringing that knowledge internally is also important as
12:22
well to like you know let the product team know that like
12:25
yeah, hey like large multimodal models are like coming and we
12:29
need we need something for like text, image or text to video,
12:32
you know not just like LMS, UM this year that those kind of
12:35
insights I think I see our team Duma Um initially miss anything.
12:41
Yeah, I think you hit the nail on the head.
12:42
I think like, yeah, the big part of like I see myself doing is
12:45
interacting with the community a lot too.
12:47
So like like Morgan mentioned is the community builds really cool
12:50
stuff with or without weights and biases.
12:52
And we as an MLA's platform really have to make sure that
12:55
we're on top of our game to understand like ohh, like these
12:58
are actual projects people are building and using in the real
13:01
world. So let's make sure that we share
13:03
that knowledge with the world in, like, really useful ways.
13:07
Yeah. I feel like in a different
13:09
company you would be called debt, but at this one you're
13:14
you're kind of called something else, even though you're doing
13:17
kind of the same thing. And so it's cool to see that
13:20
you're getting out there and getting that experience with
13:23
what people are building in the community.
13:25
Can you name off some of the stuff that you've been seeing
13:28
that you're getting excited about?
13:30
I would say like the most recent thing I think that's really
13:33
interesting is the Allen Institute folks.
13:35
I believe they released OL MO. And I just think like I like
13:40
that project quite a lot because I think that one thing that at
13:42
least weights and biases does well is this whole
13:44
reproducibility piece. And I think that like the core
13:47
tenets of this specific framework is to be like low
13:50
carbon emissions, really reproducible, extremely
13:53
shareable. And they also use weights and
13:55
biases to like kind of use share all that out to the world too.
13:58
So it's kind of nice to see that like that frameworks exist that
14:03
will actually be moving forward. The science of like a lot of
14:07
this large language model stuff, which I think the open source
14:10
models are doing a great job of and making sure that people
14:12
aren't like repeating work or repeating frameworks such that
14:17
they all diverge like there's no like standardisation around it.
14:21
So I think all M is definitely something I think is really
14:23
cool. Yeah I'd I'd I'd just came out I
14:27
think yesterday an archive The clean clean real benchmark by
14:34
cost cost of one who's in hugging face now but it was
14:37
actually interned in rights and misses uh over a year ago and
14:41
that's a really nice project I think because we see a huge
14:45
amount of researchers use weights on buses sometimes they
14:50
they cite as sometimes they they don't.
14:52
We love if more people have cited, as we can send you the
14:55
citation if you if you're curious.
14:56
Um, but he what was really interesting.
14:59
Is that an OKR? It might be the thing.
15:01
Yeah, this year. And so, yes, everyone out there
15:05
cite these folks. Make sure that you do that so
15:08
that Morgan and Anish can crush their OK R's.
15:11
Exactly, yeah. Keep wait before we get into
15:14
this. What is clean RL?
15:15
I didn't see that one. Yeah, so it's been a project
15:19
Costa has been running for a while.
15:21
Like the goal is to have really. Yeah.
15:25
Like clean implementations and evaluations of like standard
15:31
oral UM or like models and and benchmarks.
15:35
And so the nice thing with the library is like similar to, like
15:40
the alien AI, where they're like going for like extreme, UM, kind
15:45
of like transparency and reproducibility.
15:47
They've also like released everything, Um, they're less
15:51
focused on, you know, releasing like model weights or like, you
15:54
know, it's not the end of the project, It's more focused on
15:56
evaluation. Um, but they have it all
16:00
completely instrumented in with some noises.
16:03
They're weights and biases. Like, you know, project is
16:05
completely open. So you can like go and poke
16:08
around and see how different evaluations doing.
16:10
And like I was just skimming the paper yesterday and like one of
16:13
the things that emphasises like, you know, we're also making
16:16
available like the the actual, like, you know, lots of curves
16:19
and things like that. So not just like the final
16:21
metric, but like how the model actually got there.
16:24
Um, so it was like a pretty cool project to see.
16:28
And um, like a little bit, I think.
16:30
Yeah, given the resurgence of interest in oral with them, you
16:35
know all the like PO and DPO going on with the, the LMS,
16:41
yeah, it was a nice, nice project to finally come out.
16:43
It was a big collaboration as well.
16:44
Like if you look at the paper, there's maybe like I think 30 or
16:48
40 authors there. So it's felt like a really nice
16:51
community effort from like the the oral community as well.
16:56
Yeah. You know, that's a big one when
16:58
Yeah, yeah, the authors takes up a full page itself, then you're
17:04
in for a treat. Yeah, exactly.
17:06
Yeah. So talk to me a little bit about
17:10
how you all are thinking about MLM's internally that weights
17:15
and biases. Because I I know that before we
17:19
hit record, there was some chatter that we had around.
17:23
You are seeing what's out there. You're getting to experience the
17:27
cutting edge and cool things that are happening, but you're
17:30
also getting to potentially affect product when it comes to
17:34
weights and biases. Or just make sure that weights
17:37
and biases itself is ahead of the curve when the new trends
17:43
are coming out. O How do you think about
17:46
currently just LM's internally? Yeah.
17:50
So I think that there's a variety of places where I think
17:54
LM's work really well. I think like for instance,
17:56
Morgan worked on like a quick win using LM to help clean up
18:00
some dirty data. I think that it's like
18:03
understanding where LM can actually be effective and at the
18:07
very least create like a nice launching point for people to go
18:10
ahead and like access knowledge or start work in a way that just
18:16
makes their life easier. So like from our side like
18:19
things such as creating like better translations for certain
18:24
documentation or creating like early PR with like some pre
18:29
written code in the right locations are like places where
18:32
I think LM's definitely help out quite a lot.
18:35
And I think like the one project that we love to talk about a lot
18:37
also is a wand bot which is our like internal support bot in a
18:42
way that goes ahead and like interacts with a lot of
18:45
different data sources that way that are public facing weights
18:48
and biases style documentation such that when anyone comes in
18:53
and asks like really proprietary knowledge about weights and
18:56
biases that unfortunately GT is not trained on yet and go ahead
19:00
and do that being like this like really complicated rag
19:03
pipelines. And so if if I'm understanding
19:06
this correctly, it's more about how can we help log in elims or
19:13
let the user sync their LLM usage to what we are creating as
19:20
opposed to let's put an LLM under the hood and people now
19:25
have text to SQL or something that maybe they don't really
19:28
need or they're not going to be using as much unless it's what
19:32
you're talking about with the one, but where?
19:34
Yeah, this is a perfect use case for using an LLM.
19:38
Is that a correct assumption or are you also thinking about it
19:42
as like let's plug in an LLM to the product and see how we can
19:46
do that. Yes.
19:46
So I think that it's kind of a mixture of of both those.
19:50
I think that one is feels more experimental than the other.
19:54
So like from our perspective I think it's we want to make sure
19:57
that the large language model usage that we do still ensures
20:00
that it's. Effective I guess is like the
20:03
biggest thing and so like that incorporates a lot of different
20:07
um kind of like I guess like we want to make sure that that you
20:12
want like there's a multiple pieces to make effective large
20:16
language model based applications.
20:17
I think part of it is like ensuring that the people who are
20:20
using it get to interact with it in the way that makes the most
20:23
sense for them. And so sometimes that may be as
20:25
simple as like having LMB the driving layer at the very bottom
20:29
that does a majority of the work.
20:31
And other times it's as simple as laying the LM just be a
20:33
single piece in the workflow that the user can interact with
20:37
and like turn off or on depending on how they're built,
20:40
like how they how that specific workflow is being used for
20:45
whatever they're trying to do. So be it like the docs
20:48
translation or the imputing or cleaning dirty data.
20:52
I think a lot of people are asking themselves and like I
20:54
definitely am. Is like how many killer like you
21:00
know product use cases. There are 4 alms like.
21:04
There's some obviously, like with like incredible product
21:07
market fit, like Compile and like Cursor and Alienist and a
21:11
few companies I know I think like even up in the legal space
21:14
are doing some like amazing work.
21:15
But it kind of feels like a lot of the value of ALMS is at least
21:20
we're like what when I like look at our product and and the way
21:24
services business is, yeah, more in like back office operations
21:27
just making things a little bit more efficient, a little bit
21:29
more productive, allowing people to scale more, um at least
21:34
that's like in the shorter term that's like where all the the
21:35
quicker wins are Like we have bounced ideas about you know
21:40
what would be like, where would be fun and interesting to add it
21:43
to the product. Um.
21:46
And there's like some interesting use cases there, but
21:48
I don't think any of them are revolutionary.
21:52
Um, I shouldn't, I guess I shouldn't speak on perhaps on
21:55
behalf of our like product team. Maybe they're like cooking up
21:57
something wild. But but I think in general like
22:00
like you see it in a lot of products that try and like
22:02
shoehorn and LM in just because they're trying to ride the hype
22:06
like it's UM, yeah, it doesn't work in a lot of cases like
22:10
people haven't found the right way to like really nail it if I
22:14
find. Yeah, and I feel like it's less
22:18
of that now because people have had experiences where they're
22:22
like, oh, we're just gonna plug AI into our product and now it's
22:25
revolutionary. And then when you actually use
22:29
it, you realise that it's still a chat bot and it still doesn't
22:32
understand me half the time. Or it hallucinates half the
22:35
time. And it's actually not the best
22:36
experience. So I would rather have a
22:40
different user experience. And I like that you all are
22:43
taking a bit more of an approach of like, well, we have all these
22:49
hammers. Do we need to go around looking
22:51
for nails? Maybe not.
22:54
Yeah, yeah, exactly. The the other I think impetus in
22:58
in our team to help build out these tools is is just to build
23:03
a muscle like like we've got great uh machine learning team
23:06
like everyone like has backgrounds in like ML from
23:11
academia and industry's been doing it for years Um but like
23:15
this is new right. And so I wanted to make sure
23:18
that the team are, um, yeah, they were like we build up that
23:22
muscle through like building like real use cases and not just
23:24
like toy examples um on the latest like open source model,
23:28
but like ultimately, you know, we care about solving a task and
23:33
beat, Yeah, like translate in the docks or yeah, cleaning this
23:36
little bit of data or whatever. So you know you want to use the
23:39
best models and the best techniques for that as opposed
23:42
to doing them like a fun fine tune of a of a Lama 2 model that
23:46
you know is kind of garbage compared to like 24 at the end
23:49
of the day. Yeah, it's not actually that.
23:53
Valuable you compare it to Level 3 for sure.
23:56
Yeah. Yeah.
23:57
Let's see, let's see what happens in a few months or a
24:00
year. But I do like that where it's
24:03
like, hey, let's look at the value that we can bring.
24:06
Not just like, is this gonna get a bit of attention on Twitter
24:09
for a while. Hmm.
24:11
Yeah. Yeah, exactly.
24:12
Yeah. I think I think it kind of you
24:15
in terms like our work then and like on the growth side of
24:17
things, it builds credibility in the long run and to be able to
24:23
point to like actual things you you've put into production.
24:26
Um. So yeah, it it kind of like
24:28
works for multiple reasons for us I think.
24:33
And what about this data cleanup that you're using that use case?
24:37
Can you make more details on what that looks like?
24:41
Yeah, that that's just a tiny one.
24:43
So I I shipped a small tool and then I know someone in data or
24:49
data team have also shipped something similar.
24:51
Um, super straightforward. But we have some user facing
24:55
forms where they can the user can put in their country they're
25:00
in in the back end our like CRM has a standardised list of
25:03
countries and so we just need to like translate us A to United
25:06
States of America or like Antarctica to unknown like
25:10
because not everyone, you know, they put in like lots of
25:13
different stuff in there. Um yeah.
25:15
So it's like complete overkill for for GBT 4 but we do use it
25:20
anyways. But just because the volume of
25:21
requests is like pretty manageable, it's still going to
25:24
be pretty cheap and you know again just going back to like
25:26
reliability and performance. Um.
25:29
And it's. I mean, yeah, like I joke, it's
25:32
the whole, like we were promised a GI and we got like country
25:35
standardisation is like what I took about.
25:38
Uh, But like, it works and it works much better than, you
25:41
know, big complicated regex or a person having to manually fix
25:45
them. So we ship it, we do it, we move
25:47
on. No there there is a topic that I
25:49
think is top of mind for so many people and it came up recently
25:54
when I was talking to your coworker Alex.
25:57
It also has been on my mind for months because we did an
26:04
evaluation survey in the Milos community and we got a tonne of
26:09
responses. We also are doing another one
26:13
right now and getting all kinds of great responses where I wrote
26:17
a blog post on the first one. All of that data is open source,
26:20
so if anybody wants to see it, they can.
26:23
All of the new responses, we're going to be getting an
26:29
incredible report made for it. I want to know how you, all,
26:33
being machine learning engineers out there in the trenches, are
26:37
thinking about evaluation because it feels to me like it
26:41
is one of those. Like it's like the dark arts of
26:45
evaluation still, and it depends on what your use case is.
26:49
It also a lot of times is really not clear and you still can't
26:54
trust in LM, but how are you all thinking about it?
26:59
Yeah. So I think the hard part is that
27:02
like as we've all probably heard of earlier, is like a lot of the
27:06
evaluation data that we're used for like standard benchmarks
27:09
have like been leaked in the training data itself.
27:11
So like, one thing that I always think when I go into evaluation
27:14
is like how much do I actually trust the novelty of the
27:18
underlying data that this is being used for, for the
27:21
evaluation. And then for me I also think
27:24
that it's about making sure you hit the breadth of different
27:27
topics that the that you want to evaluate over.
27:30
So like 1 project I think is great is like the LM evaluation
27:34
hardness which allows you to test amongst a bunch of
27:37
different data sets and so in a lot of different projects and
27:43
models I've seen trained or fine-tuned.
27:46
I think that the the one graphic that or like 1 visualisation
27:49
that I think is pretty effective is almost like these radar
27:52
charts. So looking consistently at a
27:55
bunch of these different data sets that you consistently look
27:59
over as you like compare different models that you're
28:03
training or seeing the efficacy of fine tuning against maybe a
28:06
GPT. I think that it's important to
28:11
grab data sets that are proper I guess for the different tasks
28:17
that you're solving for and then seeing how those compare to the
28:21
other models that that you want to see that you want to be
28:24
better than for instance. So like it's very easy for you
28:27
to like go to open LM leaderboard, find the one data
28:30
set that this specific model is not good at and fine-tuned
28:33
towards it to make it slightly better for it.
28:35
But that doesn't automatically mean that that is a better model
28:39
in general. And so that's why I think for me
28:41
like the the whole like radar concept of like having these
28:44
different topics, datasets and tasks are important to for for
28:48
evaluation and then ensuring that you maintain that across
28:52
all your different experiments. And ask yourself like is this
28:54
data set still valid after all these different iterations of
28:57
models exist, and like the training data and the fine
29:00
tuning data underlying it is now like in a way slightly being
29:05
falsified in a way not to like be too, uh like negative about
29:09
it. Yeah, I think another thing.
29:14
Maybe a lot of folks looking to use alms like like open source
29:20
LMS, especially in practises. I'm not sure how closely they
29:24
actually look or understand the different benchmarks out there.
29:30
Um, like do they They do they like even know what the MU task
29:39
is or set of tasks is like does it matter that their model can
29:44
like do multiple choice? Well, if it's, you know, if it's
29:47
like whatever text is Sequel model at the end of the day, um,
29:52
there's like, there's like, especially I think and it's
29:54
hard, right. Like especially if you're a
29:57
software developer coming into this space, it's like super
30:00
intimidating. Like you're not gonna go to the
30:02
hugging face leaderboard, look at the four or five metrics in
30:05
there. Go and read those like papers,
30:07
like look at a bunch of the examples in the evals and then
30:10
pick your model based on there, right.
30:12
You probably just gonna pick the like the model that's highest on
30:14
the leaderboard giving your UM capacity to to serve it or like
30:18
fine tuning service UM. And yeah, you could argue maybe,
30:21
like rigorous teams should do all that work, but they're not
30:25
ML engineers at the end of the day.
30:27
Like they're trying to just, you know, add another like billing
30:30
block to their to their product or whatever.
30:32
Um, so that's like like super hard as well.
30:36
And then, yeah, there's the whole gaming of leaderboards and
30:39
benchmarks on on top of that, either like intentionally or
30:42
accidentally, um, which is actually an issue.
30:44
But one thing we saw, I noticed that was quite interesting when
30:47
we were evaluating our support bought in going back to the
30:51
radar charts. Um and he's mentioned was so we
30:55
were like you know the kind of good.
30:57
It's good practise to evaluate evaluate your rag system for uh
31:01
another number of different different aspects and like not
31:03
only answer correctness or or accuracy but also like how well
31:07
the retriever is doing. Um, but it was interesting that
31:11
we noticed with them when we moved from, I think, uh, GPD
31:19
turbo going to get this, right? Yeah, GPT turbo.
31:22
Umm, the the like November preview to the January preview
31:28
model? The context precision of the RAG
31:34
system actually went down and so.
31:35
So that's just a measure of like how well did the UM system do in
31:42
like paying attention to like the context it was given you
31:45
know in when it like gave output the final answer and that metric
31:49
actually went down. Even though our, like accurate
31:52
overall, um other metrics like the answer correctness and like
31:55
you know the ultimate results we kind of care about went up.
31:59
And it was kind of interesting because like the there's so much
32:02
uh, you know code and examples and knowledge about weight and
32:05
misses on online that that's all been like scooped up.
32:09
And, you know, you ask Chachi PTSD about weights and rises and
32:13
we'll give you like a pretty good answer for like a lot of
32:15
questions, um, without, you know, plugging into our Iraq
32:19
system. But it was interesting to see
32:21
that the Jan January model um knew when to ignore the context.
32:29
You know we just like didn't have enough UM you know relevant
32:33
context for example in our docs you know maybe have a gap in our
32:36
docs but we still retrieved a few a few samples even though
32:39
there maybe like low quality. And so the the the like January
32:43
preview new to like ignore those samples were charted on the
32:46
context precision but like ultimately led to a final answer
32:49
because it was like no, I know better than this.
32:51
I'm just gonna answer from my you know like internal knowledge
32:53
instead. So that was kind of a fun little
32:56
insight we got from doing somewhere of our evaluations.
33:00
So that brings up a fascinating point that I've been thinking
33:04
about a lot lately, especially when it comes to rags, is how
33:08
you evaluate the retrieval side of the rag.
33:13
And it feels like there was somehow baked into the update in
33:18
January. A I know better than you, so I'm
33:22
not even going to retreat. And so like how how can you even
33:28
evaluate that part? Because you need to be able to
33:30
evaluate the retrieval side so that you can get better answers
33:36
at the end of this whole rag, like after the generation,
33:39
right? And I feel like that's one of
33:42
the foundational pieces, because if it doesn't retrieve the right
33:45
thing, then you're looking at something that has more
33:49
probability of being wrong at the end of the output.
33:58
Yeah, I think we're in a lucky position.
34:02
Like I said, there's so much good with the message content
34:05
online that's been like baked into these models.
34:08
It's almost like a bonus because it it, it seems to pay attention
34:11
when the context is decent and it ignores when it's bad.
34:14
That's that's kind of anecdotal. We haven't like studied that
34:17
properly. Um, but yeah, agreed If this was
34:20
like a system. Just looking at your metrics
34:23
right, this is you looking at a few metrics on on the final
34:26
output saying. Well, it didn't actually
34:30
retrieve, but the output still was quality, so it doesn't
34:34
matter if it retrieved or if it didn't.
34:37
Yeah, yeah, exactly. I mean, we do need to dive
34:39
deeper into like the individual samples there where it did
34:43
poorly on context, precision and like, yeah, get a better feel of
34:47
like what's going on. Because it short term we like,
34:50
it feels like a nice win. But like, yeah, that's not a,
34:52
this isn't a sustainable way to like improve our system.
34:56
We can't rely on, yeah, GPD 5 scooping up all the good stuff
35:00
next time around. Or even if you're trying to help
35:04
others that are in this situation that don't have a
35:07
bunch of information on the Internet, it's like, ohh, yeah,
35:10
well, we didn't have that problem because we've got all
35:13
kinds of great material that GPT trained on, right?
35:16
So how do you lend a hand to those in need?
35:20
Exactly. Um.
35:21
So we do measure. Yeah, Context, Precision,
35:26
Context. Uh, recall answer or context.
35:31
I think faithfulness and and context Anish, maybe, you know,
35:37
it might not know the other one. There's like faithfulness and
35:39
relevancy I think is the other one.
35:40
Um, kind of. I think there's a lot of blog
35:42
posts out there at this point to kind of running through them.
35:46
Yeah, absolutely. Gave it like a name Ragas or
35:49
something. If I've seen that like phrasing
35:51
come around, it's like, yeah, I think that's the four things you
35:54
mentioned are like the four tenants of robust or whatever.
35:58
Yeah, we actually use um. Yeah, riggers for some of those
36:00
evals. Brand of my team kind of runs,
36:03
runs one body, know he's used it.
36:05
UM, that would be like where where I would start to like
36:08
improve the system and then like, it's like anything in like
36:12
Emma, at the end of the day, you just need to be looking at a lot
36:16
of data, a ladder, like samples. Um, one one thing actually we
36:21
did through one month generations we have like this is
36:25
we're on our second, um, evaluation set.
36:29
Um the first one you know we like took a bunch of like
36:34
queries from the world and a clustered them like found kind
36:37
of representative questions from each and and that gave us like
36:41
about 100 and 3000 and 40 and we did something like values on on
36:44
that. But we we looked again at the
36:46
evaluation said like a couple of months later and kind of noticed
36:49
like some of the questions were uh maybe duplicated or less
36:53
relevant. So we've actually like whittled
36:55
that down to I think 98 now and it's probably like a higher
36:59
quality eval. So it's and we we've moved on
37:02
that evolved from like 50% to like 80% and we want to get it
37:06
above 90 as we like. We kind of scale it out to a
37:09
broader audience. But I think once you've got over
37:12
it in like over 90, we want to go back and look at our
37:15
evaluation set and make it harder again because we don't
37:17
want to like, like I feel like 1995 or 99, you know, is kind of
37:23
like almost not not a relevant right.
37:25
But, like, you want to make sure that, like, you're not just
37:29
hitting numbers that, like, look pretty on a report, Um, but
37:33
actually, like, nailing all those, like, hard, hard cases
37:37
and making your bowels like, yeah, challenging.
37:41
Yeah, I guess I do. One thing I do want to expand on
37:43
is I I do I did really like the approach that was done with Juan
37:46
bot 2 for like the whole evaluation piece in that like
37:51
most people should be doing when they're building applications is
37:54
that you don't necessarily know how your users may automatically
37:57
interact with the LM based application you're using.
38:00
So like the most important thing is to make sure that you gather
38:03
all this data from people actually using that application
38:07
and then condensing it like standard user mining techniques.
38:10
So like from our end, like we did that clustering tech tactics
38:13
to figure out which topics, especially because of the
38:15
support bot, which I think is a common use case, figure out what
38:18
support topics are people asking about, choosing the right
38:21
question that's representative for that specific topic and then
38:24
building it into the evaluation framework itself.
38:27
So I definitely think that like every company using LLM should
38:31
do what they've been doing with a lot of their other data
38:33
products is that make sure that they're actually mining the user
38:35
data for the application that's using LLM and incorporating that
38:39
piece into their whole evaluation framework.
38:42
And are you also, when you're mining that data, are you also
38:45
making decisions on OHH, maybe we should update the docs so
38:50
that people don't even have to ask this question or maybe we
38:52
should change the product a little bit so that this is more
38:55
intuitive? Yeah, for sure.
38:58
I think like maybe even the valuation piece is not the only
39:01
part where we see that because our our docs SPA is like in our
39:04
Slack page, sorry, our Slack like a Slack channel.
39:08
It's very easy to see directly like hey this, this and this
39:12
answer is being returned. He's either old information that
39:15
needs to be updated or it's like not retrieving the right thing.
39:19
And so like from our end, the reason we care about the
39:21
retrieval is like one is a grabbing that right context, and
39:24
if it's not grabbing that context, that means that it
39:26
doesn't exist. So for us, it definitely does
39:29
inform us for how we should be updating or thinking about a lot
39:32
of our documentation for sure. Ohh, that's cool.
39:36
Yeah, that feels like a super tower charger to the docks and
39:40
like the whole debril efforts. Yeah.
39:45
Demitrius I wanna put you on the spot if you've there's this
39:48
funny a funny case I see and would look to pick your brain
39:51
and the evaluation side we I see one bar occasionally answer and
39:59
usually I'll give it it's when an answer like it requires some
40:02
some code and like maybe like that's not immediately obvious
40:05
from the docks. Then it returns an answer which
40:09
isn't wrong, but isn't like the the preferred way to do it.
40:14
It basically finds workarounds of like how to use the API to
40:16
like, you know, four or five lines of code instead of like a
40:19
11 method that it's missed for whatever reason.
40:22
Would you? How would you classify that?
40:24
Is that like, correct? Incorrect.
40:26
Uh, do we need an OK like ish? Risk category, Yeah, I would say
40:31
that's incorrect, right, Because there's a better way to do it.
40:34
So you're like teaching people how to be worse engineers in a
40:38
way. Even if even if it gets into the
40:41
the results they're looking for. Yeah, I I guess at the end of
40:44
the day it's it is true. It's like what are you aiming
40:48
for? Are you trying to make it as it
40:53
it's that constant trade off? This is a great question to
40:57
think about because it's the trade off when it comes to do we
41:01
care about tech debt and we just want to move fast and whatever
41:05
we'll figure it out if we find product market fit in six months
41:10
or do we want to build for scale and have everything bulletproof,
41:14
what phase are you in? Because depending on what phase
41:18
you're in, I think you'll take that answer in a different way.
41:22
Yeah, that's it. That's actually a good.
41:24
Point looking at it in our developments cycle, yeah, yeah.
41:28
Yeah, the other. Piece I was going to mention
41:32
that you you touched on, which I think is fascinating to zoom
41:37
into is that. Really diligent teams.
41:41
Should be not only looking at what the best model is on the
41:45
leaderboard, but also what are these leaderboards, What are
41:48
these data sets even mean? And then you go out and you
41:51
create your own data set and you see how it matches up against
41:55
that. And I do think one of the things
41:59
that people aren't doing enough or I haven't seen enough of is
42:06
the sharing of data sets or like data sets as a service or
42:10
evaluation datasets as a service type thing.
42:13
Because it's like, oh, as soon as the data set is public then
42:19
you get that gaming the data set type.
42:22
I mean the gaming, the data set type actions happen, But if it's
42:28
like you are able to send model fine-tuned model to data sets or
42:36
evaluation data sets as a service and see how they rack up
42:40
against it, that could be interesting at the end of the
42:44
day because you're able to not really have to or it's a third
42:50
party service and you don't have to.
42:56
Worry that it's been. Gained as much as if it was just
43:00
with open data sets. You know maybe you guys have
43:03
seen something like that. I feel like I've seen one or two
43:06
people release data sets that are specific verticals and it's
43:10
like new datasets for the financial data and then their
43:14
company, is that like, Oh yeah, we are an evaluation company.
43:19
And I've also heard a lot of debate around is evaluation of
43:23
full-fledged product or is it just a feature inside of a
43:28
product. So these are all things that.
43:32
I often. Wonder about?
43:36
I think the sceptic in me is. Always if it's hard to hard sell
43:40
for me to like almost go and use that service for off top of my
43:44
head. Only because because we can't
43:47
see the data, it's hard to understand the credibility of
43:50
the underlying data source. Like unless they give me some
43:54
really deep understanding of like what types of data live in
43:58
that data set in which that like cells the secret sauce.
44:02
Then I I think it's very hard to ask a lot of machine learning
44:06
engineers or data or even like people in like The Who are like
44:10
security folks in a way. Like making sure like this LM is
44:13
at a point where it's ready to be touched by customers to go
44:16
ahead and say Oh yeah, this third party service which won't
44:19
tell us any information about their data other than like
44:22
whatever their marketing page says, yeah I'm going to go trust
44:25
its results. So I I do think that for me the
44:29
whole evaluation as a service, well, I think it's an
44:31
interesting concept and I think I'll probably you might even see
44:34
them coming out here and there as like data is like a valuable
44:37
resource in itself. I just think it's gonna be a
44:40
hard sell for me at the very least.
44:43
And what? About if it is.
44:45
More of a OK, you you submit your model and then we evaluate
44:50
it together type things. So it's not like you just throw
44:53
the model over the fence and I give you back a score.
44:56
It's like you you throw your model to me, but then you've got
44:59
this dashboard. I've evaluated it in every way.
45:03
I also know that you're going to use it for this specific use
45:06
case. I wouldn't recommend it for
45:09
these parts. Kind of like red teaming in a
45:12
way. Red teaming as a service, I
45:14
think there are a few companies that do that off the top of my
45:17
head. I think like Surge AI does that
45:20
and more specifically for for these like popular models that
45:24
come out. But that's one thing that it
45:28
feels like could be done. Again, going back to that person
45:34
that's just grabbing a model, that's a lot more work to say.
45:39
Alright, now I'm gonna try and red team this model.
45:42
I'm also gonna figure out, I'm gonna go read all these papers
45:44
on what all these benchmarks mean and familiarise myself with
45:48
that. You have to be very diligent.
45:51
However, if that diligent keeps your model from saying some shit
45:56
it shouldn't or doing something it shouldn't with when it's
45:59
customer facing, is it worth it? That's again, it's a constant
46:02
trade off really is the TLDR and.
46:06
And looking at it. Like that.
46:10
The. I think it's a.
46:12
It's a. You could solve a lot of the
46:15
objection just like is a reputational or like credibility
46:19
thing. Like if Kaggle did this, if
46:21
Hugging Face did this, I think people would have a bit more
46:24
faith maybe in a in a black box, but agreed, like ultimately,
46:27
yeah, they're like evaluating together, story is much stronger
46:32
and I think. You pair that.
46:36
With, you know, a strong capability to like generate
46:39
synthetic data like for that eval like specific to the
46:42
customer. Um, yeah.
46:44
Feels like pretty compelling. Because there's enough to be
46:48
worrying about improving your like, LLM system, never mind
46:52
going off and learning then about how to generate good
46:54
synthetic data to evaluate. Like, why not just go to an
46:57
external service for an expert? For that part, maybe and and the
47:01
other. Piece on this.
47:02
I was just reading a paper that it was like a TLDR of all the
47:07
different ways to mitigate hallucinations that have come
47:11
out so far. And it has a whole taxonomy on
47:14
these hallucination mitigation techniques.
47:17
And it feels like, wow, that is another thing that could be
47:21
really cool, seeing how these different techniques work for
47:26
your specific use case. And yeah, you can do it.
47:30
You can go and spend the time. There's like 32 different
47:32
techniques that they identify. So you could go and figure out
47:36
like this one looks cool. I guess it's more like throwing
47:39
a dart at the wall and saying, all right, I like this name,
47:44
let's try that technique and see if it mitigates the
47:46
hallucinations a little bit more.
47:48
And then who knows if it actually does?
47:51
Maybe it does, maybe it doesn't, and you really got to put it
47:53
into practise. So then you may find out that it
47:58
doesn't and you're screwed. And so there are things like
48:03
that where it could be cool to have someone else that does it
48:07
for you. They give you a stamp of
48:08
approval and they also map it out for you like.
48:14
So it's not just like this black box that you were talking about.
48:17
And so I definitely think that hallucination piece.
48:18
Is definitely a very compelling for sure.
48:21
I think that's like I like. Kind of like you're alluding to.
48:25
Is like there's a lot of techniques that will keep on
48:27
arising that will actually build upon previous techniques or
48:31
still stay consistent between models.
48:33
And I think that a lot of the hallucination mitigation tactics
48:36
will stand the test of time a little bit more like as you
48:40
create more models these techniques will still be useful.
48:44
So it's almost like in a way it's like like earlier I
48:46
mentioned the LM evaluation hardness.
48:48
I can imagine it being like a hallucination like mitigation
48:52
harness in a way where you want to test all these techniques
48:55
alongside either generated data like evaluation data as a
48:59
service or some sort of synthetic data and see how those
49:03
like line up. So I definitely think super
49:05
compelling use case if no one has built it yet, they should go
49:07
ahead and start building that type of stuff.
49:09
There we go. That might be the one that I.
49:10
Just convinced myself to build and, but it it also feels like
49:14
there are ways when it comes to like weights and biases and what
49:20
you all are doing on being able to plug things like that in
49:25
there, right? And say again, it's not really
49:31
what you're trying to do, but you make it easy for customers
49:36
to say, alright, cool, you want to try this hallucination
49:38
mitigation technique. We've got you covered.
49:41
It's not like in our product, but we make it easy for you.
49:48
Yeah, 100% we. Um, are just gonna.
49:52
Ship. Maybe it'll.
49:53
Be out by by the time this gets published A integration with
49:57
Eleuthera eyes um elemental harness which will yeah let you
50:03
push all those like evaluation metrics into rights and misses.
50:06
But it will also generate uh sort of like not just into the
50:09
workspace but generate a with report like specific to that, to
50:14
that model to you know just give it, make it more presentable,
50:18
um, for that particular report. And then you'll be able to add
50:21
more evaluations to that report if you want.
50:23
And yeah, same for UH hallucinations or like any other
50:28
like toxicity and PII detection. Any like of these metrics you
50:31
want to track, um, you can like log those metrics into weights
50:36
and biases. You can add a bunch of like all
50:38
your data or like a sub sample of your data, um to these quick
50:42
inspectors. There could be like the best
50:44
examples, the worst examples, you know?
50:46
Um, yeah. It's really like, it's really
50:48
powerful to be able to just have a, you know, reliable.
50:54
Performance to kind of like. System of records to be able to
50:56
like dump whatever you want into it and like visualise it nicely
50:59
and also being able to share it like that's the other thing we
51:03
we see. Especially in these cases like
51:04
you need to especially say if you're a developer and a team
51:08
and you're trying to convince your your like manager like your
51:12
CTO that that we should like do want to do something like fun
51:15
with LM's like you need to be able to like convince them that
51:19
little yeah it is performance and also like safe.
51:21
And so being able to have a place where you've stored all of
51:25
these and not like random spreadsheets or like Jupiter
51:27
notebooks that you're like flicking through, but like nice
51:29
polished reports in with some rises like is just kind of makes
51:32
their life a little bit easier to 100%.
51:35
That's so cool to see. Well fellas, I've taken up a
51:39
tonne of your time. This is awesome.
51:41
I love seeing what you are doing at weights and biases.
51:44
And thanks for coming on here. Thanks, Midrash.
51:47
Appreciate. It.