Latest / Agentic Conversations (formally mlops.community) / Evaluation Panel // Large Language Models in Production Conference Part II

Transcript

Evaluation Panel // Large Language Models in Production Conference Part II · Agentic Conversations (formally mlops.community)

0:00
Now. We've got a panel coming up all
0:04
about evaluation, so I'm going to bring our host with the most
0:09
onto the stage. Abby, how's it going?
0:12
Hey, what's up? I am excited for this panel that
0:15
you are going to lead right now. And before we jump on and do the
0:19
full panel, I just wanted everyone to know that Abby and I
0:22
Abby's like one of our most frequented Co hosts on the ML
0:26
OPS Community podcast. So if anyone wants to check out
0:31
the podcast that we do, there is a QR code.
0:34
Go ahead and take a shot of that snapshot or just scan it and you
0:40
will see all the different podcast episodes that we have.
0:45
Abby, if I had to put you on the spot and say your favourite
0:47
podcast that we have recorded, which one would it be?
0:51
I think I'll have a hard time picking from 2/1 is by Alex Alex
0:57
Ratna from Snorkel and the other is from two people I'm very good
1:02
friends with now, which is Marianne Beshaq.
1:05
That one just came out, so if anyone wants to check out, they
1:09
had this really cool ML OPS framework and maturity
1:13
assessment test that you can do. If you are wondering where
1:17
you're at with ML in your organisation, they talked in
1:22
depth about that in one of the most recent episodes, so I'll
1:25
leave a link to that in the chat and let everyone dig in.
1:28
That was a really good one I must say.
1:31
Now let's bring on the rest of the guests.
1:35
First up, Josh Tobin is here with us, Mr Full Stack and
1:43
Learning, and also the surprise guest of our in person workshop
1:51
that is going on in San Francisco later today.
1:54
So it's good to have you here, man.
1:56
I'm glad that we are getting to do this.
1:59
And this is happening. We've also got Sahini Roy coming
2:03
up. Hello.
2:04
And this is where I get to say the last person.
2:07
Then, Abby, I'm handing it over to you to take over.
2:10
All right. And where's Amrutha?
2:12
You there? Yes, there it is.
2:14
OK, so now there's too many people on the stage.
2:16
I will remove myself. I met you all.
2:19
Take it from here. OK, hi everyone.
2:23
Ohh, so very quick intro for Josh America as well as Sophie.
2:29
Josh is the founder and CEO of Gentry.
2:32
They're working on tools to analyse, explore and visualise.
2:37
And so you can improve your models using Gentry, the habit
2:41
and USD key that is available and Brutha is the CEO and Co
2:45
founder of structured dot IU. They're building their
2:49
engineering tools for LM, so everything from your intuition
2:53
to your gear processing pipelines, they've got you
2:57
covered. So funny is the senior developer
3:00
relations manager at NATO and one of them there.
3:04
There's a lot to mention about her.
3:05
She's been very active in the. Develops as well as eleven
3:09
months community where one of the things we want to talk about
3:11
today is the open source toolkit that they've recently released,
3:17
Nebo, Guardrails and media or any more work that they haven't
3:21
released on Ellen based conversational systems so.
3:26
I want to start with all of you first, because you've been in
3:31
this space for quite a while, All of you from ML to now in
3:36
NMS. How do you think about evaluations that your company?
3:40
Yeah, I think for those of you who have been in the ML world,
3:43
pre Ohms, there's kind of a way of thinking about evaluation
3:47
that we're used to. Which is based on the fact that
3:50
in the pre LM world, most of the time the project that we're
3:53
working on starts by building a data set.
3:56
And most of the time since we're training a model, we have a
3:58
clear like objective function that we're trying to optimise on
4:00
that data set. And so that makes a naive
4:04
passive evaluation really easy because it's like, OK, you hold
4:07
out some data from your training set.
4:09
You use the same metric that you are training the model on and
4:12
that gives you an evaluation of how your model is doing.
4:14
Now that hides a lot of complexity under the rug, but I
4:18
think moving to generative AI and LM, some of those
4:22
assumptions are violated and that's I think what's making
4:24
this so difficult for a lot of companies to figure out.
4:26
So first of all, when I started LM project, I usually don't
4:30
start by building a data set. I usually start by thinking of
4:33
what I want this thing to do and then just coming up with a
4:35
prompt to try to encourage it to have that behaviour.
4:38
And so that means that. On one of the key challenges for
4:42
evaluating LM's is like what data do we even evaluate them
4:45
on? How do we know what the right
4:47
data set is to to test these models on?
4:49
And then the second question comes from, for the second
4:52
challenge comes from violating the other assumption that we
4:54
have in traditional, which is that you know, oftentimes in in
4:57
generative ML we don't have a clear objective function, right.
5:01
Like, let's say that you're building a summarization model.
5:03
How do you measure whether 2 summaries of the same document,
5:07
whether one is better than the other, or whether one is
5:09
adequate and it's really not? Obvious question.
5:12
And so those two challenges I think are at the core of like
5:15
what makes this really difficult for a lot of companies and I'll,
5:17
I'll pause there and I I think there's a a a framework that you
5:20
can use to think through how to do this well.
5:22
But I I will, I'll save that maybe for a later question.
5:25
On man, what a what a cliffhanger on that.
5:28
I would say there's so many things to think about when it
5:32
comes to evaluation. I agree that you know, thinking
5:35
with the end in mind first. That's definitely what we got
5:38
our clients to do like similar to what Josh was saying, but
5:41
also it's really tough. When you're creating something
5:44
for such a generalised environment, I think a good
5:46
example. Of of where this was interesting
5:50
actually you know what we'll we'll pause in the example cause
5:52
I think that'll be helpful later.
5:54
But just coming back to more broadly what do I think about
5:56
evaluation, think ultimately it comes to accuracy and speed,
6:01
right? But accuracy is so dependent so
6:03
on on what the actual goals are. Thinking about, I guess more
6:07
detailed questions is hopeful, does model actually do what you
6:11
instructed to do and when it comes to accuracy, just the
6:14
model actually answer something correctly, is it coherent?
6:19
Those are two very different ways of measuring.
6:21
You know how accurate, how how well you know things are set up.
6:25
I think hallucinations is is another really important thing
6:30
to be considering. So to the model makeup, any part
6:32
of the response, how are you putting controls in for that?
6:36
You know, are they? Are they deviating?
6:38
Another part I think is context. So does the model actually
6:42
remember what you're talking about?
6:44
Are you staying on topic? I think also for for my
6:49
particular realm when we talk about the framework Nemo
6:52
guardrails which I'm sure we can get into in a little bit for how
6:55
it can actually support but is it safe?
6:57
Is it secret? Are we going to execute
6:59
malicious code it's going to call an external application
7:02
states that robust enough to prevent individual
7:05
identifications. You know especially when
7:07
thinking about healthcare and privacy codes toxicity bias.
7:11
I mean, there's so many, so many different factors, but if you
7:13
really have to sum it up to the top two, is it accurate and
7:16
what's the speed and? And latency for it, but so much
7:21
to talk about. This is a really great session.
7:23
Amrita what? What do you think?
7:26
Yeah, great answer. So one thing I always think
7:29
about is when you're doing evaluation for, you know, let's
7:32
say like a classifier, there's also some notion of like what
7:36
ground truth looks like, what a correct answer looks like.
7:39
If you're building a classifier of like Ohh cat versus dog or
7:42
red versus blue, like there's a ground truth that you can always
7:45
operate from and like evaluate against.
7:48
I think with LMP, one thing that I notice is because everything
7:51
is so qualitative, even from person to person, like thinking
7:55
about what does a good answer look like?
7:56
What does a? Answer look like what is the
7:58
expected length, how concise should it be?
8:01
What kind of tone should it have?
8:03
There are like I think attributes of like what a good
8:06
answer looks like and that varies from like based on the
8:09
expectation of the user of the model.
8:11
So I feel like there's an opportunity to build and
8:13
evaluation mechanism that is highly personalised and I think
8:17
the primitives of how to build that are things that are still
8:21
up in the air right now and you know, really excited to find
8:25
that. I think I really love the
8:27
collective perspective here, but I wanna ask you more about how
8:32
do we think about that perspective for different
8:34
companies. So right now there are two
8:36
different kinds of interest groups when it comes to large
8:39
language models. The first are the model builders
8:42
and the second are the people who are developers who are
8:46
building their own application on top of these large language
8:50
models. What would an ideal performance
8:54
mean for both of those interest groups and how do you?
8:58
Decide how to think about building an evaluation
9:02
framework. Mean you're probably on each of
9:05
those sites. Yeah.
9:06
I mean, I think one thing that you're pointing out is 1 big
9:09
change that I've seen in the industry in the last six months
9:12
or so is called the ChatGPT effect, right?
9:15
In the old world, the olden days, you know, back before ML
9:18
was even a twinkle in your eye. Machine learning projects or
9:22
these like just really technically complex.
9:26
High effort projects that most companies right Like there's I
9:29
personally don't know of any examples in large companies
9:32
where a project, a deep learning project took less than call it
9:35
six months and most of them took more than a year.
9:38
Now in the post Jack GBT world all of these companies like you
9:42
see these announcements every week or every couple of weeks
9:45
from companies that are building announcing their ChatGPT powered
9:48
feature. And if you talk to these folks,
9:49
the amazing thing is a lot of these, a lot of these products
9:53
were built in like weeks, three or four weeks.
9:56
So you ask yourself, like, how does that happen?
9:58
And the answer, unfortunately for those of us in the on the
10:03
MLS side of the house, is the reason why they shift those
10:05
things so quickly is because the ML people on the team didn't
10:08
build them. There's the software engineers
10:10
that built them and you know that's not, that's not to say
10:13
that ML folks don't have a role to play going forward.
10:15
I think actually evaluation is one of the critical places where
10:18
ML folks have a role to play. But what what strategy BT did is
10:22
it really kind of lowered the barrier to entry and also the
10:26
level of intimidation that most folks in the org have about
10:29
interacting with these systems. So if you're like an exec in a
10:31
company and you're sponsoring a deep learning project, you kind
10:34
of like hire the the nerds, the PHD's and you let them do their
10:37
thing for six months and you hope that something good comes
10:40
out of it. Now if you're building like a
10:41
cheap project, you've been playing around with that GPT on
10:44
your own, you know, you know that it's not that difficult to
10:47
get it to do what you want to do.
10:48
And So what we're seeing is that a lot of folks in non-technical
10:52
roles are a lot more involved in the process of building these
10:55
applications than they were six months or a year ago.
10:58
And so the implication that that has for evaluation, I think it's
11:01
actually really positive which is that you know one of the key
11:04
things for evaluating LM based applications is that we need to
11:09
find as an industry is the right mix between.
11:11
Automated and human based evaluation.
11:14
But the great thing about having more stakeholders involved is
11:16
that you can involve those stakeholders in the process in
11:19
such a way that they are helping you progressively evaluate the
11:22
quality of your model as they go.
11:24
And so I think the non-technical stakeholders, you know sometimes
11:28
actually doing a lot of development and prompt
11:30
engineering sense, but they're also like the producers of
11:32
evaluations and then the technical folks are more
11:34
consumers. 100% I mean like I think that's a really great, I
11:38
think that the the line is also I think for domains.
11:42
Specific information like grabbing people who have been in
11:45
the industry or for that particular domain for 2025
11:49
years, right. And bringing that information
11:51
and layering and sitting down with, you know software
11:53
engineers and engineers and and actually layering that in.
11:56
It's a delicate balance. I think again between human
11:59
evaluation and automated evaluation.
12:02
It's it's an interesting question.
12:03
I have to Mull a little bit on that.
12:05
I don't know Amrita any any other things to have.
12:08
Josh, it's so great that you're going first.
12:10
I feel like you're you're getting the right like
12:13
framework. For us.
12:14
Yeah, I don't think I have too much to add on that one.
12:17
Yeah, I I think that was a very, very comprehensive answer, I'll
12:22
I'll say which is basically like I I guess it, it depends which
12:28
is most of at least my personal experience has been talking to a
12:33
couple of companies is none of the large language models at
12:36
least at big companies have been deployed in production.
12:39
We're not really talking about opinion.
12:41
We're not talking about Googling all of those companies.
12:44
Or let's say about banks, which were huge when it came to
12:49
financial modelling and as such. Or were they adopters for
12:53
machine learning models and as such?
12:55
And it's it's great that we have produced that time for everybody
13:00
to get started and have more stakeholders because that was
13:03
one of the problems with machine learning which is like bringing
13:06
everybody together on a single table and say you know let's
13:10
let's assist what you really want out of this model.
13:14
So my second question here is we're seeing so many evaluation
13:20
benchmarks, right? How do we create stable
13:23
experimental setups to be able to evaluate and monitor?
13:26
Accuracy of our applications during and post deployment.
13:31
LM based applications during post deployment.
13:35
Maybe I can kick off on that one.
13:37
So one thing I often think about is how do you?
13:40
Anytime you have an experimental setup, you want to keep all the
13:43
variables the same as much as possible.
13:45
And I think in this case, having a set of prompts that you test
13:49
against and benchmark against on like a continuous basis is like
13:53
super important. So one thing we've noticed even
13:56
from. Fermentation that we've done at
13:59
our company is you can put the same input into GPT 3.5 turbo
14:03
and it gives you a different answer one hour later versus
14:07
like one hour before. And there's like a lot of
14:09
reasons for that and it's like not like deterministic in the
14:13
way that it answers. And so it's the outputs are you
14:16
know they're going to be different even if you have the
14:18
same inputs. But I think that like if you do
14:21
a qualitative evaluation where you keep the inputs as stable as
14:26
possible and like maybe have. Like some sampling and ongoing
14:31
measurement of the system health and like perhaps alerts if it
14:35
like drops too low, you can make turn this into like a.
14:39
Observability problem as well in a lot of ways and I think that
14:43
building those kinds of systems will help like you know, like
14:47
help us respond when when the quality of the output goes bad.
14:54
Like that. I mean not to do the not do the
14:56
founder thing, but that's definitely one of the things
14:58
we're betting on a gantry is observability playing a key role
15:00
here. One thing I would want to add is
15:02
I think if your role is a researcher then or if you're in
15:07
a really early stage of a project where you're kind of
15:09
deciding what model to choose, then public benchmarks are
15:13
helpful. I tend to prefer the sort of ELO
15:16
based benchmarks. I think they correlate a little
15:19
bit better with my subjective experience of interacting with
15:22
models and how they seem to perform.
15:24
But if your job is not a researcher or, you know,
15:29
prototyping something, but your job is building an application
15:31
with language models, then you should not rely on public
15:35
benchmarks. Like.
15:36
They are basically almost, almost the equivalent of useless
15:40
for you in that role. And the reason for that is they
15:43
are not evaluating the model on the data that your users care
15:46
about, and they are not measuring the outcomes that your
15:48
users care about, right. And so if you think about like
15:50
measurements in machine learning in general, there's kind of
15:54
like, I think of it as like a. Pyramid of usefulness versus
15:57
ease of measurements. The most useful signal that you
16:01
can always look at is outcomes, right?
16:02
Like is this machine learning? Is the system that this model is
16:05
part of? Is that solving the problem for
16:07
your end users? Two opinion measure but really
16:11
really valuable to measure. And then you know, easier to
16:14
measure but less useful than that are things like are there
16:17
proxies that we can look at like accuracy or asking another
16:21
language model if the output of this language model is good,
16:24
that correlate with the metric that we ultimately care about.
16:27
And then all the way to the bottom of the pyramid, really
16:29
easy, really accessible, super low hanging fruit but not very
16:32
useful is like looking at publicly available benchmarks.
16:35
You know, you're just not really learn that much about your task,
16:37
but doing that. Yeah, my two sons, You know,
16:39
like. That standardisation is king
16:42
when it comes to this kind of thing.
16:44
Automated testing I think Seneca monitoring systems and this
16:47
custom dashboards through that observability stack I think is
16:51
is getting really exciting. I've seen a lot of really
16:53
interesting tools that are that are doing this really well.
16:56
I'm actually curious to dive a little bit more into to Gantry's
16:59
response to this as well. But I know Fiddler shout out to
17:03
arise. I think there's CEO or CPO is
17:05
doing a presentation on this later.
17:07
I think their stuff is really interesting.
17:09
Hugging face also did some. Customer evaluation metrics that
17:13
I think make it much, much easier for for the proper side
17:16
of it making sure you're synchronising and evaluating
17:18
consistently throughout. I think Josh you mentioned
17:21
earlier doing user human feedback for order of
17:23
development ongoing. I think is is gonna be
17:26
continuing to important for for how we set up the evaluation and
17:30
monitoring the accuracy long term.
17:32
But you're right, I think those those public standard
17:35
benchmarks, this is probably the the base and then continuing to
17:40
to evaluate and use your own metrics will be.
17:42
Super important. Yeah.
17:44
So one of the questions I wanted to ask you because we're going
17:48
into the domain of talking about, you know, we need to have
17:52
very domain specific company specific benchmarks.
17:57
What are the popular use cases for large language models that
18:00
you've seen in production right now?
18:03
One example that that might come to mind is I think Google's I
18:07
don't know is it P ALM. However they choose to say it,
18:12
they release their. Your second version I think back
18:14
in in April and it's is a really interesting case study on how to
18:18
develop something that's very domain specific because I I like
18:22
it because it's very much designed for purpose.
18:25
I think Josh said in the beginning about how do you
18:28
design the prompts and the testing data appropriately for
18:31
the outcome in mind, right. How do you design for the end
18:34
goal in mind? They use data sets that were in
18:38
Q&A form that were long and short answer form inputs were,
18:43
you know, search. Worries for biomedical
18:45
scientific literature and and and robust medical knowledge So
18:48
that was actually a really really great example of of a
18:51
good foundation but I think they had metrics that also aligns and
18:57
they tested it against I think the was the was the medical
19:02
licencing questions as as their metrics for actually determining
19:07
is it successful but but in addition to that what I like to
19:11
stay they reached out for human feedback not only from.
19:14
Clinicians, clinicians to answer the accuracy of the specific
19:19
responses but also from non clinicians for you know a range
19:23
of backgrounds you know tonnes of different countries and
19:25
languages to make sure the information that was coming out
19:28
was accessible and and something reasonable and I think that's
19:32
actually you know everyone's gone out of you know gold spiral
19:36
asking medical questions. So I think it was actually very
19:39
relatable and and the results were were really phenomenal and
19:43
their ability and and the results to.
19:45
To answer those medical specific questions in a, in a you know in
19:50
a in a strong way. I hope I've I've kind of dug
19:55
into that but I I think that's one particular example where
19:58
where I think that that was done really well.
20:00
I think another one is actually the Bloomberg GPT.
20:03
I don't know if that's being done out in production, but I
20:05
think as far as domain specific benchmarking and assessments for
20:10
financial specific questions, that one was, was a really
20:13
strong example. I think that use cases that I
20:15
found really compelling are in like the Customer Success and
20:20
Customer Response categories. And I think like it's one of
20:24
those things are, you know, being able to respond to your
20:28
users quickly is like a, you know, a huge win for companies.
20:31
And being able to scale customer interfacing roles is like a big
20:35
challenge and can be expensive. But I think with LM you have an
20:39
opportunity to really leverage like knowledge bases that you
20:43
might have and expose that in a way that can like, you know,
20:46
enable your customers to like interact with your product in a
20:49
more meaningful way. And I think that's really
20:50
powerful. There's also like a really good
20:53
built-in way to evaluate the quality of responses here
20:57
because you can just ask the user like ohh did this solve
21:00
your problem for you? And you can also look at like
21:03
you know other metrics like ohh how often are they coming back,
21:06
like how how many messages did it take to resolve this issue.
21:09
And there are like a lot of really great ways to kind of
21:13
proxy the quality of the responses.
21:15
And so I think that's a good one.
21:19
Josh, you want to add on? That I think there's kind of
21:22
like in terms of real use cases, there's.
21:25
The there's trifecta one category information retrieval,
21:29
so it's things like search, document question answering,
21:32
things like that. Then second category is chat.
21:37
So a lot of companies are building sort of chat bots into
21:39
their products, use cases like customer support, product
21:43
features and then the third is? Yeah.
21:50
Like text generation, sort of marketing copy type use cases,
21:54
those are the three that that that we see the most.
21:57
Which one do you? Think is.
22:00
Most robust right now where is is doing like meeting the needs
22:03
like the the best right now. I mean, I get asked this all the
22:08
time. And I Yeah, it depends, right?
22:10
Like, it's for all of these. There's such a range of
22:13
complexity, right? It's like really easy to just
22:16
kind of dump, you know, ChatGPT API into your product and call
22:21
it a chat app. But then, how well do you
22:24
actually need it to work for it to be useful depends a lot on
22:27
the context of your product and what you're trying to get it to
22:30
solve. So I would say all of them are
22:33
relatively usable depending on the broader product context and
22:37
I think this is kind of like. General point.
22:42
On how to think about machine learning applications in general
22:45
is like I don't think I just made the mistake that I don't I
22:49
don't encourage other people to make which is like I don't think
22:51
it's makes sense to think of ML use cases as group by like
22:53
technical use cases. It's really you should think
22:56
about them in terms of products use cases because that has a lot
23:00
more of a role in determining the difficulty and challenges
23:05
that you're going to face when you're building this
23:06
application. Then you know what model you're
23:09
using or what set of techniques you're using.
23:12
Blue models together. Because ultimately, at the end
23:14
of the day, the thing that makes building products and how hard
23:16
is that machine learning models are just.
23:19
Don't always get the answer right.
23:21
That's always been true. It'll always be true.
23:24
It's a it's the nature of the technology, it's probabilistic,
23:26
right? So it's not always going to be
23:28
right. And so the question is like, how
23:30
do you build a product in a context where you know one of
23:33
the components of your product is going to get the answer
23:36
wrong? Like you know that it's going to
23:38
get the answer wrong. And so the broader product
23:40
context of, like what does it take to actually work around
23:44
that limitation is more important for the difficulty of
23:46
the project than what set of algorithms you're using.
23:51
Yeah. Yeah.
23:52
One of the things I also wanted us to touch upon maybe probably
23:57
the last question that will get before we open it, open up the
24:01
panel to audience Q&A is as you're building the model, you
24:07
need lots of data and all the data may not be good.
24:12
There are there are techniques like early stopping to be able
24:15
to philtre the right kind of DSS.
24:18
And after that, yes, you've you've built your model.
24:21
You're you're now trying to deploy, but you're also putting
24:26
some checkpoints and doing some sort of logging as well
24:30
throughout the process. How do you guys think about
24:34
evaluating the performance at those specific points?
24:38
What tools have you seen being used and what are the gaps that
24:43
you've seen in in terms of the tooling industry when it comes
24:47
to a router for you, very specifically when it comes to
24:51
the assets? Where are the gaps that you've
24:53
seen? And Josh and Sophie for both of
24:56
you have wanted to understand more.
24:57
So from like the perspective of the model checkpointing itself,
25:01
as well as the prompt, as well as like the prompt drifts and
25:05
stuff. So I'll let Amruta go first, her
25:09
let her cover the data side of things and then we cover the
25:12
modern side of things. Yeah, absolutely.
25:15
So in terms of like, data set quality, it definitely makes a
25:18
really, really, really big difference.
25:20
I think a lot of, a lot of systems are very garbage and
25:23
garbage out and like, you know, if you start off with like a bad
25:27
like baseline, you're going to be kind of disappointed with the
25:31
results. And I think like oftentimes,
25:32
like curating the data set itself is like a really big
25:36
challenge. I think sometimes having rules
25:39
or heuristics about like, oh, like this is like understanding
25:42
your data set like, OK, this is like the missing data, the
25:45
quality of the data like this is what's incomplete.
25:49
This is what's perhaps inaccurate or not in the same
25:53
form as like the rest of the data.
25:55
I think like, like you know, some standard data set cleaning
25:59
techniques make a big difference.
26:01
But then I think also curation makes a big difference.
26:04
Like what you are looking for. You like hypothetically wouldn't
26:08
really need a model as well. Like if you could define that
26:11
with words perfectly, it would be difficult to.
26:14
You wouldn't really need a model to do that.
26:16
It could just be like a bunch of if statements.
26:18
So you know, you're kind of dealing with like ambiguity
26:20
around what you're looking for, and so there's also ambiguity
26:23
around that, like curation of that data set.
26:26
And so I think it's like actually an iterative process.
26:28
Evaluation isn't really like a one time thing, but it's
26:32
something that you have to iterate on and like try to find
26:35
over time as well. Josh, do you want to add on top
26:39
of that and going to the water side of things?
26:42
Yeah, I think I'll take a slightly different position.
26:44
I think I totally agree with what you're saying, if we're
26:47
talking about machine learning, but this is the LM's conference.
26:51
And so my strong contention is, if we're talking about alms,
26:56
most of you. Like most of the people in this
26:58
room right now, you should not be thinking about training
26:59
model. There is very little likelihood
27:02
that you're going to get better performance on an NLP task by
27:05
training a model than you will by prompting GT4.
27:08
That's just, you know, that's unfortunately just the truth.
27:11
It's a it's a new world that we're living in.
27:13
Like we've talked to many companies at this point that had
27:17
kind of ongoing NLP projects, six month projects, nine month
27:21
projects, year long projects that they're able to beat their
27:24
performance by switching over to large language models.
27:28
Calling an API, doing some prompt engineering, maybe some
27:30
like few shot in context learning in a matter of weeks,
27:35
right. So I think if your goal is to
27:39
build products with machine learning, especially with with
27:42
NLP, then I think the the rule of thumb is like no training
27:47
models until product market fit. Like if you if you haven't built
27:50
this thing out and demonstrated this thing can be really good
27:52
using models that you get off the shelf for NLP, then it's not
27:57
going to work. Like you're not going to be able
27:59
to do better by training your own model.
28:01
For other domains of ML, that's I would have a very different
28:03
answer to that question. And then on the tooling side, I
28:07
think, yeah, you should just come talk to Gantry.
28:09
We've got good solutions for all the problems that you just
28:12
described. Tell us about it.
28:13
Give the give the pitch. The pitch is, you know, I think
28:18
the the way that machine learning evolved is like there's
28:23
so much focus on training models and artists is kind of from the
28:26
beginning. And I think this is doubly true
28:28
in the world, is that training model is not the hard part
28:30
anymore. The hard part is, you have this
28:33
model, you've trained it. And you've gotten over the hump
28:35
of, like, you know, the statistic that like every ML OPS
28:38
company used to say like 85 models, 85% of models never make
28:41
it production, which is like a B statistic.
28:43
And it's definitely not sure if it ever was true.
28:46
But you've gotten over that hump.
28:47
You've got the small introduction.
28:49
That's when the hard part actually starts, right?
28:51
Which is like, OK, how do we know if this thing is actually
28:53
working? How we know it's solving the
28:54
problem for users? How do we maintain it?
28:57
The percentage of time that M engineers start spent on
29:00
maintenance increases can, you know, quickly dwarf the amount
29:04
of time that they're spending on actually building new features.
29:07
And then lastly, you know, if you even if you have models up
29:11
and running in production, you're maintaining them, you're
29:13
leaving a lot of performance on the table if that's all you're
29:15
doing. Because if you think about like
29:17
what distinguishes really great machine learning power products
29:20
from, you know, average ones, it's like if you look at like
29:23
chaps, CBT or Tesla or things like that, the thing that they
29:26
all have in common is that they are maniacal about building this
29:29
loop around the model building process.
29:32
That involves like looking at outcomes, looking at user
29:34
feedback, feeding that back into the training process.
29:37
Right. So if you're not doing that,
29:39
you're machine learning product is just not going to be great.
29:41
It could be OK, could solve the problem, but it's not gonna be
29:43
great. And So what the entry does is
29:46
have an infrastructure layer that supports opinionated set of
29:50
workflows that teams of folks working on machine learning
29:54
products can use to collaborate on this process of taking a
29:57
model that's been deployed and using production data to
30:01
maintain it and to make it better over time.
30:03
And so we make that a lot cheaper, easier and more
30:08
effective. Machine learning together.
30:11
To follow that, I mean long time fan of Chantry, long time fan of
30:14
your blog series. So I it's just it's kind of nice
30:18
to to hear your your perspective on it.
30:20
I think for me for tool sets, I know there are a couple of ones
30:24
about prompt engineering that are coming out.
30:26
I have to play with them a little bit more to to know you
30:29
know which ones are most effective.
30:31
I'm getting an inkling on which one is going to be the best.
30:36
I think from NVIDIA standpoint obviously we we talk about
30:40
guardrails a lot it's our new open source framework that came
30:43
out. We talk about you know I I
30:44
mentioned a little bit like how do we make sure even the if the
30:48
the data in data out problem is is already something that that
30:52
we're gonna, it's challenging right.
30:54
But I think if you create the right guardrails for it.
30:58
So making sure that we see on topic, making sure that you know
31:01
when someone asks the top bot about your product, different
31:04
offerings, are you going to go and talk about?
31:07
Competitor offerings instead of your own, right.
31:10
Also making sure that you've got, you know, I talked a little
31:15
bit about like secrets and malicious code and management of
31:19
external application access and interactions leading to
31:22
misinformation and toxic responses and inappropriate
31:25
content. That's all something that we're
31:27
all going to have to work through within guardrails.
31:28
We'll get you part of the way there.
31:30
I think we're still experimenting with with kind of
31:33
some other tooling that can help with the rest of it, but
31:36
definitely I would say. Check it out, it's open source,
31:39
Nemos open source. I think it might you know,
31:42
supplement a lot of the other tools that that kind of were
31:44
mentioned today. So.
31:47
As much as I would love to continue this panel, I sadly
31:52
have to cut it off because we are running short on time and as
31:56
you know, that is my one job. It's to keep us on time.
32:01
And so I loved everything you all were saying and there was a
32:05
tonne of questions in the chat that were coming through.
32:08
So all of the panellists, it would be awesome if you jump in
32:12
the chat, respond to some of these questions that came
32:14
through, and I will. Leave it here and get you all
32:20
out of here, but ask you to stick around and say a few
32:23
things in the chat.