Latest / Agentic Conversations (formally mlops.community) / Building LLM Products Panel // LLMs in Production Conference Part II

Transcript

Building LLM Products Panel // LLMs in Production Conference Part II · Agentic Conversations (formally mlops.community)

0:00
Hey everybody. We got a special edition of the
0:03
Milos Community Podcast today. This was a recording of the
0:07
panel discussion that we had at our most recent LM and
0:11
Production Virtual Conference that took place about a month
0:15
ago and I wanted to jump in and just let you all know that we
0:20
will be having another one on October 3rd.
0:23
We don't have the registration open yet, but we do have they
0:27
call for speakers open. So if you or anybody you know is
0:32
doing anything cool with LM and you're actually using them in
0:35
production and you can talk about what you are doing and
0:39
what some of the challenges that you had to overcome are, we
0:42
would love to hear about it in our next LM and production
0:47
conference. So without further ado, let's
0:49
get to this panel about building products with LM's huge shout
0:54
out to all of the panellists who joined us.
0:57
And a very special thank you to my man Sam Charrington who was
1:02
the host of this panel. If you do not listen to his
1:07
podcast this week in Machine Learning and AI, go and check
1:10
that out right now. And as always, it would mean the
1:13
world to me if you share this with one person you think it may
1:18
be valuable for And of course like subscribe.
1:22
Give us a review, let us know what you thought of the episode.
1:26
Where ever you are listening to this, let's get it.
1:29
Back. Thank you.
1:31
All so much for joining us. I think this is everybody.
1:35
We're missing one person. Ah, we're.
1:37
Missing Ah. Ohh man, you were the most.
1:41
Important here. Alright, well.
1:45
I'll let you all take it away. Thank you so much for joining.
1:47
Us Alright. Hey, thanks so much Lily, I am
1:50
in Hardy agreement. This is a very exciting topic
1:53
and one that I get a lot of questions and comments around.
1:58
Hey, Allons are super exciting, but how do we actually build
2:01
products around them And that is what we're going to dig into in
2:05
this panel. I will quickly introduce our
2:08
panellists so that we can dig right into the heart of the
2:11
discussion. We've got a Smith Arathusa Smith
2:14
is a machine learning engineer. That prompt docs, where she's
2:18
applying LCM's to enterprise observability.
2:21
For DevOps engineers, we've got George.
2:23
Matthew George is the managing director at Insight Partners,
2:27
where he's invested in companies like Weights and Biases and
2:29
Fiddler. We've got Natalia Barina, a
2:32
former AI product leader at Meta, where she focused on
2:36
issues like transparency, control, and explain ability.
2:39
And we've got Sahara Moore, who's currently at Stripe, where
2:43
he's leading the work around making LMAO.
2:46
Bang at the company I am of course Sam Charrington.
2:50
I am host of the Tumble AI podcast.
2:53
If you've not come across the podcast, I encourage you to
2:56
visit to malaya.com or your podcast platform of choice and
3:00
check us out. We've got 600 plus episodes on
3:04
ML and AI including rocks on LMS and generative AI for you to dig
3:08
into. I asked our panellists for their
3:13
spiciest takes on LM. And we've got some really
3:18
interesting responses there, and we're gonna start there just to
3:22
get the well started. Spicy and Natalia wanted to kick
3:26
us off. With yours.
3:29
A. Is that LLM?
3:31
Hallucinations. Are a feature and non bug.
3:35
Why people lie? We can't verify even what people
3:39
say, so why do we expect that we'll be able to verify what
3:43
machines are saying? And how can we even
3:46
differentiate truth from lie? There's a lot of grey area in
3:50
this. My again, The other part of it
3:53
is I don't think we'll ever get rid of hallucinations if the
3:57
Bulls aren't. Dry.
3:59
Instead, the right thing to do is to use LNG for the right use
4:03
cases where they can shine. In particular, that means around
4:07
creativity and inspirations. And yeah, that's my hot, spicy
4:13
take. Awesome, Sahara, why don't?
4:14
You get a first I. Stick around for some time, we
4:20
don't need to have great ways to mitigate those.
4:23
So my end I actually think that open open source language models
4:26
are the future and we got this recent paper from Berkeley about
4:30
the false promises of open source and MMS. But at the same
4:33
time, since methods released in last February of Lamma, we can
4:37
think about our open source LMS can be quite powerful.
4:40
So first they can run at the edge and which is a great fit
4:43
for everything that is privacy related use cases and also
4:46
independent of Internet connection.
4:48
We have like great work. Like M CLM and also Lama CP,
4:52
that gets us really close to being able to run this amazing
4:55
technologies at the edge. Developers can fine tune them
4:58
for specific purposes so you can achieve better performance or
5:01
less compute and less latency. And they also give you a lot of
5:04
flexibility without predefined constraints or policies, which
5:08
is sometimes an issue, for example when you use the open
5:10
eye. API and licencing is no longer
5:14
an issue. We got some really strong
5:16
powerful alms and that are now commercially permissible like
5:20
Falcon at Microsoft Release and this paper Orca a few weeks ago
5:25
that basically shows how. Smaller Lambs and operators and
5:29
Lambs can also perform well and when you imitate the right
5:31
things and not necessarily the style like maybe other open
5:34
source Adams. So the exciting futures and
5:37
looking forward to see what people will do with those and.
5:41
My spicy take is that you know, we're all here because of the
5:44
generalised ellens. Everyone's talking about it, but
5:47
I think the future is going to be very domain specific
5:51
specialised models where each company, each organisation data,
5:55
they're propriety data is going to be the key going forward.
5:59
Everyone's going to have their own data, have their own
6:02
specialised models, and they're going to release these as like a
6:06
model as a service where they the propriety data is going to
6:09
be the big thing. I love.
6:11
How all of our commentaries like somewhat interrelated.
6:14
To each other. Mine is that we're going to run
6:17
out of public corpus to be able to trade these large language
6:20
models at scale. And that's going to happen in
6:22
the next half a decade when let's just say the GPT 5 style
6:26
models start to emerge. Mainly because you know we're
6:30
approximately training on 20% of the human corpus and we've
6:33
exponentially used every last run of generative pre trained
6:36
model on scale up to this moment.
6:38
So there's another 10X happening in terms of.
6:41
Where the extent of the training data is going to come into
6:46
training the next generation model literally publicly
6:49
available human corpus. And so coming back to escapist
6:53
point as well as the hardest point moment ago, we are going
6:57
to move to a world where private data quickly on smaller form
7:00
Thatcher large language models likely emerge into around open
7:06
source. Which is where some of these
7:07
small or large letters wells are most relevant?
7:10
Is where this sort of. Half a decade and beyond, it's
7:13
gonna go for the future of building transformer based
7:17
architectures. Now, are these spicy takes so
7:20
spicy that there's some argument that anyone on the panel would
7:25
like to pick on any of the take we've heard that far, Or anyone
7:29
in the audience, for that matter.
7:32
Anyone jumping out to disagree? I guess I could go, I'm not
7:37
disagreeing. But there is another point that
7:40
the there are open source models, you know, like companies
7:43
where like right now we're seeing so many companies out
7:46
there that they're training, they're not training these
7:49
models, but they're using LM's into their products without any,
7:53
without the huge ML team, right. There's no management of VR
7:57
training your whole ML pipeline, your whole ML infrastructure and
8:01
with these. Open source like these APIs that
8:04
Open AI cohere have. It's so easy for these companies
8:07
that don't have an ML expertise to include LM's into their
8:11
product. So I do see it coming from that
8:14
side too. There's a role for services.
8:17
Yeah. Yeah, I think if we were taking
8:20
the other side of this argument, you can make a case that A, the
8:26
quality of these smaller form factor open source models might
8:30
not be ever as good enough as the large research houses.
8:34
And with the effectively producing and over time the
8:39
research houses that are producing these large language
8:42
models are going to have additional capabilities.
8:45
You know, like you know, the function libraries that emerged
8:47
like Huggins that emerged. About a few months ago where
8:50
there there's just going to be a general arms race where the open
8:55
source community and the use of, you know, called the public
9:02
datasets is not going to be enough.
9:06
And you're going to see the research pages take on a lot
9:09
more workload when it comes to leveraging private data,
9:13
multimodal sets of data. And there's likely a chance
9:17
that, like, you know. Sort of the things that are kind
9:20
of being predicted at least as we we indicated can have an
9:23
alternative feature. So you could kind of argue very
9:26
easily for the the, the counter billing point here that this is
9:30
all going to stay very much in the hands of three or four, you
9:34
know, well funded research jobs being open.
9:38
I could hear anthropic I21 as the continued leaders in the
9:43
space, but it doesn't seem that way right now.
9:46
It does seem that there is there there is a prevalence.
9:49
Towards smaller per factor models and open source and in
9:52
particular. Actually just maybe adding one
9:55
that is you could look at timeline.
9:56
So I actually think that at some point it will be point source
10:00
models will be good enough. So now we have GPT 3.5.
10:03
We already have models reaching this level of performance.
10:06
So maybe in one year from now we will have a GPT 4 or even more
10:10
than that potentially. I know it's kind of hard to
10:12
think about it at this point, but if we take history so far
10:16
and we believe that some of those papers about open source
10:18
models achieve this. Performance.
10:20
So I think it's a matter of time we get there.
10:22
And then the other question is potentially as we are building
10:25
products in production, what are the requirements for such
10:27
products? Do we actually need a GT4 like
10:30
model to do question answering on documents or answer very
10:34
specific questions. So I think that as we build more
10:37
picks and shovels in this space, we improve on evaluation.
10:40
We better understand what are the use cases we would like to
10:42
serve and get better results even with smaller and open
10:45
source model. But this will be an iterative
10:47
process. Well, we'll touch on all of
10:49
those themes and in particular model selection a little bit
10:53
later. I want to take us to one of the
10:56
very first issues that a product builder needs to think about,
10:59
and that is the degree to which their use case is a good fit for
11:03
LM. Right now, LM's are this shiny
11:05
magical hammer and everything is a nail just waiting to be
11:08
smacked with it. But Natalie, you've got some
11:13
really interesting experiences thinking about applicable use
11:16
cases for LLM's and AI generally, and I'd love to hear
11:19
you time on what you've, you know, frameworks for thinking
11:24
about this. And in particular, are there
11:26
kind of recurring use cases that people get excited about that
11:29
are ultimately doomed to failure because they're not a good fit?
11:33
If you want to throw that spicy, take in there.
11:35
Yeah, yeah, absolutely so. I would start off by talking
11:41
about what are the fundamental characteristics of LM that
11:44
really make them shy. And it's really about fluency.
11:49
So This is why LM's are a big deal and why everyone's excited
11:55
about them. They have this uncanny ability
11:58
to give answers. They're surprisingly human, like
12:01
This is why we have so much time.
12:06
And that's really what fluency is.
12:08
The ability to articulate very natural sounding answers that
12:12
are human like and that. Is their.
12:15
However, on the other hand, as we have already touched on, we
12:20
have also hallucinations, or as other people like to call them,
12:25
confabulations, where LM sound perfectly plausible, however can
12:30
be disastrously wrong. We can't trust them.
12:35
And so a way to think about a framework for use cases around
12:41
is around 2 axes and you can visualise the accuracy as these
12:46
kind of on the on X. So it's your horizontal and I
12:50
think about fluency as your vertical axis.
12:54
And then you know, you could you could kind of map your youth
12:57
pieces along those two lines and if you do that, you will see
13:01
that places where you don't need accuracy when you might need.
13:05
Who need fluency and really benefit from LMS or things like
13:10
writing a greeting, writing poem, writing fiction, writing
13:15
children's stories. So these are really creative.
13:19
He was cases where you need fights and inspiration, then you
13:22
could think about you know other.
13:25
On the other end you might have use cases where you need a lot
13:31
of accuracy but you don't need fluency.
13:33
So these are really about. Being precise on the information
13:38
and getting the right answer. Um, so really to me those are
13:42
kind of the tool buckets, the creator and productivity use
13:46
cases versus the second bucket which is more around decision
13:49
making. So I'd consider, you know, like
13:53
where did you argue cases fit along those two lines And once
13:57
you map them, I would recommend thinking about, you know, what
14:00
is your business decision framework.
14:03
And there's really a few aspects of that, and they are, well, our
14:07
LLM's going to make your business obsolete, go all in and
14:11
invest. Two, can they boost your
14:14
revenue? So this might be customer
14:17
support, call centres, et cetera.
14:19
Then you have to think about your builder.
14:21
Bye. 3rd, can you help, can you get a competitive advantage and
14:26
this is like that vertical scenario that that is beautiful
14:30
and if we see those come to fruition and then finally you
14:35
might be in a position where you just you know LM make no impact
14:39
to your business and product and.
14:41
You could experiment. But you don't really need to
14:44
apply them. So those are the two two kind of
14:49
frameworks. One is around use cases and then
14:51
the second one is around making that business decision and
14:54
seeing what your situation is before making an investment or
15:00
before even beginning to experiment, which I I highly
15:04
encourage everyone to experiment at.
15:06
That's pretty cheap. And then click.
15:09
That's where I'm going to conclude.
15:10
Anyone want to jump in and elaborate?
15:14
I can add a few like to send so from my experience and it's
15:19
right but also elsewhere like the biggest gap that will make
15:22
us work more with them and feel more confident it actually again
15:26
like evaluation and so in the world.
15:28
Let's imagine a world where we every time we change the prompt
15:32
we can immediately know how it well performs on like biassed
15:35
society. How often are the glycinate and
15:38
also? How it correlates with our own
15:39
performance benchmark then I will be a lot more confident
15:43
deploying something just production.
15:45
So in a way it reminds me of how like deep learning and machine
15:48
learning what decade ago where it was really we all like a bit
15:51
scared to deploy something to production because my ideal the
15:54
result don't expect. But then evaluation monitoring
15:58
like what we now call ML OPS really made it a lot more easily
16:01
and and streamlined and therefore increased confidence
16:04
in applying these kind of. Here's a real element that I see
16:08
a lot, particularly on the part of, you know, product owners,
16:12
executives, they, you know, it's easy to get enamoured by a
16:17
ChatGPT demo. But when you think about the
16:20
brand risk, that's possible when your LM, your customer facing LM
16:25
starts. While hallucinating publicly, it
16:30
really, you know, put some pause in, especially with the
16:36
valuation frameworks not fully, fully baked.
16:39
Has anyone come across any useful ways to kind of drive
16:45
team alignment and alleviate that kind of fear, Natalia?
16:49
And I would recommend. Plug for my own work, but one.
16:54
Of the things we developed at Meadow was something called an
16:58
AI system card and it's essentially a way to publicly
17:02
report on the risk and to present the evaluation.
17:06
And actually this is what chat TPT did or Open A I did with the
17:10
latest version of GP is they published their AI system card
17:17
and they literally went through and they said OK, this is how we
17:20
looked at all the safety risks, this is how we.
17:23
The weighted toxicity. It's a very exhaustive document.
17:28
There's 40 pages. There's a lot in this space.
17:33
And the other thing I would say is, you know, you have to think
17:36
about evaluation is expensive. Evaluating your AI product, it's
17:41
gonna take time, it's gonna take resources.
17:44
So you have to figure out like you know how much and you can
17:47
you do. But I think everyone should
17:50
absolutely do it. And I encourage people to go
17:53
through and look at the T4 System card, which is a really
17:56
good guide for a very comprehensive evaluation of an
18:00
LLM and all sorts of issues that can go, you know, we we
18:03
mentioned privacy here and just robustness bias.
18:09
Toxicity. There's so much in here.
18:13
Yeah, there's there's a lot to do.
18:15
Yeah, yeah. Prompting is obviously been
18:19
raised quite a bit at this event so far, and it's core to LM and
18:25
it's in a lot of ways a new way to create software and systems
18:29
with the rule book that's being developed as we speak.
18:33
That the can you talk a little bit about your experience with
18:37
building prompt based applications?
18:40
Sure, I would love to. I think prompting is a whole
18:43
topic on its own. We could do like a whole panel
18:45
just on that. I've you know for the past six
18:48
months, it's the whole like how we build the application.
18:52
It's changing so much with these prompts and you know with tools
18:55
like Lang chain out there, it's very easy to get a POC out over
18:59
a weekend. But now pushing that POC,
19:02
pushing your idea into production.
19:04
That's the hard part. And there are a lot of things
19:06
that you can actually do with Prom.
19:08
And let's take take for example like you know, one use case that
19:14
everyone's been talking about is like a document question
19:16
answering system and. You can provide a lot of contact
19:21
along with the prompt that you develop and this can be you know
19:26
document one with it, URL one document 2 with the with the
19:31
significant URL. But how the output comes out,
19:35
you can always structure this output.
19:36
You can prompt it say hey I want this output in a particular
19:39
format. I've seen that help a lot.
19:41
Let's say like you want it in a JSON format with XYZ and you
19:46
know markdown tags, all these hell but.
19:49
Another thing that also helps is adding examples.
19:52
Lots of examples, relevant context into your prom has
19:56
helped a lot for me at least. It's like a week RHF.
20:00
Kind of like a few shot examples that you can add within your
20:03
context of your prompt. But not just adding examples of
20:08
how you want this prompt to complete.
20:11
But you can also get a lot of people say, hey I had examples
20:15
that overfits to these examples, but another way to overcome this
20:19
overfitting. Is by adding this relevant
20:22
examples of the contextual retrieval everyone's talking
20:25
about with like vector databases out there.
20:28
Fetch the relevant context and you can provide a list of
20:32
examples there and you can do vector database search like
20:36
semantic similarity search from what is the user's query.
20:40
Get relevant context added into your prompt and that that has
20:44
I've seen do a great job there. Another thing when it comes to
20:51
getting these lens into production is that I've faced is
20:56
the latency. Right as you change prompts.
20:59
You know there's so many agents chaining of prompts, latency has
21:02
been a huge issue and you can actually solve this by a lot of
21:07
UX features. Depending on your use case, you
21:10
can. You know streaming is a great
21:12
thing that we've seen chat DPT do.
21:14
You don't feel the latency because ChatGPT keeps throwing
21:17
you throwing you these messages. Another thing is you could you
21:22
could let your Proms into multiple things like for example
21:25
like documents searching first, fetch the documents, provide
21:27
that to the user, let the user look into the documents while
21:30
you fetch the answer. So splitting your prompts and
21:33
getting intermediate messages out there is does wonders, and
21:37
hiding the latency is what I've seen.
21:40
And yeah, I think that's my overview.
21:44
Like a really short overview of like prompting tip that eye
21:47
blur. Awesome.
21:50
So how are you? Have a take on that.
21:52
Yeah, I mean one maybe one thing I found really useful for
21:56
avoiding illucinations. So first we know the normal tips
22:00
actually like act as something we think about GPT and LMS is
22:04
this huge tree of many activations neurons and you
22:07
always want to help it understand what we expect it to
22:10
do. So the more information you can
22:12
provide within the problem and help it know what past activate,
22:15
the easier it will be for it to help you in achieving your goal.
22:18
So specifying the format. Times stating what role it
22:22
should act is always been useful.
22:24
One concrete tip for avoiding illucinations within the prompt.
22:27
In many cases you can pick out. I think about NLMS as like this
22:29
satisfiers so they always want to provide you with an answer.
22:32
So just by giving it an out they will actually follow.
22:36
So for example, if you want to classify an Amazon review as
22:39
positive or negative, but maybe sometimes you're missing context
22:42
or you're getting input that is not a review, you can always ask
22:45
it to provide maybe like, say, otherwise return and a or error
22:50
and then. You're doing so it will actually
22:52
avoid losing an answer that is actually wrong.
22:55
And and and return an NA or some other predictable form.
22:59
And talk more broadly about your experience combating
23:03
hallucination. You know, what kinds of things
23:06
have you run up against that stripe and and how have you
23:08
approached it beyond that? This prompt trick you mentioned?
23:12
Yeah, so I mean first illucinations where Elms makes
23:16
data makeup data that is incorrect.
23:18
And I feel like what's the main difference between other ML and
23:21
deep learning models before is that they do it really
23:24
confidently. So it's really hard to tell if
23:26
it's actually wrong or not. That's one of the biggest
23:29
shortcomings and building up on Natalia's point is very use case
23:33
dependent. And so if you are doing working
23:36
with the brainstorming partner, one thing versus building, maybe
23:39
have behind the scenes classification model or.
23:43
Pipeline that will then have substantial downstream tasks.
23:46
So first about illumination and understand what is your use
23:49
case, What is the impact of mistake?
23:52
Then wait to mitigate hallucinations.
23:55
And what I've seen very useful is first giving the 11 out as I
23:58
mentioned, but also prompt chaining.
24:00
So there is like a few approaches like self reflect and
24:04
also like another video I've seen recently Smart GPT, where
24:07
you basically ask the ask a question and you ask the LM to
24:11
criticise the answer as an expert in this space and then
24:14
provide even the resolution. And by doing so, it increases
24:17
performance and substantially at the cost of latency and cost.
24:22
And so that's another. Idea like prompt chaining and
24:24
and kind of. You get an answer and you check
24:26
if the answer is is actually the.
24:28
And another option and mitigation path is actually see
24:31
patients. So forcing the LM to quote the
24:34
source for its answers and it's really useful for using
24:37
hallucinations. And it will also allow your
24:40
users or yourself to check if the generated the response makes
24:44
sense. So you can actually go to that
24:45
page in that PDF and see if this makes sense.
24:49
And lastly, I've seen this is relatively nascent and recent.
24:52
There is this research called LM Blender from Allen Institute of
24:57
AI where they used they combine multiple LMS so each LM kind of
25:01
peaks in different areas. So if you combine them all and
25:05
know which one to call in the relevant time, they might help
25:08
you prove performance overall and the voice of the nation.
25:11
Yeah. What we're seeing a bit of that
25:12
in some of our companies that are more advanced work here.
25:16
So Jasper, for instance, is a great example of someone who's
25:18
actually. Using a blender style model to
25:21
basically orchestrate across multiple lens to just understand
25:25
and produce the best results where there is effectively you
25:29
know sort of mics in the latent space of the model.
25:31
One of the other areas that we're also kind of noticing has
25:35
pretty significant amount of opportunity is where you can
25:40
take call it visualisations of the latent space of a model and
25:44
be able to understand where there's practically holes in the
25:48
Latin space and. The folks that had a gift for
25:52
all the Novak team was actually doing some pretty impressive
25:54
work here in terms of visualising the latent space of
25:58
the model and then helping understand where to introduce
26:01
additional retraining runs to improve the fidelity of models
26:05
overtime. So that's that was a called
26:07
pretty exciting area that we're seeing opportunity particularly
26:11
when you're dealing with a lot more small factor models that
26:14
you're trying to train for domain specific purposes like
26:17
retraining based on. Where there's missing
26:21
developments, the data set trained, latent space properly
26:25
is something that's exciting. George and one of the earlier
26:29
sessions, Josh Robin had this great point about the the
26:32
companies that we really admire for doing a great job with AI.
26:36
Tesla was an example he gave. Didn't just kind of throw out
26:39
this model and kind of declare victory.
26:41
They built this continuous feedback and improvement loop
26:45
around their product and that's really the, the differentiator
26:48
and why we admire those companies.
26:50
What have you seen? In terms of companies building
26:54
these kinds of feedback and improvement loops around LM
26:58
based models in the wild. Yeah, I'm glad you asked that
27:01
question to him. I I think when you look at the
27:04
best MLM's and just transform based architectures that are
27:08
going to continue to evolve in this space, they're inherently
27:12
all going to have some reinforcement learning
27:15
associated with them. And whether that be a human
27:17
feedback loop, a machine feedback loop, a human and
27:20
machine back loop. I mean, just think about the the
27:23
fundamental difference between what we saw with chat cheap T
27:28
early on and. More recently, right.
27:31
It wasn't just the fact like it started to evolve the model
27:36
itself. And you know, why are there sort
27:38
of so many less hallucinations around?
27:40
Check GPT today. Well, it turns out that it is
27:43
the feedback loop itself. It's the RHF that improved the
27:46
model effectively over time. So I think we're going to see
27:51
quite a bit more, call it model fidelity model interfering by
27:56
the fact that there is just good feedback loops, just naturally.
28:01
Part of any sort of, you know, call it.
28:05
Model that's being built for the purpose of these domains as
28:09
they're kind of proving the application experience around
28:12
it. I don't think this is a journey
28:14
where you kind of just send the model off into production and
28:19
then never call it improve it beyond just introducing the next
28:24
generation of the model. I think in most cases there's
28:27
going to be some feedback loop for reinforcement learning
28:30
that's going to continuously improve models of fine.
28:33
And when we think about that kind of continuous improvement.
28:36
Improvements, you know, it's hopefully relevant or related to
28:41
some kind of evaluation metric. We touched on evaluation
28:45
previously, Sahar, dig into that a little bit deeper.
28:49
What would have you seen working from an evaluation perspective
28:52
for these kinds of models? Yeah.
28:54
First of all done by saying that I'm too surprised how we asked
28:58
quite behind on evaluation as as a community like I space, it
29:01
doesn't prevent us from running past and deploying production,
29:04
but still there's something to be made there.
29:07
I know that many folks are working on making operation
29:10
better like lynching and other smaller startups, but it's still
29:14
a gap and I feel like when it comes to valuation there are a
29:17
few ways and I we go about that. So the first one is you know
29:21
automated manner so some papers have used.
29:24
LM secretary before to score the generations.
29:28
So because generations are more like open-ended text, more
29:31
subjective and alms lend themselves really well.
29:35
One useful tip is actually use. If you're using GPS for frequent
29:40
5, you should use GPT 4 to to judge the inferior model.
29:44
So using superior models to judge inferior models works work
29:48
actually surprisingly well. And you also need to make really
29:51
clear what are your kind of metrics and what you're
29:54
measuring. For example, you might want to
29:56
have a benchmark or a set of examples for illumination.
30:01
The more examples about electricity and what you define
30:04
in your company as things that you do not make their way into
30:07
the user. You might want to have more
30:09
performance products, performance specific benchmark
30:12
for specific use case. So hopefully.
30:15
We will get more and more tools that help us to streamline these
30:18
processes and and I expect other players like laying chain ways
30:23
and biases also released a few goals to help us get it there
30:26
and but still it's a lot more manual than we I have expected
30:31
it to be at this point. Smith, what have you seen?
30:34
Yeah, I think I I chime in with Sahar like what he said.
30:39
I think another thing to bring in is prompt versioning.
30:43
That's been a big thing, prompt versioning like you change a bit
30:47
of your prompt it it can change like the output of something
30:50
that worked before and it deviates though benchmark
30:54
questions, that's been a big thing.
30:56
At Prom Dobbs, we have like a bunch of benchmark questions
30:59
that we have that that we want to evaluate against.
31:03
And another thing is that like the harm mentioned, you could
31:06
have like GPT say, hey do you think this answer is similar to
31:11
the answer that I expect? Right.
31:13
And we could even do what I've done before is have like a
31:17
scoring method where you have hey this is the question, these
31:20
are the keywords I expect in the output, right?
31:23
It doesn't have to be the exact output, but these are the the
31:25
answer should be having certain keywords.
31:28
These are the significant keyword and you could check
31:30
against those keywords. And the other one is you could
31:34
do like a quick semantic similarity.
31:37
Do you think that this answer is similar to this answer?
31:39
How the score and you could have like a whole evaluation score
31:43
like weighted against these and another thing to make sure is
31:47
that the benchmark questions that you have should span across
31:50
multiple areas. It shouldn't focus and adding
31:53
flying benchmark questions, just keep adding them in and it it's
31:57
not going to you know, you could keep running this maybe daily or
32:02
at some frequency and see where that deviate.
32:05
Actually, I'm I'm curious, Smita, how do you track all
32:08
those feedback requests and responses from music?
32:12
Something in production? By feedback request, do you mean
32:16
like the? You you found the new case where
32:19
your LMS events as well. Yeah.
32:24
So I think at that point you have that what what is that
32:27
question that deviated and I would think about like what is
32:31
the answer that I expected at that point what did the user
32:34
expect? And then I could add that to the
32:36
benchmark or I could even add like as I mentioned earlier
32:40
examples my whole like I like the semantic like the vector
32:44
store database add some examples there to improve the improve
32:49
like the output at that point. And that something that you know
32:52
you need to have like and George said that you need to have this
32:55
entire feedback loop and it's all there's there's no right way
33:00
to do it and there's no nothing right now.
33:02
So it's going to keep evolving. And are you finding that you're
33:07
able to put all that feedback back into the process through?
33:13
Through prompting or through RHF, or you also needing to
33:18
apply heuristics, regexes or whatever those might be to.
33:23
You know to enforce constraints on the LM, it's it's definitely
33:27
a lot of like you mentioned like regex matching.
33:31
I think that's been a big thing because it's not just going to
33:35
be that the output like it's not going to be that the output
33:38
always is going to follow the same format, right.
33:40
I'm going to prompt it to it can go, it can go some way.
33:44
So making sure that like the J maybe like the Jason format or
33:49
like tag things like that. I think like Open Eye came up
33:52
with the function calling yesterday that they released.
33:55
That's a huge thing that the model's been fine-tuned on.
33:58
So that's that's where that's where it's heading.
34:01
People know. Regex and and heuristics as The
34:04
Dirty secret of the big. Guy and.
34:10
Exactly, exactly. So we we've talked about
34:13
chaining Blender pipelines, all these things.
34:20
One of the, you know, the challenges that they introduce
34:24
is these API's aren't free. You know, whether we're talking
34:29
about open AI or inference, calls aren't free.
34:32
So there's an economic component that's particularly important
34:35
for folks that are trying to field products.
34:38
Natalia, can you talk a little bit about how product builders
34:42
should think about the economics of these kinds of models?
34:47
I think everyone on this panel does.
34:48
Is that the more explicit detail and examples you put into a
34:52
prompt, of course, the better model performance, as Sahara
34:56
talked about, one of the ways to assuage some of the
35:00
hallucination issues. But that means that you get, the
35:05
more expensive your inference will get with those longer, more
35:09
detailed Proms. So I think that's, that's the
35:11
first thing you understand. Yeah.
35:14
If you think about Open AI, they charge both for input and output
35:17
tokens depending on the task. It you know, it adds up, but
35:22
overall prompt engineering is still pretty cheap and easy way
35:28
to experiment as opposed to building a whole ML.
35:32
Product. So I'd encourage people to
35:36
really use prompt engineering. You know, the next thing I'd
35:40
say. Is set up your experiments, and
35:42
be very. Experiments are not free, so be
35:45
very thoughtful about how you set up your experiments.
35:47
Make sure you're solving the right problems.
35:51
Set clear expectations depending on like.
35:53
I've worked in large organisations, which means a lot
35:55
of alignment. A lot of.
35:57
Buy INS You have to explain to leadership team, partner teams
36:01
what are what kind of expectations they should have
36:04
from your experiments and minimise the risks.
36:07
Time box or experiments. Think about build versus buy and
36:10
have a clear data story. Umm, so I would say you know all
36:15
of those things. Experiment in order to figure
36:18
out your cost, and then I would say the second part is something
36:24
we had already alluded to in the earlier conversation.
36:26
Consider smaller models. Smaller models will be cheaper
36:30
and there will be trade off around accuracy, but that might
36:34
be a really good way to minimise your cost.
36:37
Umm. And yeah, those are my
36:41
recommendation. I want.
36:42
I want to hear what? What other people think?
36:48
Yeah, I was curious from George's perspective as an
36:50
investor, you know you you're seeing a lot of these economic
36:53
conversations. You know what our how our
36:57
founders kind of thinking about it and having those
37:00
conversations with their investors.
37:02
Yeah, I'm glad you asked that. I think we're seeing
37:04
particularly some of the. In the companies that are more
37:10
call it earlier in their growth journeys, they're the ones that
37:13
are experimenting faster and trying things in a way where
37:19
they're bringing products to market in literally weeks,
37:22
right. So give you a good example, mid
37:24
stage growth company Honeycomb is in the observability market,
37:28
right and they built you know next generation of their ability
37:31
pollution. They introduced a better query
37:33
builder though that instead of. You know, asking questions from
37:38
a distributed tracing standpoint, using a very
37:41
esoteric query builder to ask the question, they introduced a
37:44
natural language overlay. Now very, very straightforward
37:48
to introduce something like that using a finely tuned 11 or you
37:53
know, introducing a product feature like that within weeks.
37:57
And interestingly, that became, I would say, the most actively
38:03
used feature inside of the Honeycomb product within.
38:06
Two days of it launching. And so that ability to just kind
38:11
of try something, experiment, iterate launch and moving
38:15
quickly is something we think is a pretty profound opportunity,
38:19
particularly for any startup that's going through this
38:22
journey. We're also by the way just
38:23
quickly finding the fact that a number of larger scaled out
38:27
combines have a pretty tremendous advantage here, which
38:30
is the private data itself over time.
38:33
We don't think this is a moment where it just incumbents will
38:36
fall by the wayside. Because just generative AI
38:39
movement is happening. It actually turns out they have
38:41
one of the the most important weapons inside of inside of this
38:47
scale and capability that's going to occur in this space and
38:49
that happens to be the private data itself.
38:52
And capital. Hi, Capital.
38:55
Sahar, you wanted to jump in? Yeah, just adding on that point
38:59
about how do we, how do we make it make sense from a cost
39:02
perspective. Also fine if you have longer
39:04
like context Windows fine tuning is also quite helpful.
39:07
We have found that it's fine the model, it's a one time pass you
39:11
have to pay, but then future generations will have weight
39:13
loss prompts and toolkits or weight loss token tokens.
39:16
And then another metapoint is usually what I've seen is we
39:19
have two different type of companies, right.
39:21
We have the enterprise companies which so far I haven't heard
39:24
from anyone working in this space, Not only it's right, but
39:27
in general. Having cost is an issue, so
39:29
given the impact that usually has on the bottom line business
39:33
and for enterprise companies, it's still not a blocker for
39:36
startups. I would encourage actually
39:38
startups to currently kind of put this aside.
39:40
Of course there will be a path for this becoming cheaper and
39:42
cheaper if you look at the trend from the last 12 months or so.
39:45
And items are becoming cheaper and more powerful.
39:48
So I wouldn't let anyone kind of be blocked by this profit
39:51
margins unless it makes no sense.
39:53
And then that's a different. Yeah, I totally agree they're
39:57
working on a startup like it. If the cost isn't a huge concern
40:01
right now because the value to the customer and the
40:04
expectations of the customers increasing, they do always
40:07
expect like, yeah, I just want to ask it in normal English,
40:11
right. That's always been like I don't
40:13
want to remember this query language.
40:15
That's one thing. And coming to Natalia's point
40:18
about like the tokens you are charged for the input tokens and
40:21
the output tokens, those adding prompt engineering to constrain
40:25
right constrain your output. You could either say you know
40:28
like if if your use case is like some summarization use case,
40:31
keep the length to to like a short extent because we've seen
40:35
that the completions can go on and on.
40:37
So constraining it having like the formatting, output
40:40
formatting expectation that can reduce it, as well as semantic
40:44
caching if that's something that for your use case it makes sense
40:48
if you don't need the completion to happen every time.
40:51
If the similar question have been asked before, just get that
40:54
out. But you don't need to do API's
40:56
opening API calls all the time. 11 quick observation I want to
41:00
make. We're talking about costs, but
41:02
in reality for enterprise they're at the very beginning of
41:06
the journey where they're stared at the wrist and even wading in.
41:09
And so once they get past that the I think costs might be a
41:15
consideration. But by that time I think the
41:16
cost will be so reduced. We're seeing every day we're
41:19
seeing smaller and smaller models on edge.
41:21
There's been super excited about and open source, I think
41:25
increasingly like the story didn't costs changing very
41:28
quickly? Costs are coming down very fast.
41:31
Yeah, yeah, I think we're coming up on time, but I don't see
41:34
anyone here to kick us off, so we'll we'll pass.
41:37
On the other. Question.
41:39
Well, I think we're doing a service until someone comes and
41:43
says for the next session we did say we wanted to come back to
41:48
model selection. You like one more question?
41:54
OK, perfect. You know we've we've throughout
41:59
this talked about how there are new models emerging all the
42:02
time. We just talked about how the
42:05
models, you know, there are smaller models, bigger models.
42:08
You know, still. You know there are advantages,
42:13
advantages to to both of those. So our talk a little bit about
42:17
your experiences from a model selection perspective.
42:20
How much time are you thinking about when you are approached
42:22
with a new product opportunity, you know what's the the right
42:26
model to start with and and where do you start that
42:28
conversation? Yeah.
42:30
So and we usually think about the way right, we have the open
42:34
source models and the commercial models on the commercial side
42:37
and coming from more of an enterprise perspective.
42:39
The number one sector to start experimenting is what you have
42:42
access to. So like getting something off
42:45
the ground and working with cloud from a profit requires
42:48
requires agreement the same way that open the Eyes and Azure on
42:51
the other cloud provider. So usually if you want to
42:54
experiment I will go with what's most available at this point.
42:57
But then when it comes to performance it's still like it's
43:00
all comes back to evaluation. I'm so excited about this space
43:03
and there is a leaderboard and there is also like also
43:07
leaderboard for. Vector the basis and embeddings
43:11
we're supporting the channel in the chat, So what we usually do
43:15
we start with something like GPT 4 which is usually the most
43:19
performant. If it works correctly.
43:20
Well we can always downgrade because you 3.5 cheaper, but
43:24
more importantly faster and so depends on the use case.
43:28
But I would advise folks to start with the most the more
43:31
powerful model, see if it works really well, iterate the problem
43:33
and then potentially downgrade if it makes sense.
43:36
And then when it comes to open source models, I feel like the
43:39
more we will build our own. Aviation, it's literally in the
43:41
last four to 8 weeks and this amazing progress is happening.
43:45
The easier it will be for us to also incorporate these models.
43:47
And lastly I will say companies like AWS launching Sage Maker
43:52
like with Laughing Face and other also bedrock and and these
43:57
kind of generativity capabilities help us all
43:59
experiment a lot faster. So we would expect probably in
44:02
the next six months or so we will have a smarter engines
44:06
slash layer do allow us to plug different language models and
44:09
experiment a lot faster. And then potentially using even
44:13
more than one LM for the same use case based on our latency
44:16
requirements, cost requirements and accuracy.
44:20
Awesome, awesome Dimitrios. What's up everybody?
44:26
This is Angled 2. I'm trying to do both stages at
44:30
once, which you can imagine is a lot of fun, and I've been
44:34
listening in on this. Absolutely loved it.
44:38
Thank you all for joining. I wanted to just come on here
44:42
for this moment because I have to say, like George, I've gotten
44:47
to meet you in person. It was incredible.
44:49
You blew my mind. And I hope to meet the rest of
44:52
you asked to Meetha, Sahar, Natalia and of course Sam in
44:56
person. But I wanted to just wrap it up
44:59
with this that if anyone is not listening to Sam's podcast, go
45:04
and do that. He has been.
45:06
Absolutely. Inspirational for me and this
45:10
whole community and so much help that I had to drop in here and
45:14
tell everyone that. And because I know you're too
45:16
humble to say that. So for the people that do not
45:20
listen to Twiddle this week in Machine learning and AI, get out
45:24
there, listen to it. We'll drop a link in the chat.
45:27
And I know that I'm pretty sure most of the panellists are huge
45:30
fans of this and I'm pretty sure you probably got some fans out
45:33
there listening right now. So thank you all for doing this.
45:36
This was. All thanks everyone.
45:38
You keep one. This is Skyler.
45:42
I lead machine learning at Health Rhythms.
45:45
If you want to stay on top of everything happening in ML OPS,
45:48
subscribe to this podcast. Now.
45:52
Now now. Now now.
45:59
Now. Now.