Latest / Agentic Conversations (formally mlops.community) / Cost/Performance Optimization with LLMs [Panel]

Transcript

Cost/Performance Optimization with LLMs [Panel] · Agentic Conversations (formally mlops.community)

0:06
Oh yeah, man. All right, Mario, and be guests
0:17
of honor that I am going to end up replacing myself with Lena,
0:22
Lena, you are the one who has been tasked with leading, this
0:26
panel that is coming up. I'm going to hand it over to you
0:29
and let let's get it started. Thank you all for joining us,
0:33
amazing. Yeah.
0:38
So I'm also the male engineer and I have an amazing panel for
0:41
you today. So if two people who run large
0:44
language models of production, we have Daniel was a research
0:48
scientist at Neva, which is an ad-free private search solution.
0:53
Try not quite nice and they using that language models in
0:57
search for all kinds of questions, like summary,
1:00
semantic retrieval query generation.
1:02
So I'll be super interested to hear what you learnt by applying
1:05
it to the sludge space, then we Mario was a staff engineer at
1:10
intercom and they put one of these early Duty for power
1:14
projects out which was a chatbot.
1:17
And then we have two people from the tooling space.
1:20
Really also very excited to have here Jarrod, which is the
1:23
co-founder of prompt layer, it's a platform to manage your
1:26
production, LM apps with reliable prompt.
1:29
Engineering monitoring of the problems.
1:31
Like versioning cost latency and you can track historic usage,
1:36
history and performance. So, So I'm really excited to
1:39
talk more about evaluation and last but not least, we have
1:42
Lewis who wears many hands. He is a professor of computer
1:46
science and Washington and also the co-founder and CEO of up to
1:50
ml where they offer easy and efficient deployment.
1:54
And very interesting part for me especially was that you can just
1:57
upload any computer vision or an LP model and it will be
2:01
optimized and it will tell you which instance to run it on.
2:05
And soon, you can also run your model stairs.
2:08
So I'll be very interested how you help users to make these
2:11
model run where you more cost efficient and faster.
2:15
Okay, so maybe let's start with costs.
2:19
We know generally running large language models is quite
2:22
expensive, much more expensive than normal page, like, and apis
2:27
Mario, and Daniel, maybe because you run it for products was
2:31
caused an issue. How did you approach this the
2:36
cost angle of running in production?
2:39
Yes, cops was an issue especially with scale and I
2:43
think that's kind of like where we started out.
2:45
Out, we start off with some, these kind of like foundational
2:47
apis as a way to figure out. Is this a product that's useful?
2:51
Does this help our ranking signals to our users like this?
2:54
And then once we actually started getting that both the
2:57
cost of running that and how much we could run this on really
2:59
becomes a bottleneck. So if you think about like
3:01
summarization systems potentially running is like in
3:04
production real time for users that's very possible.
3:07
The API it's kind of slow, it can get expensive but what if
3:10
you want to run some reason? Let's say the entirety of the
3:13
women index which is Is on the order of billions of documents.
3:17
At that point, no longer can kind of become actually cost
3:19
possible. So, you have to move in house or
3:21
some type of smaller system at the same time.
3:24
It's just a question of like, how widely do you want to cast a
3:27
net when using these models? How many people do you want to
3:29
access them? And what is their expectation
3:31
for turnaround time? Yeah, do you have a guideline
3:36
like, how long would you start to have a working prototype and
3:40
also, maybe could you speak on imagine?
3:41
You're running it and house now but it's still expensive, right?
3:44
So you have a ball park compared to other services.
3:47
How do you evaluate that? And how do you bring it down to
3:50
which level? So on the evaluation side or
3:53
initially? Using kind of these foundational
3:54
models, it's about kind of like having rapid iteration.
3:58
The ability to kind of revive things via prompt and get this
4:01
in front of users to actually figure out what they thinking
4:04
and once users Pakistan like oh this is great.
4:06
This is the use case that I'm working on.
4:08
Then we tend to kind of focus in on figure out how can we distill
4:11
this extract this into a smaller model within these kind of
4:15
smaller models the process? We generally, think about just
4:18
less around, just the dollar cost more in kind of like, where
4:21
can I run this? And for us, the golden threshold
4:24
has been, can we make this model run on a single?
4:27
A 100 GPS are 810 GP or on any type of CP requirement because
4:32
like when you moved from the a It's to an 8/10 say you're
4:36
serving AWS that moves you from having to pay 40, bucks an hour
4:40
because you have to get a tat them at once to on the antenna
4:43
you can pick up a spot instance for 30 cents and so like at that
4:46
price point you can do pretty much anything and then if you
4:48
even move it to CPU, the game changes even further where you
4:51
can move to billions of items without necessarily even
4:53
thinking about this cancer. A lot of our work we're moving
4:56
like first more semantic search, these are small, queer encoders,
5:00
these are 30 million 40 million 100 million parameter models.
5:03
And some that case, We can actually run the query encoder
5:07
on CPU, so it sits in the same system that's actually doing the
5:11
retrieval and so that can just simplify production.
5:14
It just a question on really, like, figuring out what scale
5:16
you're going to, what is your latency, throughput that you're
5:19
looking for and what is precise? And one of the things that we
5:22
saw was the foundation, IEPs are actually really slow and
5:25
partially, that's actually really good.
5:26
Because it changes your perception where it's like,
5:29
before people's perception via Google search was like anything
5:32
over 100. Milliseconds is terrible.
5:34
Now People are used to eating three four, five seconds.
5:37
And we've been able to see that we can take.
5:40
If we're running toe here, we're running open an eye.
5:44
What's it called? So it's very anthropic there on
5:46
the order of like three seconds for a response for single batch.
5:49
We can get a model that is 20 billion grammars to respond in
5:54
400 milliseconds. So that like meth Improvement in
5:58
speed is huge, just bring it in-house.
6:00
And that's just because you don't have to go on the
6:02
internet, it doesn't have to go through their orchestration, you
6:04
can Deal with your own. And then the other thing is,
6:06
when you bring this in the house, you really become a
6:08
master of your own fate, where one, any type of latency
6:12
optimization is actually all about you.
6:14
You don't have to worry about API rate limiting, you don't
6:16
have to worry about like, their entire system going down.
6:19
I think that was actually one of the things that really pushed
6:22
neither to bring these things in-house because sure opening I
6:26
do ARP has 99 percent uptime. That's 99 percent for every one
6:29
that was not what we're experiencing at all.
6:31
And when we're trying to deliver this product and we're like
6:34
well, This use case doesn't work because their overall systems
6:38
down in, we can do anything, but sit in our hands, it really gave
6:41
us kind of like the impetus to bring this in-house.
6:45
Now, we experience the same issues.
6:47
We also had regular there were regularly not available
6:50
sometimes 50 minutes at a time, so there was a problem.
6:53
Okay, anyone else on cosmetic surgery?
6:56
I can I can I just jump right in here to say, Amen.
6:59
Like I love it was just what he just said more like, this is
7:02
exactly our point of view is that it often you don't need the
7:05
highest-end harbor, right? So you can live with Hardware
7:07
that actually hits you or latency and throughput
7:09
requirements, you know, even though there seems to be a
7:12
strong let's say Shortage of high-end silicon.
7:16
Many people don't realize that they don't probably don't need
7:18
high in Silicon to deploy their models.
7:20
And, you know, basically what dr.
7:22
Now offers exactly what you talked about, but in the general
7:24
way in a fully automated platform for broader users to
7:28
deploy their own model, their own custom model or another
7:30
source model in a black community control automatically
7:33
choosing the silicone for you in turning models into high
7:37
performance portable artifacts and code that you can run on
7:40
different Hardware. I just want to say you know we
7:42
should definitely follow up after this because Is our real
7:45
value here is 100% aligned, so many panels will need to
7:48
disagree with each other, so somebody should disagree with
7:51
with him to make decisions thing, right?
7:53
So you can disagree about the most promising ways to go about
7:58
cost reduction. So if you each had two name,
8:00
like your top two methods because you can do so much
8:03
right, if you, if you start reading on the internet, what
8:05
are the top things to do to reduce cost?
8:07
You have a list of like 20 items so maybe you can disagree on
8:10
that. Like that, that means if you
8:12
give me each your top two, okay? SOC.
8:14
Something has a to first of all, make pick a model is visiting
8:18
that you need an optimized as much as you can because then you
8:21
have better performance in a given Target and then second
8:23
choose the right silicon for the one that has the lowest cost
8:27
possible in hits your performance requirements.
8:30
So you're also saying a different model type like a
8:32
smaller type of model. Exactly.
8:34
Yeah. The right model optimized as
8:36
much as you can hopefully automatically and then to take a
8:39
ride harder for it. Yeah.
8:40
My two things would be kind of like structured kerning and
8:43
knowledge distillation. Where You can take the large
8:45
model and then you actually just remove portions of it until it
8:49
fits in whatever systems you have in.
8:50
Like, I like direct example for this, like, we're serving a flan
8:54
T5 11d player under model that physically doesn't fit in an
8:58
8/10 GPU until you remove at least 60% of the way.
9:02
So you have to just because like the actual size of the model,
9:05
once you've done that, you can make that model fit in that
9:08
small and GPU, but then you have to use some form of knowledge
9:11
distillation. So the model doesn't actually
9:13
have this like Icing quality. In generally speaking, those two
9:16
things together really provide like big big performance
9:21
improvements and get really crafty with your distillation.
9:24
And I used to work at narrow magic, which focused on
9:27
distillation for CPUs and everything else and we were able
9:29
to get 30 40 50 x improvements on text text classification test
9:35
by basically just pushing in that like knowledge,
9:38
distillation is far as we could with the biggest teacher model
9:41
in the smallest student model and that just works extremely
9:43
well. The downside to it, I'll take my
9:45
own going against myself the other corner.
9:48
There's a non zero cost actually moving all these things
9:51
in-house. Like if you think about the
9:52
beautiful part about these foundational models is the
9:54
feedback cycle is super fast because if you think about like
9:57
saying you're doing something with the prompt for search
9:59
engines, you want to rephrase how your system works.
10:03
And so you just change the prompt and it runs it's good if
10:05
in production is good to go. If you want to do this in-house,
10:08
you basically say you're like, you want the system to take a
10:11
more neutral tone, you have to go through all this, iterative.
10:14
Approach to make data sets to store this data set into
10:17
something smaller. Take a small compressor into
10:20
something smaller, figure out how to boost your production.
10:22
So like your iteration time goes from being on the order of
10:25
minutes to potentially days weeks.
10:27
And that's kind of like this place.
10:28
Nice trade-off of, like, how much do you want to explore?
10:30
How quickly you want to be able to iterate on products, with
10:32
customers to? How quickly do you want to be
10:35
able to optimize him control your own fate?
10:38
It's running and house. What do you say, what size of
10:40
team do you need? Just for the upside of Running,
10:44
Man, A few models, let's say to kind of complete.
10:50
Guess it depends. It depends.
10:53
What type of model probably. But just maybe.
10:54
Can you give listeners like a ballpark on the on the amount of
10:58
work that you had to do? So I would say the bringing
11:01
these models in the house and making models there was roughly.
11:05
There's about four of us that were working on anything from
11:07
data modeling to. I'm the compression guy.
11:10
So that's what gets me up in the morning, but there's other folks
11:12
are working on the modeling side and otherwise, I think, Overall,
11:15
it Neva directly working with models, have probably around ten
11:18
of us. Okay, and Mario, dark thoughts
11:23
on the cost side. I can maybe you like few
11:25
thoughts on how you can control cause before you actually move
11:30
anything in the house. So we don't really like, we
11:32
don't train or find, you know, limes ourselves.
11:34
So right now, we are pretty much on the consumer side just going
11:38
already apis, and it's tricky. It's expensive.
11:42
And I think, even the Vinci can be a very expensive model,
11:46
depends how you use it, and most of the time.
11:50
Like we just first do pretty much visibility exploration of.
11:55
Is it possible to do what we want to do in some like really
12:00
isolate the cases? Like can we summarize one
12:03
conversation for example and to let yourselves that capability.
12:09
We would take the best model first probably similar, that's
12:12
possible. And let's try to go down down,
12:14
you know, just try to Pare down the model to the lowest model
12:18
that gives us results are okay. Okay, like Daniel mentioned
12:21
that's often like I think producing games 25 years ago
12:27
where everything is too slow and you are totally like limited by
12:32
how much you can process and kind of like Cuts both from like
12:37
cost of latency perspective like you just can't summarize all the
12:40
conversations in the come because will bankrupt
12:43
straightaway, there's like too many of them.
12:46
So but like one thing that's one thing that's interesting.
12:50
More like cost reduction is just basing engineering thing which
12:53
is just call it class and that's not always possible.
12:57
Of course I think like probably like in Danielle's case an Evite
13:00
that's little more tricky. You're doing semantic search.
13:02
But like the systems we build are usually a complex set of
13:08
things that talk to each other. And there are multiple.
13:13
There are multiple components that are implemented with LMS
13:17
and sometimes you can put like a really simple classifier.
13:20
Fire before doing something to figure out, whether it's even,
13:23
really would make sense to call along at all.
13:26
So that's, you know, that that's one technique that I think can
13:30
work quite well, because not all tasks are some things you can
13:33
find the tool. That's really simple and is
13:36
going to be able to say whether the more complex and smart tool
13:40
is going to has, has any chance of finding an answer?
13:42
Yeah. And I just a cup Mario's first
13:45
point there that like the first step.
13:47
Yeah, you could do these bringing And bringing it locally
13:51
or whatever. But if you're using these large
13:53
Foundation models from open AI or whatever, the first step is
13:56
just knowing what's going on just building virgin one and
13:59
kind of see understanding how the cops works.
14:03
And this is like, it's a bigger step than you'd expect and just
14:07
understanding which prompts in your system, which parts are
14:09
expensive and like knowing where to go from, there is a pretty
14:14
important actually. So just echoing that otherwise.
14:17
Yeah. So this is some great tricks,
14:19
you can employ. When when starting off and then
14:22
maybe later when you scale to bring it in the house, okay,
14:25
let's talk about latency. It's okay.
14:27
You said one more thing and you would think that I think was
14:30
supposing places but I do for said explicitly was you know
14:33
there's plenty of Open Source ready to go models that can do I
14:37
think Daniel mentioned in notified flying, for example,
14:39
can we find for users and people realizing that you don't let you
14:42
do not need to pay the open-air? Cohere tax.
14:46
If there is a use case that you know the open source model can
14:49
do well Any kind much more efficiently in the platform that
14:52
you control that you might not need to do it on your own that
14:55
they're already gives you a very very significant rule for cost
14:58
savings there, right? So just know your use case.
15:00
You don't kind of like this analogy of user CPU because it's
15:04
General, but a CPU is expensive but then use the GPU to the book
15:07
of the computation because that's more efficient.
15:09
I think this. Is there a chronology here?
15:11
Where you have these you know General expensive models like
15:15
upt for that have to pay to use but then for things that, you
15:18
know, that you want to do and you Can specialize it and use an
15:21
open model. You probably going to pay a lot
15:23
less by probably orders of magnitude and just to make it
15:25
interesting and push back a little bit.
15:27
I'd actually say that please the way I see it.
15:31
If you're if you're I mean it depends where you are in the
15:34
stages of development. If you're building burning one,
15:36
if you're trying to ship something we're honestly most
15:39
people working with lme's today are trying to ship version 1.
15:42
I'd say don't bother with open source models, get something
15:44
working on PPT or some person. Yeah yeah, yeah.
15:48
But what just like a tissue Writing code here, you write in
15:51
the CPU, you make sure it works and then you'll float to the
15:53
efficient compute unit. That does what you need to do
15:56
similar thing here. You start simple with the model
15:58
that, you know, is more expensive.
15:59
And as you understand what you actually need from it, then I'm
16:02
fairly confident. You can find a local Source
16:05
keeper multitude, does what you need to do, right?
16:07
Question-answering summarization, and stuff like
16:09
that. I think the part that's
16:11
important to consider the cost reduction.
16:13
Also, just his Staffing a team like the star have here is I've
16:17
always worked in like the search World of like, big search
16:19
engines. So There's always been a lot of
16:20
money behind this. There's teams everything else.
16:22
My brother used to work at fine Tech and at some point, they
16:26
were looking for something that could classify businesses.
16:29
And when they poked around and figure out like what would this
16:30
cost to have our own system? You have to hire people.
16:33
You have to hire these 1ml engineer, probably infer
16:36
engineer. You have to run it in pretty
16:37
much. Their bottom Barrel costs would
16:40
be on the order of five six hundred grand a year just for
16:43
like the team and they can make many 9% like what Alex was
16:46
talking to them. But in reality they just needed
16:48
something that was 17. Share, and like you say, just
16:51
plug them into a system. They never had to hire an LP to
16:55
end. Never had an MLT.
16:56
They had the system that was beautiful because like they
16:59
could go in and someone who could just tune The Prompt until
17:02
it works. Well enough.
17:03
And it just like, by using this large system, they save money
17:07
because they didn't have to hire expensive people like man and
17:10
it's an understanding use case because they were only pushing
17:12
10,000 examples through today. So like their cost is not as
17:16
High versus on like a search engine side.
17:18
Or when you're like doing all the Ian's of intercom has like,
17:21
if you wanted to do all the summer is then that kind of cost
17:24
calculus change. It's a question of scale.
17:26
So clearly if you, if you use it, like the x0 plus type of
17:31
thing with a human in the loop, then using a z, i will be always
17:35
more cost-efficient. It's a good point.
17:39
All right, everyone, I just want to jump in real fast because I
17:42
have a huge announcement. We will be having a part two of
17:46
our large language models in production.
17:48
The first conference was Success that we had to follow it up
17:51
because there were so many incredible speakers that reached
17:54
out after the first one. I couldn't help myself.
17:56
So it will be on June 15th, and it is going to be online.
18:02
So anyone from anywhere can join, but if you thought the
18:05
first one was good, then boy. Are you in for a treat?
18:08
We have some confirmed speakers already.
18:10
I'll just go through a list of them, Mac tape from data bricks,
18:14
you know, the founder of data, bricks, and Spark, and all that
18:17
good stuff. We recently just had him on the
18:19
pod. He's coming back to Keynote.
18:21
You've got chip hewn, who is an amazing person and incredible
18:27
follow on social sharing. So many insights and has been
18:30
dropping knowledge bombs when it comes to large language models
18:33
recently, you've also got Scott McKay from MIM dot AI.
18:39
You've got the Deep speed team is gonna be there from Microsoft
18:43
that is pretty epic and Sahar more from stripe.
18:47
Another stripe alumni is Emmanuel.
18:50
Send he's formally of stripe. Now he's working at one of these
18:53
large language model companies that you've definitely heard of,
18:55
but I don't think I can spill the beans on yet, and we've got
18:59
Lance Martin in as from number station.
19:02
George Matthews from Insight Partners.
19:05
This is just a glimpse at the whole list.
19:08
Go sign up. You've got all the details in
19:11
the link in the description. Huge shout out to the sponsors
19:14
redis for showing us some love and making sure that this event
19:18
happens. Our Gala Any scale, continual
19:21
chroma. Thank you all.
19:23
And you can expect all kinds of bad musical improvisation.
19:27
The brakes, like I normally do it.
19:30
And who knows? I may make that I hallucinate
19:33
more than chat GP t-shirt available for sale again.
19:37
Check it all out. Links in the description, will
19:39
see you soon. Okay, so moving on.
19:43
Which is another. More.
19:48
You know, spoiled from normal classification models but just
19:51
moving into the LM space and seeing that.
19:54
Just one token takes 3.7 seconds on P90 was shocking and then you
19:59
normally you need like a whole sentence so that can take
20:03
several seconds. And how do you deal with that?
20:06
If you have a user on the other hand who was like yo I'm really
20:08
very impatient things after 1. Second application is broken,
20:12
what are typical? How do you go through the life
20:14
cycle? When you start, you start out,
20:16
maybe not looking at speed. What?
20:18
Of speedups can be a possible to expect.
20:23
Do you do this trade-off between the model size and speed or
20:27
like, how are you thinking about this, and maybe also throw your
20:30
best tips? Like, if you had can do one or
20:32
two things on speed, what you do.
20:36
If you don't have a lot of time, literally, I can walk through
20:40
directing that we didn't even we've talked about is our
20:42
initial summarization model and initially use a large language
20:46
model, you get the summary. Basically looking at about three
20:49
seconds per item and we're doing a batch of 10.
20:52
So we're basically 30 seconds you we did this and figured out
20:55
that like oh turns out this is actually used when very good
20:58
entering her prompt and then brought this into a smaller
21:01
model house just doing in to a T5 large model, but serving that
21:04
on like a bad size of 10 on the A-10 still took like eight
21:08
seconds naively. So we kind of went about all
21:10
these ways. So, first off, there's a great
21:12
Library called faster Transformers, Nvidia, kind of
21:15
supports that they don't fully support it so like you Fiddle
21:18
with it a little bit, but just moving from native python
21:22
serving to faster Transformers, basically brought us down from
21:25
like 8 seconds to like 1.8 seconds.
21:27
So huge games just on changing. How it served it worked on the
21:30
GPU kind of funky build process, but it works super well.
21:33
And then from there, we threw in some notion of structured
21:36
pruning and asymmetrical spurning on the encoder.
21:38
Decoder side, just because the encoder produces this conceptual
21:42
representation on a sequence sequence model.
21:44
And you want the best conceptual representation possible, and
21:47
also only runs once said of Costas pretty much doesn't
21:49
matter version of the decoder runs again and again and again.
21:53
So we heavily compressed the decoder.
21:55
So we basically took this model that start off with twenty four
21:57
layers on either side and brought it down to twenty four
21:59
layers and one side and four layers on the decoder.
22:02
And that meant that, like, by the time that we were done with
22:04
faster Transformers, this compressed model we got down to
22:08
roughly, like, 300 milliseconds per batch of 10, which that
22:11
allowed us to basically say, hey it now will cost 20 thousand
22:14
dollars to create a summary for everything in our index, and we
22:18
Actually do it because we like decrease these costs versus when
22:21
it was 10 seconds. 30 seconds and think just not track.
22:26
Yeah, I guess I just wanted to underscore making sure to use a
22:29
very best kernels. A very best binaries for the
22:31
given harder. They running it on because it's
22:33
a few can do without changing your model at all or any
22:35
batching parameters. Any other Supremacy start with
22:38
that and see how far you can go, you know, if you the problem is
22:41
that this is dependent on the hardwood Target very specific
22:43
than the harder Target. So the more you can automate
22:45
that the better they'll put a plug in for what I've come out
22:48
does. We can help you search
22:49
automatically for what's the right library or compiler using
22:52
tensor, t or T V M or, you know, could be n for NVIDIA or the
22:55
equivalent for all the harder targets.
22:57
Then after that, you know, change the batch size.
22:59
Sure. You might pay a little bit more
23:00
harder to decision, but its latest really matters to you
23:03
reduce the batch size that a little bit more.
23:05
So, I'm really happy to hear. Sorry.
23:12
I really like what you're talking about because like if
23:16
you're not real they're just talk like talking to some
23:18
external API. Your hands are so tight.
23:21
So effectively, you have like two ways to improve.
23:25
The latency one is by shared capacity.
23:28
If they if they can give it to you and then, you know, you can
23:31
play a bit more. We'd like to put latency curve
23:34
and like, very want to sit there and the second one is, maybe
23:38
like what I said like if you're okay with, not finding an answer
23:41
and if you can detect It quickly if you just don't call it and
23:44
return the answer straight away like they are you expert dimes?
23:47
I think it's becoming quite for like chat applications.
23:51
You know, just piping the tokens directly from LM to the user.
23:55
So user is, you know, seeing straight away that something is
23:59
happening. But sometimes, dreaming do that
24:02
streaming effectively? Yeah, like some like directly
24:05
from lm22 to the end user experience.
24:08
Like sometimes you want to check, what's the answer one?
24:11
It's full Fully generated. And in these cases you have to
24:16
wait for it. And for example, if we're GPT
24:19
for shared pool of capacity like the the latency I'm seeing is
24:24
something like 100 ml per token. The tone for English is usually
24:29
one of our per token, they change the tokenizer 4254, but
24:32
it's extremely high. So I'm really, I'm really happy
24:35
to hear what Daniel and those are talking about because I hope
24:39
we'll be able to actually, you know, move some of these things.
24:42
Like once you understand like we choose cases are actually the
24:45
most important for us and which use cases make sense to to
24:48
optimize and bringing house. There's some tricks you can
24:51
still do on. When you when you call an API,
24:54
you can work with a prompt because the main database is
24:56
actually in the output, not in the size of the input.
25:00
So you can, you can tell it to because ice or give you back a
25:03
number of paragraphs, like be a bit shorter and the output.
25:08
So this is something I mean, we can clearly see from the panel
25:11
that If you want to be really fast, you need to move it in
25:13
house. But let's say you're in this
25:15
first stage where you still need to understand the cost and the
25:18
speed, there's a small amount of trick like semantic hashing and
25:22
what Mario mentioned that you can do and maybe Jared you're
25:25
also have a dashboard where users can track latency they
25:29
observe, is there some insights? You can share?
25:31
Like how do they like what You observe in terms of you users
25:35
with respect to agency? Yeah, I guess the interesting
25:38
thing here yet is, what is the low hanging fruit to kind of?
25:42
With latency you mentioned one is obviously like how long the
25:45
responses but the other one which is also very obvious but
25:49
falling back to different types of models.
25:51
So obviously, if you can detect, how how badly you need to keep
25:56
it easy for over 3.5, if there's use case, where you can fall
25:59
back to 3.5 a faster model or even something kind of left,
26:03
although that might be the fastest what they have now.
26:05
But or maybe Claude or something like that, kind of, there's
26:09
becoming in fact, there already is probably This whole axis of
26:13
which engine do I use for this problem as opposed to kind of
26:17
just saying we're using gbd4 for everything or we're using our
26:21
in-house for everything. I think these are all tools that
26:24
have pros and cons and one of the pros and cons of each has
26:27
Lanes, I can't resist bringing up again, like 11 thought it was
26:31
mentioned very briefly that streaming the output like I'm
26:35
more on the user side. These days interacting with his
26:37
boss they go to buffer the entire answer before they show
26:39
it to me. A simple thing that just Human
26:42
decency to say it's being typed, or streaming it to the user,
26:45
because then you see the something's happening, right?
26:48
So he attacks by slack bot, Siri buffers entire album, before
26:51
shows it to me. Like if you just said, hey, the
26:53
boss is typing, just let this GPU work hard for you that we
26:57
already go a long way. So don't change the latest just
26:59
manage human expectation. They will probably go a long way
27:02
as well. That's such a good point, the
27:04
actual UI experience and there's some drawbacks in terms of like
27:08
safety of the response like if you stream it, you Uncheck the
27:12
response so that could be a concern for some type of
27:15
applications, but I like your second idea of like typing.
27:18
Like when you see someone typing in some different types of like
27:22
a human I don't know. Different shared applications
27:24
that helps you see they're typing.
27:26
It's a good idea. Okay, I think because I had real
27:30
quick. Yeah, I think there's a lot of
27:31
ux ways to do this to solve this problem.
27:34
That's one thing. And then also just one of their
27:36
addition is these latency amounts of like keeping
27:41
dee4diesel. Insane amounts of the models
27:43
being offered are not constants either.
27:46
We've like been looking kind of at our high level like user data
27:49
and the lane season like very spiky of things like even before
27:54
and like is very laggy. Some days and not other days.
27:56
So, if you really care about optimizing, that's probably
27:59
another thing to look at. Yeah, that's definitely an
28:02
issue. They don't support any essays
28:04
yet, so I guess that's hopefully soon to come, well, be having a
28:08
latency page to kind of help with that soon.
28:10
So stay tuned. On something we talked about
28:14
earlier, I meant he's like that sizing is hugely important both
28:17
for like the streaming component.
28:19
But also, if you think about like, when you're actually doing
28:23
this, like greedy token by token decoding.
28:25
If there's any difference in the length of each batch, everything
28:28
basically defaults to the longest bench.
28:30
So like there is some literature out of Washington, I think that
28:34
in machine translation they found those like 70% of tokens
28:39
in batches were effectively used as tokens because he was driven.
28:41
By the longest padded sequence and so like, especially if you
28:45
have a high variability in your outputs doing anything but a
28:48
small batch size King like be bad because it's like if you
28:52
have a short response you don't get to enjoy that.
28:54
Basically you're all your responses are limited by your
28:58
longest output. Hmm.
29:01
Make sense. Do you have?
29:03
Do you control for that and your prompt somehow or do you?
29:06
How do you handle that the control of a larger size?
29:10
Okay, on our side we've done a bunch of optimization there even
29:14
as you mentioned on like making the gbd3 up a more concise
29:17
percent we kind of can just do them but it's really just kind
29:21
of fiddling around with it. Like one of these things that we
29:24
saw was with this kind of asymmetric pruning that we did
29:28
when you move the batch sizes larger, all the gains went away.
29:31
Way. And that's because like one of
29:32
those smaller you really could like highly optimized it because
29:36
you're not waiting on this decoding versus when you have a
29:38
large batch size, you're just basically dominated by the
29:41
longest sequence of all your compression is completely kind
29:44
of just broken. That makes total sense.
29:47
Okay, let's talk a little bit also about monitoring.
29:50
I'm very curious what you guys found, useful both like human
29:55
base and I am based modeling any other type of techniques that
29:59
you found useful Can jump in here with like a high-level
30:03
overview and I know talk to Mario personally about this.
30:07
I'm sure he has some things to add to but kind of what we've
30:10
seen our users doing kind of breaks down into three big
30:14
categories. First category is, which I think
30:16
is the real source of Truth is end-user giving you a thumbs up
30:19
thumbs down or you can use behavioral-based maybe like
30:23
clicking refresh or closing out is a thumbs down as a negative
30:26
signal on like how the prompt is doing.
30:29
That's one category. Another category is this This
30:31
whole mturk hiring out, click workers to do, very boring, very
30:36
boring way to do it in my opinion, the third category,
30:39
which I think is really kind of where the future is here, is
30:42
synthetic evaluation. So how do you actually use an
30:45
llm to rate how good it is? And the one interesting thing
30:50
I'll add here, which is kind of a thesis.
30:52
We're developing based on our conversations at prom player is
30:56
kind of via - is the right way to do this instead of trying to
31:00
understand. Which prompt you have how good
31:03
they are. It's better to reverse the
31:05
problem and say, how can I evaluate what's failing in
31:09
Prague which users are getting bad results.
31:12
Like is the chatbot being rude as kind of fit the most trivial
31:15
example of this. But that that's the way we see
31:17
it. Yeah, to be direct on that that
31:21
were calling this critic modeling, and it works extremely
31:24
well, especially with the most recent models we found that.
31:27
So, like, on a seven-point scale, The GPT for is f by an
31:33
average of one label, which I used to do a bunch of human
31:36
evaluation, that's pretty much the best.
31:38
You're going to get off. Like a train judges, and it
31:41
works across fluency, accuracy other things.
31:44
The one thing here is their huge positional biases.
31:47
So, we found that if you ever try to compare two things, the
31:50
outputs don't work as well. So it's best to try to focus on
31:53
individual comparisons. And then potentially using some
31:55
like, wheat supervision signal to figure out pairwise things.
31:59
Can you give us a concrete example of a new me?
32:01
Critic. Like what is the problem you're
32:03
giving the critic or how does it work?
32:05
If you're writing a thing as the engine, you were tasked to
32:09
evaluate how good a summary is for something.
32:12
You are evaluating Anna terms of accuracy.
32:14
Something that is completely accurate has X characteristics
32:17
as a seven, something that is slightly missing.
32:19
Something this is six provide all your description of labels
32:22
give it to the model. The model will respond.
32:24
Pretty damn well. But I will say like the caterer
32:27
don't try to say hey I have these two summaries which is
32:30
better. Or because the model will
32:33
there's position bias. Depending on what you put first
32:35
or second the model will favor with last.
32:38
And at the same time the outputs aren't as usual is kind of like
32:41
the old Costco thing where you never ask someone why they
32:44
prefer something. Because if you ask them to
32:47
justify it, then they will not actually give you their
32:50
Preference. They will give you an answer
32:52
which they can justify. So don't ever ask model to
32:54
justify. Okay, just saying recency bias
32:59
is also known in human decision-making by the way.
33:01
Like, if you give a man, like a verbal statement of choices,
33:06
people just forget very similar to what we observe in the
33:09
network. The first option, intend to pick
33:12
some of us and decision design. Very interesting Behavior
33:14
research. We got a bunch of stuff in house
33:18
and we found someone to a Jared mentioned.
33:20
We're like the tail Laden sees can get very out of hand where
33:24
you can see, massive spikes, you can be caused by.
33:26
He's weird Corner cases that you never thought of like we were
33:30
seeing massive tail agencies, occasionally and that turns out
33:33
we were running a model on a sliding window and occasionally
33:37
hit super long documents. And like the solution was like,
33:39
oh what do you do? Just truncate after 10,000
33:42
tokens but sometimes in the only way that you'd see, this is
33:46
like, you look at what these outliers are.
33:48
And in most cases, any time we've seen an outlier, the
33:50
solution is super simple. It's just like, oh, we didn't
33:52
know about this weird Behavior and the easy fix we no longer.
33:56
Have that I cut my call. This terrible have a worse name,
34:00
but I call this terrible experience debugging, which I
34:03
regularly do. I wasn't recommender systems.
34:06
So also money toward like the really bad experiences and they
34:09
just picked one every week to Deep dive on the root causes
34:12
very similar to you saying like, monitoring the bad experiences
34:17
like the edge cases elongated see and then you'll find like
34:20
groups of problems that you can fix one over one.
34:23
Okay, cool. I think we're out of time.
34:25
You spent one more thing if you don't mind.
34:26
So putting a plug you come try to octal male profile at you
34:31
with a single line of code, you can get your mother running
34:33
across even harder in the see, the profile of the latency takes
34:36
four, different options is going to give a very actually running
34:39
the modern provided for you. So love to hear from you soon.
34:42
Yes, please try out. The interesting components are
34:46
dangerous to hear from others like Bravo and chat like how
34:49
they deal and labeling becomes too expensive as well.
34:53
If you need like you before for labeling then you know it's
34:55
expensive again. N.
34:57
So whether people have like some smart sampling techniques or
35:00
maybe like pre-filtering initially are there Solutions
35:03
out there for that. It seems so nascent field to me
35:07
that pretty much like all the tooling is just in the process
35:10
of building. So yeah, I'd be happy to hear in
35:13
chat of like what people use for things like that.
35:15
There you go. Oh, that's awesome.
35:17
So we will continue this conversation in the chat.
35:22
Yeah please let us know what everyone is thinking.
35:26
And And thank you so much. All of you were incredible and
35:30
these are like, these are very poignant questions and I really
35:35
think that you all educated us a ton on how you're going about it
35:40
and what kind of best practices you found thus far.
35:45
Wow. Alright.
35:47
Thank you later. You all off speed.
35:49
See you later.