Latest / Agentic Conversations (formally mlops.community) / Large Language Models in Production Round-table Conversation

Transcript

Large Language Models in Production Round-table Conversation · Agentic Conversations (formally mlops.community)

0:00
Hey everybody, super glad that you are listening to this large
0:04
language model in production Round Table.
0:06
I want to mention before we jump into it, that because of the
0:10
success of this Roundtable we ended up organizing a full-on
0:13
conference. It's a virtual conference.
0:15
It will be held on April 13th and you can register now, we're
0:19
going to have incredible speakers.
0:21
Like the founder of human Loop, the creator of Lange chain, just
0:27
to name a few. And also the CTO of Of adapt.
0:30
AI. They just raised two hundred
0:32
fifty million dollars. Wow.
0:34
All right, so if you want to register for that, check out the
0:39
link in the description and we will see you there on April
0:43
13th. I'm not going to chat for too
0:46
long. I just want to talk real fast
0:50
about the survey that would just put out in ml.
0:54
Ops Community, this survey is kind of like, playing off of
0:59
what we're about to talk about. And I'm fascinated by like,
1:03
using large language models in production and how people are
1:07
doing it if they are doing it. And if you're not, why are you
1:10
not? So please feel free.
1:14
You fill out that survey? It'll take like literally five
1:17
minutes. It's the most basic thing ever.
1:19
And that is my like one call to action that I have for everyone.
1:25
Before we jump into this great conversation, I'm going to hand
1:28
it over to Diego, man. I appreciate you running this,
1:32
so we will talk soon. I'll be back in a bit.
1:40
It said that everybody here. Great subject, great amount of
1:44
experts. So excited about the next hour,
1:47
we're going to be chatting about large language models.
1:49
So I figure we kind of start really quickly with a quick
1:53
lightning round of introductions, a Rebecca.
1:56
I will start with you, everyone. Yes, you were excited to be here
2:00
about this discussion. I'm Rebecca, I'm a research
2:03
engineer, Facebook, AI research. And so, I work on primarily an
2:08
LPR focusing on the robustness of Large language models.
2:11
I also has worked on during us and responsible, a are aspect of
2:18
MLP as well as foot by you. David.
2:21
Hey gives everybody a vice president at an usual Ventures.
2:26
Be fun. I focus our investments in ml so
2:29
I've been thinking a lot about language models and spent before
2:33
that like six years working on in my lab so across a couple of
2:37
startups techcom and determined AI.
2:39
Yeah, Percy was at Ford where I did ml platform stuff, like,
2:44
like a lot of folks have talked to in the community.
2:46
So honest. Hi.
2:48
My name is honest, I'm a machine-learning engineer digits
2:51
at the great chance to write machine learning related
2:54
Publications, one about natural language processing and then
2:56
about the Emma Ops and we use Lander block language models in
3:00
production to the cystic album. So what digits is doing is like
3:03
basically your real-time engine to understand books for business
3:06
owners and accountants and basically no time and we Those
3:10
language models to help start the conversation between
3:15
accountants in the business owners.
3:17
So we can talk about the details later, I thought and James,
3:21
everyone, really started. To be here.
3:23
I'm James CEO and co-founder of Bountiful and we're building a
3:29
monitoring and testing tool for foundation model workflows.
3:33
So one tool for FM app developers that allows them to
3:38
actually observe and then test A cost quality and latency across
3:43
their workflows. Excellent.
3:44
Great. And I'm Diego, Oppenheimer.
3:46
I'll be your opinionated moderator.
3:48
Today is to give you, I'm currently serve as a partner at
3:53
factory HQ, which is a venture fund that specializes in the, I
3:57
Investments. Previous to that.
3:59
I was an executive running on the lobster company called being
4:02
a robot and previous to that. I was the CEO and founder of a
4:05
if Mia which was one of the first ml up companies.
4:08
I know this is lucky enough to be part of of this envelopes
4:10
Community. Since I believe there was double
4:13
digits of people in the community.
4:16
So it's been really, really exciting.
4:18
Time to see all the hard work that the Matrix in Peril and
4:22
etcetera and friends have put into this guy.
4:24
Because well, what size are we at now Dimensions?
4:26
Because it's a like cross 15,000, right?
4:29
Yeah. It's around 15K, it's wild,
4:31
easy, amazing. So, I'm really excited.
4:34
So a lot, a lot of hype around, Chachi PT and kind of what these
4:39
large language. His models look like and so I
4:42
figure it out. We could start with grounding on
4:44
like what do we actually mean by a large language model?
4:48
I'm going to pick on Rebecca, on this one, to kind of Define,
4:52
what's the proper definition of what we would look at as a large
4:55
language model? Sure.
4:56
Yeah. So this is a bit of a contrarian
4:58
opinion, but I think this is actually really poorly defined
5:02
and this is just a new term for, you know, what we can say is
5:05
transfer learning which is actually existed for decades,
5:08
but I can give people a bit of an Overview for those who aren't
5:13
as familiar with the background and history of them all to
5:15
starting in the 1990s. This is when people start
5:18
talking about machine learning and artificial intelligence, and
5:21
this was a period of smaller statistical models.
5:25
And graduated, in 2010, we start seeing deep learning.
5:29
So down the Coon coined, this term.
5:32
So we had larger datasets and we're computation.
5:35
We see higher level teachers emerge through training on these
5:38
data sets. And so really this The process
5:40
of homogenisation. So rather having bespoke feature
5:44
engineering pipelines for each application.
5:46
Like you see that you could have the same neural network will
5:49
deep neural network architecture.
5:51
Use it for for a lot of different applications and this
5:54
process of homogenisation has continued and that's really what
5:57
we see leading up to today. Where we have a really powerful
6:01
large language models can be used for a variety of
6:04
applications. And so it's a, another
6:06
inflection point was 2017 when we Add the transfer, where's
6:12
paper? Like the attention is all you
6:13
need paper. And then in 2018, we saw the
6:16
Bert model. So then he's you know, the years
6:20
for 2018 on till now we just saw so much like exponential growth
6:25
and LLP mostly happening on the research side, but now on the
6:29
production side as well. So it's been super exciting time
6:32
for LLP. You know, 2018 we had Kurt and
6:37
then 2019 we have the Roberta paper.
6:40
Which showed that as you train with architecture, but will Big
6:44
Ten eyck do this? That you actually can be pretty,
6:47
pretty much like all the tasks and glue Benchmark.
6:50
And then we just saw, you know, this is this is like what showed
6:53
that a scaling matters and scaling is really powerful and
6:58
then we just saw this influx of, you know, money and resources
7:02
being poured towards training, large language, models
7:05
sweetheart, Bert large or 340 million parameters gbtb2 with
7:09
1.5 billion. L T5 with 11 billion GT3 with
7:13
175 billion which I imagine a lot of people here.
7:16
Probably you think for your apps and and now we have like really
7:21
powerful as from opening on that we can use and I would say that
7:26
this is more of a succession of just us.
7:30
Moving from very specific models for specific tasks to large
7:35
pre-trained models that we can use for a variety of tests.
7:39
So yeah. Say that this idea that's really
7:41
Central to Foundation models. Or large language models is this
7:45
idea of transfer learning which is that you can train a model on
7:51
a lot of information and use it for a variety of Downstream
7:54
test. So instead of having like one
7:57
model for sentiment classification and one model for
8:00
question answering and when model for maybe like breaking
8:05
your ads you would just have one large pre-trained model.
8:08
That's very powerful and you can You can just apply it with very
8:11
minimal changes to a variety of accounts from tasks grow.
8:16
So I guess to clarify a couple of things we look at it as you
8:21
know large language models is a subset of foundational models
8:25
regular. I think foundational models in
8:27
general would actually apply to not just language models but
8:29
also kind of like image will be a model whatever's coming out.
8:33
There's some characteristics to these large language,
8:35
foundational models. I think in generally we can call
8:38
them that their no code. Terms of like we access them the
8:42
natural language which really expands the ability for people
8:47
to use them, right? I mean, I think that's really
8:49
what's one of the biggest things that's inspiring?
8:51
The second kind of what you mentioned.
8:52
Rebecca, they're kind of, you know, there are no tasks, right?
8:55
So they're actually generally, you're pretty General, they're
8:58
not task-specific and they generalize quite well.
9:00
And I guess we're still figuring out to what other tasks they
9:04
generalize on a daily basis. And in one thing that I know of
9:07
that, which is, I think also really interesting is that they
9:09
don't really see. Over from a traditional ml cold
9:12
star problem, right? Which is like you can actually
9:15
start working with them without any sort of like pre-trained
9:17
data. So one thing and I know you said
9:19
you had a contrary view by like one of the things that have been
9:21
like thrown out there is like what do we consider large?
9:24
And it feels like some people say like okay like the billion
9:28
parameter kind of like you know, line is the you know what will
9:31
dry? It felt kind of arbitrary.
9:34
I mean sounds like a nice little round number but you know do you
9:36
have any comment on that? Yeah, yeah.
9:39
So I think This is also super interesting.
9:42
I'm sure people have seen the Llama models coming from
9:45
Facebook, but now we're actually seeing that maybe you can do
9:50
more with smaller models but just train it on where data.
9:53
So for anyone who's curious, you can look up the chinchilla
9:56
paper. So there's this idea that models
9:57
have to be chinchilla optimal, which means that if you have
10:02
like a one billiard parameter model, you have to train on x
10:05
amount of data. But then if you increase the
10:08
number of parameters, you have to train on.
10:10
Maybe like 10 X or 1000x, more data for it to be chinchilla
10:15
optimal. So really, it's not just about
10:19
the parameter sighs. It's also about the amount of
10:22
data that it was true. Not because a lot of the first
10:26
generation of these are childish models were undertrained.
10:30
That's what researchers are realizing right now about
10:34
awesome. Great too.
10:36
So we have these, you know, access to these really large
10:39
models. You know, and I think there's a,
10:42
I think it's kind of funny because we went from, you know,
10:45
very, very specific like task models, tons of training data
10:49
and we're going to go do this, this, this and this and now to
10:53
the more generalized models to now like really or say, well
10:55
actually what we really want to do is kind of like the
10:57
generalized models but also combine them with smaller models
11:00
and I think a lot of the reasons by combining them smaller models
11:04
is really around. Okay, these things are actually
11:06
really hard to put into production.
11:08
They cost a fortune and And the kind of like economics are
11:13
slightly flipped, right? I think like anybody, who's so,
11:16
I've spent almost a decade looking at just production
11:19
problems. And as soon as we kind of look
11:21
that these like a lens, I was like, I don't like how we're
11:24
like, how we're going to make businesses out of it.
11:26
But cool thing is here is that how does it actually spent quite
11:29
a bit of time doing some of that work of like, okay, how do we
11:34
actually make decisions? We use apis, we use our own
11:38
models. Do we continually train our own
11:40
Also eyes. I'd like to give you the foreign
11:42
time to talk to us about like your use case, whatever, you're
11:45
willing to share but also kind of like what the thought process
11:48
was, right. And how you broke down getting
11:51
to you know, a specific in production, use case here.
11:56
Yeah. So there's like we use large
11:59
language models in a variety of cases.
12:01
Like, we have this case of, like, regenerating some text
12:03
for, for some users, we use logic language models to Define
12:08
very custom embeddings. And I think to me that that is
12:11
the beauty right now like back with like five years ago, we
12:14
said like leaders in the oil with a large language models
12:17
because there's so much work. Already pre like are
12:20
pre-trained, it's more important to have clean and good smaller
12:25
data sets. So we can imagine that custom
12:28
use cases. So it's not like that but
12:31
quantity of the data anymore where we can bring them into
12:33
production, I think Rebecca made a really good point.
12:35
You need a lot of like data to train it to get your initial
12:39
state but once you Start reusing them.
12:41
Then you can fight saying this with a lot of with smaller data
12:44
sets, so that there is a, the number one would be like,
12:48
there's a good motivation to use those language models, because
12:54
we can get to a production ready model with this smaller sample
12:58
sets. I remember, like back in the
12:59
day. You like 2015.
13:02
And so on, we were working on their and their similarity model
13:06
where we didn't have large language models and we Spend a
13:10
lot of time like, curating the deer just to get to initial
13:13
State. The other thing is, like, when
13:14
we talk about like large language models in terms of
13:16
generating text, we need to think about, like, how do we
13:20
integrate this into the part? So for example we need to
13:23
protect the users against losing Nations or against like weird,
13:28
generator text. And so what we do on the digit
13:30
side when we generate a text, it goes through and it's a very
13:35
stringent not-safe-for-work filter.
13:36
If it's like the slightest rating there, we store this
13:39
several wait. And then we have, we check
13:42
constantly against patterns of hallucinations and if it follows
13:44
them to one of those patterns, we also throw the sample and
13:48
then we make sure in our app, whatever we show is, like
13:53
reviewed by an accountant before.
13:54
They actually send it to the client and then that is another
13:58
layer of like there's a human in the loop who's like you could
14:00
Satan does make sense of this next, its but at the same time
14:04
those large language models allow to integrate amazing.
14:10
Vacations are, for example, we are helping business owners to
14:13
understand their their books in real time, but accounting is
14:16
highly subjective. So we use large language wants
14:20
to understand sir to understand the subjectivity with luck
14:23
language models. And then we use like, we trained
14:26
like similarity based in machine learning models or we use those
14:28
generated AI, some of the decisions you made previous two,
14:33
starting to work with these models.
14:35
So I think, like it really, really?
14:36
What I'd love to understand a little bit better.
14:38
Is like you have the option of Using potentially many Pi, you
14:43
had the option of grabbing some of these open source models that
14:47
exist today. Maybe actually, is it word like
14:50
for the panel actually, like talking about what exists in the
14:53
open source Community? Because I think a lot of people
14:54
are unaware that there's a huge movement that is, you know, it's
14:59
not just opening. I building these models or the
15:02
large lecture, we like big cloud providers.
15:04
Really, actually quite a big community of researchers were
15:07
built. Open-source barge language
15:09
models like it. Luther in a couple others.
15:12
So I'll actually let any of the panel has any of you know it's
15:15
going to kind of like take that one and I yell because it sounds
15:18
like you've all looked at it. So I don't know if I got any
15:22
volunteers here. Nobody pop.
15:24
Yeah, there you go. A lot of time kind of in the
15:28
open source communities. I mean, number one, it's
15:31
unbelievable. Some of the capabilities that
15:34
kind of communities of people are turning out.
15:38
I think I'm gonna actually message in the chat.
15:40
If you can, you Discord service that I think this she just
15:43
jumping in to summarize some of the best work if you want to go
15:47
and understand the open-source bushes.
15:51
And so yeah you got to Lucia. I you got copper II who are
15:54
championing. Some of the attempts to kind of
15:56
build open source alternatives to some of the largest models
16:01
lie on as well. Doing really interesting work.
16:04
I think like I'm not a developer so I'd love kind of alongside me
16:08
and is or anyone else to Kind of give a developer's viewpoint on
16:12
this for sure, when talking to companies who are building at
16:16
the application layer, right. I think they start at least in
16:22
our experience they start building on top of apis and then
16:25
they start to consider. Okay there's this sea of
16:28
open-source equivalent models that I could be using.
16:30
That might give me some Advantage based on what I really
16:33
care about for my users. Let me start investigating this,
16:36
right? And a great place to start is
16:38
literally jumping into Those three discount, Savage to
16:41
understand what's being done and obviously hugging faces well, is
16:45
kind of a lineup of One models, exist, or could be used great
16:50
too. So, one of the things to talk
16:51
about and then Allah will pass this on to Hanna's, right show.
16:54
These models are available for download which are great.
16:57
I mean, they're not as easy as accessing them, as I got, you
17:00
know, hitting an API in. A lot of cases, you're going to
17:02
have to do some inline pre-training or some fine-tuning
17:05
yourself to actually make them usable.
17:07
But they're portable, right? Right.
17:10
And they're not relying on, just being behind an API.
17:13
So we have a lot of use cases that are potentially not good
17:17
for just using somebody elses API and sending data.
17:19
These are definitely options. So you're not kind of boxed out
17:22
of using Ela Adams because of it.
17:25
So, with that Preamble honest, love to look like I, you know,
17:28
walk us through the thought process here of and you don't
17:31
have to give other the obviously any secrets but would love to
17:33
understand like considerations how you thought about the
17:35
problem like what were those qualifications and then you
17:38
know, kind of like we can walk into like You did it, you know,
17:42
there's no there's no secrets that I would say like we
17:44
actually in fact, for published a blog post about like how we
17:46
implemented it. So somebody wants to Repro this,
17:49
go ahead and Conquer and yeah, but so when we saw this business
17:54
problem, with generating questions, for example, always
17:57
at the people, go to the opening, I play ground to try it
18:00
out and you see, like, okay, is it actually feasible with a few
18:03
prompts and and then the, the first I was like, can we
18:07
actually Repro those things with an off the shelf?
18:10
Model, which we could host in-house.
18:13
Because the thinking there is like, if that is possible, then
18:16
we have the opportunity to quantize and prune the model to
18:19
reduce the latency. And we have this major benefit
18:22
of LED, not handing over the data to the, to an external API,
18:26
which we're not in control anymore.
18:28
And so, for us with security and financial data, this is really
18:32
Paramount to everything. We do, we promise our customers,
18:34
are we? We're trying to be like we're
18:37
keeping everything secure and in-house.
18:39
So even going to open Guys like it's that burden is tremendous
18:44
and would change, like even a whatever application of the
18:47
terms and conditions with our customers and therefore there's
18:51
a huge motivation to like cambree used open source models
18:54
and host them. All ourselves is another benefit
18:57
of leg. You have lower the network
18:58
excuse, if you have the model sitting right next to you and
19:01
your instance, there's like lots of cool benefits.
19:04
But to would get away with saying, like, do you have to
19:06
punch in a model? But in that moment, don't you
19:10
also have like lots of opportunities to like make those
19:13
models smaller and make the domain specific make them useful
19:17
to your specific use case and the model can then in the moment
19:21
and forget about all the other Villages, concerted the string.
19:25
So speaking of use cases and you know, you kind of mentioned one,
19:28
we're like very particular needed to be able to kind of
19:31
Bring It in-house like David. You've been looking at a bunch
19:35
of things you've been with. You had the developer ecosystem
19:38
Doodle for Road a pretty cool blog.
19:40
First about the developer ecosystem.
19:43
But like what use cases are you seeing kind of how to maybe kind
19:47
of talk a little bit about Adam, think about when to use look
19:51
let's focus on production right and production use cases.
19:54
And what you know when to go in One Direction, one to go in the
19:56
other direction. What you're seeing that's really
19:58
interesting right now. Yeah, I guess like I have a
20:00
framework that is to think about this which is like, okay.
20:03
So we would just talk about Foundation models and David
20:06
coming for a while. But like this capability kind of
20:08
got dropped in our lap magically by opening.
20:10
A, i a couple years ago and so like when you, when you think
20:14
about what's being used in production now, I think a lot of
20:17
how to think about it is just like, how much time do we had to
20:21
execute with this amazing new technology we have.
20:24
And so, I frame everything to like, how long does it actually
20:27
take to build it and so the obvious stuff that you saw in
20:29
production first that because like the most simple application
20:33
of language models and stuff like Jasper and Poppy AI that
20:36
are completing text, right? Or in the Like sales category,
20:41
like Reggie and auto bound books, that are kind of like
20:43
generating sales, copy and sales emails and things like that.
20:46
And so why it's easiest thing. Like you the production is like
20:50
basically just a call to open a i you had one API called your
20:53
application and your good and and you can do really incredible
20:56
things. And so a lot of the stuff that
20:58
requires no infrastructure, you see really successful in Pride
21:03
right away. And then you get like one layer
21:06
down like what's the next level of complexity as you try to do
21:09
things? I think like the most obvious
21:11
stuff is okay. We want to incorporate external
21:13
information about the world or about our users into these
21:17
models. Sort of like, what hottest was
21:20
just talking about in one sense, like you need to understand
21:23
what's going on in the world to be able to do anything and we
21:25
counting or documents or spreadsheets, whatever it may
21:27
be. And so there you have to start
21:31
like building infrastructure databases Integrations, pulling
21:34
data from whatever sources you need.
21:37
And they're just like the fundamental time.
21:38
It takes to do engineering. And so I think we're in the
21:41
moment right now where a lot of what we're seeing.
21:43
Come to production are tools that were, people are finally
21:47
finished that engineering workload, that it takes to like,
21:49
make all that happen and it even could be some not to fine-tuning
21:52
and things like that, but for the most part, I think it's like
21:54
just like Grab some stuff from a database connect to slack or
21:57
whatever, you know. So I didn't act category, like
22:00
you're some big shot, you beauty.
22:02
It requires a little bit of context of like a whole
22:04
conversation and wrangling that we're all of the bespoke
22:07
chatbots that got announced when they announced their new API
22:10
last week couple of folks. I've worked with like nem, which
22:13
is a better to notion read. Sure, historical, documents and
22:17
emails, and stuff like that. To help like complete do text
22:22
and pollution but more personalized to you.
22:25
Working with a company that's in our portfolio, called apt Edge.
22:28
Who do this awesome, like customer service work where they
22:31
have to connect to, like, Jared tickets in zendesk ticket since
22:34
lacked like know, what's going on?
22:36
So, it's like my tear to of complexity.
22:40
And for those Park people like in that chair, still are doing
22:45
the amazing work that has done to like, actually go grab a
22:47
model. Fine-tune it that's hard work
22:50
still and this stuff is like, kind of somewhat
22:52
straightforward, even if it's more complicated.
22:55
So I think of like, what's the next tier of complexity?
22:58
And I think there's like two big categories you can get to one is
23:04
like increasing model complexity.
23:05
So like being chains of calls, we have like want to do.
23:10
Agents and you want to call it to language model and get a
23:12
response and parse the response. I call it the language model
23:14
again. Yeah.
23:16
And that's just kind of gets on unwieldy still.
23:19
Like we haven't really done all. It's just actually the kind of
23:21
hard to be confident in the system's.
23:23
You build with that in a way where you're like ready to put
23:26
in front of users, I don't think a lot of people have gotten
23:28
there yet. So there's like these open
23:30
source Frameworks like laying chain and stuff out.
23:32
There were a lot of people are doing cool stuff that making
23:34
chains but trusting that it exists probably, why James exist
23:39
is because like Trust Some of that back and forth with the
23:42
model is still challenging. And I think this is another
23:45
engineering Gap where I think it's just going to take us time
23:47
as a community and as a set of developers to figure out like
23:53
how do you do it and how do you manage it?
23:54
And what you can trust and the other side stuff like
23:58
automation, where you're like like a lot of stuff to see now
24:00
is copilot. Honest talked about this where
24:02
you like you have a human in the loop to validate and that's
24:06
because like this evening we already it's like hard to build
24:08
all the tools we need to trust models right now.
24:10
But but where we want to go is up the mission where we have
24:16
lme's doing everything but that again requires just a ton of
24:20
engineering work and going down that route.
24:22
Like that's where people really think fine-tuning is necessary
24:26
you kind of have to or rlh like the more complicated methods of
24:30
owning and tuning models that you can trust.
24:34
It's yeah I talked about like there's a big difference when
24:36
you give the L and the keys to the car versus the autopilot.
24:41
There's a lot more things that can happen and so it's like long
24:44
answer but I think there's just a what we're seeing is more and
24:47
more engineering complexity getting layered onto this
24:50
amazing ml thing and use cases spawn up nearly daily because of
24:56
the amazing progress, we're going getting, you can sort of
25:00
track it on a straight line of how complex people are willing
25:03
to go. I think the next Frontier to
25:06
what house is talking about is, okay, can we go and weights, can
25:09
we? Distill models.
25:11
Can we do lower latency work? Because there's already that
25:15
that's part of engineering complexity.
25:17
Some people are bold and on the frontier doing it, but I think a
25:21
lot of people are still sort of in level 1.
25:24
It David, you bring up a really good point that every little
25:28
open source language model comes with it, its own tokenizer as an
25:31
example, right? Like you have your bird that has
25:33
like workpiece tokenization. You have your Excel item that is
25:36
like sitting in even like putting those pieces together.
25:40
That isn't it real whole Challenge and I told you glanced
25:43
over this, like I'm sorry about all this.
25:45
But like once you once you get over this hump and you actually
25:49
have the models and weights in front of you, then you can run
25:52
like for example the tents up over time, And like, see what
25:55
can I cut out as language model and make the ladies in Fast and
25:59
this is something which maybe you don't have to do for the
26:02
probable. But if like, if you see a lot of
26:04
inferences that investment might pay off and then you're sort of
26:08
this coupled. See, from the API, you get a
26:10
faster response and you save money long term, but it to your
26:15
point it was like this lots of engineering complexity in there.
26:17
So it's like, I can see the trade of products Services might
26:19
be easier to code or an API because you get it up and
26:22
running faster and versus long-term benefits.
26:25
One of the interesting things is your Diego mention their flight
26:27
that like a lot of what's happened is this like
26:30
democratization where so many people can develop with language
26:33
models. And so it's like, oh I guess go
26:34
write a prompt and I've done something cool and man, it's
26:38
really intimidating. Just like it always has been to
26:41
go from that to like I'm going to download weights and learn
26:43
tensorflow and tuna model and use the touch of low profile.
26:47
Like I was in ml engineer for a whole bunch of years and I never
26:49
use the profiler because like getting to that level is kind of
26:51
scary still, you know? And I was like I think a decent
26:54
one baby. It is still like, there's like
26:56
that's of ml problems that he gets really complicated, really
26:59
fast. And so, I think there's a lot of
27:02
people Sort of on the sidelines once you get to that level of
27:05
complexity because it's suddenly like, yeah.
27:08
But we've told everybody you don't need to learn them L to do
27:11
ML and then it's like they're my dearest.
27:14
Go fine-tuned 13 billion parameter model, good luck and
27:17
then think there's still a ton of work.
27:19
We have to do to make that as accessible as a prompt.
27:22
Maybe I'm Emmanuel, Mason machine learning engineer, a
27:28
stripe and author of building, machine learning, powered
27:30
applications. And if you don't, A machine
27:33
learning models to explode, you should subscribe to this
27:37
podcast. I was just going to double tap
27:52
on that, like a bunch of the. We've talked to many thousands
27:58
of generative, AI companies, write and Most of them are not
28:05
the people building. These companies are not ml
28:07
developers. They're not ml Engineers right
28:10
there, they're not ml people. They are software Engineers that
28:13
are trying to build really interesting software by kind of
28:17
plugging in these incredible pieces of Technology, right?
28:21
And they're playing in the tier 2 complexity over that you just
28:25
explained right where by, okay, we're going to be on till one,
28:29
which is kind of you my subscribers that kind of UI
28:31
rapper style, like Demo like this is cool, I can build on
28:34
this, but you actually building kind of semi agentic, chained
28:39
systems right there. Get highly complex very quickly,
28:44
but even jump into that third tier where it's kind of do
28:47
yourself ml development. A thing is just a huge barrier,
28:51
so I think we're going to see very large companies built in
28:56
that tier 2 of complexity, right?
28:58
So we saw like, Jasper. Enormous company, right?
29:03
Built on that Tier 1 I think we're going to see a second wave
29:07
of a huge number of companies that we're observing right now
29:10
building that th you can pecs T. So we're rebuilding the tooling
29:15
that enables people to actually manage that complexity such that
29:18
they can build really interesting agentic systems and
29:21
then potentially automated ones. And then I think of, who knows
29:25
when we get to like tier 3, right?
29:27
And like Anna's is done, very non-trivial.
29:30
Work as a In ml, the relevant ml engineer, to get to that stage,
29:35
I think in that to tear complexity, we're seeing a lot
29:38
more highly capitalized, companies that are just building
29:42
that Tia to our software application players.
29:45
So that's why we're building. What we're building is to try to
29:47
enable them but I think we're going to see a huge population
29:49
of this time. So I'm going to try to
29:52
completely oversimplify kind of like what three categories that
29:58
we can talk about for ELA abs and production.
30:00
When we talk about Entrust will talk about testing the results
30:05
security privacy. Anything that's regarding kind
30:08
of like, you know, the you know, the ability to use these.
30:11
Let's talk about latency and I'm thinking latency a category,
30:15
because I think it's completely under appreciated, how much
30:19
latency matters building any sort of software, maybe to your
30:21
point James about how all these software Engineers are actually
30:24
working on these problems, it might be more present.
30:27
But, you know, we'll talk about, you know, kind of lazy.
30:29
And they will talk about cost. Right?
30:31
And so I think those three things and I think in terms of
30:33
in all of these three things, I think we can break it down into
30:37
the engineering tools that are going to exist how we're
30:38
actually going to apply that what people are doing.
30:40
What research is coming in because I think new
30:42
architectures are coming out and so what we can start on any one
30:46
of them, right? So I'm going to Define, I'm
30:48
going to Define cost as literally what it cost to run.
30:51
And this isn't just about, you know, the API call, I do know,
30:56
we're talking about what it takes to run to the hardware
30:58
what it takes to you. Lift up that Hardware what?
31:01
Takes to kind of like run not at scale, as well as potentially
31:05
use the apis. Obviously I mention a little bit
31:08
about that trust, right? So trust will talk about privacy
31:10
security like the use cases there and then forgetting the
31:14
other one was that latency latency, which is really the
31:17
definition of the use cases, right?
31:18
So maybe we'll start with like latency to begin with and you
31:25
know, James I know this is near and dear to your heart.
31:27
So yeah. First of all, Define to us like
31:31
What latency is in these kind of like LM world use cases.
31:36
How do we look how we should look at it?
31:38
And more importantly maybe kind of like punching some of the
31:41
problems. I think it's an unsolved Problem
31:42
by the way so like I don't expect you to answer it for us
31:45
but kind of like we can if you want to kind of like frame it
31:48
for us. Yeah absolutely.
31:50
So I think you can observe and think about latency at two
31:54
levels. Right?
31:55
One is that the kind of instance level like specific model and
31:59
then one is it the the workflow level Right?
32:01
From a more kind of user perspective.
32:03
This is like you're trying to deliver ultimately some kind of
32:06
user experience that adds value to someone such a, you can build
32:11
an application potentially build a company Etc.
32:14
So when you're cooling, let's say you're at that tier 2 of
32:17
complexity that David described, right?
32:19
You've got a chained kind of workflow of potentially multiple
32:23
instances of models. Going to give you an idea here.
32:25
I was talking to the founder yesterday and he got lost
32:29
walking me through his own kind of Of workflow here and he's
32:33
cooling seven different instances of models, right?
32:37
Four of them. Fine-tuned Babbage.
32:40
There are a couple of embeddings models in there, right?
32:43
And each of those calls based on the AP, r-- any is bringing in
32:47
some sort of latency Chained together at the workflow, kind
32:53
of level, you have the ultimate agency that the user
32:56
experiences. I think that's going to be king
32:59
great, especially in the second level of complexity.
33:01
It's like, I need to Bill especially for those who are
33:04
building here thinking software, right?
33:05
And to your point the OE. So like maybe they're thinking
33:08
more about this than others, they absolutely are.
33:11
So we have talked about when in our conversations with companies
33:15
is a cost quality and latency triangle right?
33:19
And Ending on your use case, you care about one of those more
33:22
than the other two and like the best.
33:25
The best example of this is copilot, right?
33:28
Like get our copilot latency is everything you have to stay
33:33
within a flow state for your user experience such that you
33:37
know we could they can make those functionalities better but
33:42
they don't want to because we want to maintain very, very low
33:44
latency. That's how we think we need to
33:47
start thinking about this is again there's a latency of Can
33:49
one call to one model but then there's a latency of an
33:52
end-to-end use route. Yeah.
33:54
I love a pic. That example because I like I've
33:58
many times use that example as the kind of like state of the
34:03
art in terms of latency and it's pretty darn slow, right?
34:07
Like in terms of like when I look at right now it's a
34:10
suggestion but like I want autocomplete and jumping from
34:14
suggestion to back to the computer using doing
34:16
autocomplete will require fairly Significant engineering to be
34:21
able to get that and actually like my gas.
34:23
And I'm not the expert here, is that it will take kind of like
34:26
re-architecting some of these models or you know, far from the
34:30
ground up. So I don't know.
34:31
How do we have a question here that people are asking about
34:33
her? I'm going to quickly Define
34:34
production and then I'm going to jump to you Rebecca because I
34:36
saw you got excited about re architectures.
34:38
So what we mean about production is actually like an actual
34:41
application that's being used for users.
34:43
As you know and I would argue kind of like eating a kind of
34:45
like you do in a production matter.
34:47
So a service that is being like Offered to other people that is
34:51
running all the time. This is beyond the demo.
34:53
This is beyond the, you know, kind of like coil weekend, you
34:58
know, kind of thing. This is like you're actually
34:59
using it as part of a business process, educational process,
35:02
whatever you're using. Production is I usually like to
35:05
Define it like people scream if it goes down.
35:09
And so I think somebody was asking about like, does that
35:12
count that open API service in Azure?
35:14
Absolutely. If you're using it as part of a
35:16
production application, like if you're just using it for your
35:19
Cory a pat, you know, in on your computer then I wouldn't
35:22
consider that production. So rewinding a little bit we'll
35:25
go back to, you know, what is it going to take?
35:27
We're still in the Pas subject of latency and you know,
35:31
state-of-the-art right now is a couple milliseconds.
35:34
I think actually like there was a paper that talked about
35:37
state-of-the-art in kind of like what I call in a lab environment
35:39
was a 29 Ms. In Frank's past which is like you know,
35:45
discontinues a bunch of use cases that I've been
35:47
traditionally worked on in a male which Our kind of like sub,
35:50
15, Ms, use cases, but back to you Rebecca.
35:53
What are we seeing that's going to be able to, like, hit this
35:55
latency problem? Yeah, no, that's a great
35:57
question, a lot of thoughts and less.
35:59
Yeah, I think Hannah's the uptime maze optimizations, you
36:02
mention with quantization and model distillation awesome.
36:06
Yeah, there's a lot of great engineering work happening now.
36:10
So I actually think that engineering especially in for
36:13
engineering and foreknowledge is really relevant.
36:16
So, yes, it's people need to learn, you know, pie.
36:19
Church tensorflow, learn how to find you models, but I think
36:24
actually more and more just required engineering skills are
36:28
increasingly relevant MLP and was to the production, Isaac
36:32
zation of a, I going forward. So see, I just really want to
36:35
drive that point home as well. You need good engineer's in the
36:39
space on D and for inside. So some of the inference being
36:43
slow is indeed Diego. I think you hit the nail on the
36:46
head here. It's the Transformers
36:48
architecture. So Transformer and this is great
36:51
for a long-range dependencies and capturing context at is
36:54
actually faster in training because you have access to all
36:56
of them fight but it's not good for fast and friends.
37:00
And you know, this is something that is very inherent to the
37:04
architecture and that's why we're talking about latency and
37:07
like, imprints need so much because like, we are bottleneck
37:10
just by the Transformers structure.
37:14
And so, if you compare Transformers 2 models, for like
37:17
recurring neural networks like iron, Nuns and Transformers are
37:20
not naturally sequential. So are an empty coding is going
37:24
to be a lot faster than Transformers.
37:26
Which have your multi-headed attention Matrix computations?
37:30
So you just have more competitions and added to that.
37:32
We also you know we're talking about Allah.
37:34
I'm so there's also just more parameters and these models so
37:37
in every board costs you have a lot more layers, tensors the
37:42
upper so it's just yeah I think just everything just compounds
37:46
to being slow, all right. So yeah, I think some of this
37:51
will require you if we really want to talk like just like
37:57
we're moving this bottleneck entirely as opposed to like
38:00
working around Transformers, which is what we've been doing.
38:03
It will probably require an architecture change.
38:07
That's going to be hard because we've seen Transformers have
38:09
been so dominant since 2017 and it looks like it's becoming more
38:13
and more dominant and not going the other way around.
38:16
If people are interested and this is you know, still very
38:18
much Such an academic research territory.
38:21
But I think I've been excited about as its makeup state-space
38:24
models, which is sequential. And so it that they
38:29
underperformed a to like Transformers model so far.
38:34
But there, I think you know that takes off.
38:38
Watch out for that coming out of Stanford, then we can solve the
38:43
local paper. I'll put the paper in the chat.
38:45
The Hungry Hungry Hippos, chat. I just want to jump in there
38:48
because the are you sending? Pausing.
38:49
Right, it basically invalidates right now whole host of use
38:53
cases like that late for that fundamental latency barrier.
38:57
Like there are so many things that you can do in principle
39:00
that currently you can't do in practice because of that
39:03
fundamental latency. Very even if especially when
39:06
you're at that tear do of complexity right at once.
39:09
So the founder was talking about that.
39:10
I was there's doing this company initial pass on that.
39:14
Workflow was 15 seconds right now.
39:16
That is a fundamental identity. When we use a See, they're just
39:19
doesn't work. It's a non-starter like
39:21
everyone's like giggling a little bit right now.
39:23
He knows to get that down to like three and a half seconds
39:28
through kind of fine-tuning. Babbage instance ears, rather
39:31
than using large models Etc. Taking this tier 2 complexity
39:35
perspective and it might be good enough.
39:38
It like it there might be good enough to build a really
39:40
interesting company on top of in that tier 2 complexity.
39:45
But that won't cut it for other use cases, where there in a half
39:47
seconds, just kind of lost you at the room rate, the others
39:51
funny like orders of magnitude of like, for each person with
39:53
their seven person or seven calls to an API, there's someone
39:56
with 70 in lighter thinking of of 700 to do complicated things.
40:00
And so like there, it's hard to describe how many orders of
40:03
magnitude room there is for improvement.
40:05
It's a lot. And I think like we are talking
40:09
about atoms in production, right?
40:11
Whichever more broadly about foundational models.
40:15
He considered a kind of multimodal well that's coming
40:18
down the pipeline, right? Which is just going to be
40:20
fascinating. Maybe that's an unhelpful
40:22
grenades were thrown into the room with kind of 15 minutes to
40:25
go Omega Zell for grenade. He knows we'll find out, but I
40:29
think one thing I wanted to get people's opinion on was like
40:33
taking them moves lower approach of these large models, right?
40:37
If we assume that some of these very large models, are going to
40:40
go just get cheaper, and cheaper, and cheaper and get
40:43
faster and faster. And faster, like what changes
40:47
fundamentally over the next 12 months if okay we don't know why
40:52
but like maybe it's a new fundamental architecture but in
40:56
12 months opening II has a model that is just 10x faster and is
41:01
10x cheaper again. Like how does the world change?
41:03
I'd love to hear kind of instincts on that.
41:05
The it when we part is we have like a Divergence right where
41:08
you're actually don't get you know you don't get actually the
41:14
you know what? Like larger model Mega model
41:16
that it just gets smaller and cheaper.
41:18
But we actually get to back to its highly specialized models.
41:22
Or, you know, I think one of the I'd actually seen implemented
41:28
recently and it's really interesting.
41:29
These kind of like start with one of these very large language
41:33
models and actually use them as a router like a task router into
41:37
a much more specialized tasks. And I see how this is like, not
41:41
even I'm guessing he's going that's what he's doing
41:42
internally, which is Kind of starting from a more generalized
41:46
task, right? Because you take advantage of
41:48
the You take advantage of the kind of like, no need for the
41:52
pre-training. Kind of like, you know, kind of
41:54
like no, cold start problem. You can like go into a use case
41:58
and then route from there into something a lot more.
42:00
Specialized that cumulates latency though, right?
42:04
And so, I think like going back to kind of like I because I
42:06
wanted to close down the fought on the latency thing.
42:08
Like I think you know, the most important part that I see here
42:11
is, like, you really need to go down to the use case and say,
42:14
okay, what is this? Use case requirement right for
42:17
latency. Because, you know, if I'm
42:19
looking at Near real-time use cases like the reality is that,
42:23
at least today, right? I'm kind of out of luck, right?
42:28
Like and so, you're not really going to be able to achieve
42:30
those. You're going to have to look at
42:33
different architectures smaller models, distill it down to like,
42:37
very specialized to get anything even close to that.
42:40
And if you start thinking about again, like, you know, you look
42:42
at things like you've mentioned, like you're chaining, these
42:44
things you're just accumulating latency, right?
42:47
And you know across across those chains, So you get into any get
42:50
in here. I have a big batch process.
42:52
It doesn't really matter. But I think that's a, you know,
42:55
that's that's really interesting because I think we, you know, I
42:58
don't want to be like to deterministic here we live in a
43:00
probabilistic world but like it sink right now truly real-time
43:04
use cases are kind of out of the question in like point today and
43:09
time at least from I know kind of like production perspective
43:12
at least from what we've seen. Now that said I'm sure that the
43:15
Facebook's and Google's and you know, kind of like Microsoft's
43:18
are very much working on this problem but I think that's a
43:22
reality, at least in the, in its current state Anybody disagree
43:26
with that? Or is that kind of a?
43:28
I am? Can we get a mop?
43:30
I don't disagree, but I just think it's bold to be recorded
43:33
making claims about where we are said point in time.
43:36
I said point in time, you know, about two months and will be
43:40
like, how'd you remember? Where latency was a problem?
43:42
Yeah, I'm that but but I think latency is a function of like
43:48
the business case the the way we deploy like with, for most of
43:52
the applications were talking about, we don't need Need a
43:55
general model like GT right leg. We can go through like a debates
43:58
to silly application and what worked really well for us, just
44:00
like sort of like the teacher-student approach.
44:03
We're like we take a larger model and then break this down
44:06
into something smaller, cut out layers and and then.
44:10
So if I compare like the latency from our first birth deployment,
44:13
a couple years ago to like what we have now, it's like magnitude
44:16
sticker. It is late and we were able to
44:19
get like reduced models, we would get the profilers, we get
44:22
X, Ln, X La, we get almonds, Optimizations.
44:26
And so we get now, we can even run those models on the CPU in a
44:31
very condensed version and then the costs player play a role as
44:35
well. I'd like to like you can throw
44:36
money at it and somebody said in the chat like you can just use
44:39
the GD, right? So if your business case pays
44:42
for this, that's the option, but I think the there to make it
44:46
very domain-specific, is the biggest latency saver.
44:50
You can you can go after And there's there's there's come in
44:53
on the chat, I think like, like let's talk about, I think we are
44:57
rephrasing, we're talking about latency across the entire
45:00
workflow, right? Which also includes like
45:02
everything else, right? Like the system You're Building
45:04
likely has over. Had the problem is that you're
45:06
actually significantly adding a bunch of latency while usually
45:10
in like any of these kind of like online systems your offline
45:13
system, you know, the lane, see your systems.
45:16
Adding everything from IO to database, poles to everything,
45:20
like your now, adding African amount more as you kind of like
45:24
chain through these to these models.
45:27
So I love it. You gave me the perfect segue
45:29
Hannah's. Here, they talk about and you
45:31
may be able to kind of let put it on CPUs and anybody who's
45:35
worked in this space knows that. That is the Holy Grail for your
45:40
wallet. Given that running gpus is about
45:44
Forex what you can do on CPUs. And so, I'll go back to our
45:48
wedding. I'm definitely stealing this
45:50
James. I love the triangle of the
45:51
quality. Constantly and see.
45:54
And so, you know, determining the the use case is right
45:58
because when you look at in friends and influence on gpus
46:02
and you look at that they are on average and I think it can be
46:05
more on average. It's about 4X more expensive.
46:07
That's what really you need to be like talking.
46:09
So let's talk about cost and the use cases, you know around that
46:13
and how to think about I'm going to pick on you David because
46:16
you're probably looking at a bunch of companies and being
46:18
like, yeah, this sounds great, but if they scale like they
46:21
can't afford It that's actually like you know run this company
46:25
you know, at any like cost like that makes sense even though you
46:29
know, GPT 3.5 is now 10x cheaper or actually I'll just let you
46:33
answer truthfully. I got I think this is an
46:37
interesting question because it's really important and I've
46:40
seen what happens at scale and it's almost like were thinning
46:44
to Rebecca because like meta scale cost becomes pay for whole
46:48
departments by making one percent improvements on like
46:51
their ad. In for a ranking engine, right,
46:53
I guess, it's insane. What happens at scale with
46:56
inference for startups, at again, what are you doing
46:59
started? I just like, I don't know, like
47:02
money on fire, like beat people when, you know, and I just
47:05
haven't seen to be a constraint early on it, certainly of do the
47:08
math. So like it's really easy to back
47:10
in the envelope. How much is one inference cost?
47:13
How many infants is, what you do?
47:15
What's your margin per inference?
47:17
And you can configure all that about value.
47:19
You can get there on the apis and particular price.
47:22
Things all pretty easy to predict.
47:26
That's why I always do that math.
47:27
It's good to do that math. It's good to know where your
47:29
margins at. But you know, I think part of
47:32
how I think about it is you should thanks to folks like
47:36
honest who have gone down the road before and show notes that
47:39
you can build domain specific models.
47:41
And there is room to optimize. Like, it's a question of
47:44
figuring out, when, to optimize for cost, is for most of the
47:47
people that I work with, and for most people, it's not, they 14
47:52
meta, it's certainly true that you have to do it and it's
47:55
worthless. During 100 Engineers to work on
47:57
it baby or a lot, because it's so important.
48:01
But it's just this thing that you have all with overtime
48:03
monitor, I always recommend people be aware of but I haven't
48:07
seen it blocked. A lot of these cases early on,
48:09
that's what I know. A lot of people that were really
48:10
happy when the new chat API came out was 10 times cheaper because
48:15
yeah, this is like, for startups at least you're talking about
48:18
Runway, like, how much money do you have in the bank to survive
48:22
is changing and that's critical, but You can do it.
48:27
Like I think you can hire an ml person.
48:30
You can learn ml. You can find two models.
48:32
You can like I think there are tools out there to avert cost,
48:36
which is something you have to be thoughtful of coordinating.
48:39
Is this an exercise Han? Is that you date up front?
48:41
Like, did you think about even back-of-the-envelope?
48:44
Like, okay, we're going to go to do x, y, z.
48:46
This is what it's going to cost, like, you talk to people in your
48:49
team that were beyond the MLT. Because I'm assuming this is
48:51
like, at least involve product management or leadership of the
48:54
company. In terms of like, yeah, we're
48:55
going This route. Absolutely.
48:57
We've been in the back of the envelope calculations.
49:00
It's a little bit tricky with opening.
49:01
I'd like how many tokens doesn't actually require.
49:03
It's all this town's perfect, we get some back of level of
49:06
estimations went to the CEOs. Like would you be?
49:08
Okay is the stator goes to another Epi.
49:11
This will obviously a proxy questions there and then there's
49:14
obviously a Cost question in for the with the tasks we had
49:18
invented. It was more beneficial to do
49:21
this in-house. We just to give in like a little
49:25
bit inside. On the scale though, we have to
49:27
processed and close to 100 million transactions a day.
49:33
That is not like something we could easily ship over to the
49:37
needy. I and then we have to talk about
49:39
like latency since our up and in that moment even the project of
49:43
like investigating the model and figuring out, what could be cuts
49:47
to sort. So we could actually run it on a
49:48
CPU Festival was super beneficial so then we can we can
49:54
talk about it. Like what happens if we brought
49:56
we brought us this batch processor is streaming that
49:58
Cerberus but yeah it costs is basically the first course in
50:01
the media it's a little dark room.
50:04
So we're going to go into the last subject of the morning
50:09
here, I guess morning for us at evening for James, which is kind
50:13
of trust and Trust in the systems.
50:14
And so there's one term that I think it's worth defining, which
50:19
is like, some people might have heard of it, that these models
50:22
hallucinate and I'm pretty sure that's not because they're just
50:24
trade on a bunch of people's Burning Man, pictures, maybe
50:27
Rebecca you can actually kind of like help Define what a
50:29
Hallucination is in this in the In this context and why, I know,
50:36
kind of discuss why this needs to happen.
50:39
I would say, who is the nation's?
50:42
Yeah, just anything, that's not a factually accurate and just
50:45
falls into the accuracy problem when you're thinking about
50:49
evaluating language model outputs.
50:51
You know sometimes it's open-ended like how engaging or
50:54
interesting or these outputs sometimes it's like as a school
50:57
and our grammatical mistakes, but what a lot of people are
51:00
caring about when we're proud sizing, these As I don't want
51:04
the model to say that, you know, Michelle Obama is the president.
51:08
For example, that it's like a very steep of my brand and bears
51:12
for how our company and this is a problem that has actually
51:16
existed and language models for four years, but I think it's
51:21
become a bigger problem because now people are actually using
51:23
these islands for real-world test.
51:26
So now, you know, it's not just like researchers writing papers
51:28
about who this, the nation's kind of prove it.
51:31
It's like actual Users of these all I'm saying my model just you
51:38
know read this this text where this Downstream use case, like
51:41
maybe like generating legal text, for example and it's just
51:44
totally wrong and really embarrassing for a company.
51:48
So yeah, I think that this is, you know, we talked a little bit
51:51
about research directions. I think David may be or how does
51:57
the mentioned, you know, having basically external knowledge
52:01
sources looking in like retrieval, right?
52:04
Out of the gates, a true. But like I think we're
52:06
describing is like retrieval augmented duration.
52:10
So yeah, a large language models currently as it stands, I'm a
52:14
very poor sources of that. Some knowledge and this is
52:21
because it's just not very proud or efficient to be learning
52:23
facts. And so, I think the way forward
52:25
really is having these knowledge bases and combining it maybe
52:30
with like a retrieval and don't actually, I seem to chat bank
52:34
and actually hit it really well with how I like to describe it.
52:36
I mean, like, we're used to very deterministic, workflows and
52:40
code, and we've been building deterministic systems for a
52:43
really long time, but now we're applying kind of like a
52:47
probabilistic. You know, kind of like
52:49
implementation to some of those. So maybe some of the tips and
52:54
tricks and I got made my tricks. Really be like how how have
52:57
people been thinking about making these things feel more
53:00
deterministic and there's been a lot of techniques that have been
53:03
used by different teams, you know, to do it.
53:06
But I'd love anybody. Jump in here.
53:08
In terms of how to think about some introduction of some of the
53:11
techniques you can use to make these workflows feel more
53:13
deterministic. I think you talked about kind of
53:15
like the teacher-student kind of model that There's a like kind
53:19
of like, you know, human reinforcement like, you know,
53:21
kind of model. There's a bunch of these
53:23
different models that can be used not to confuse with the
53:26
actual models. You know, I'm terms of how to do
53:29
is open up the room to anybody who wants to kind of like clutch
53:31
on that. Because I think this is so
53:32
important for us to build trust in these systems, right?
53:36
And and truly adopt them in in, you know, most people don't
53:39
understand what a probabilistic workflow is by nature.
53:43
Humans are bad at it. I think you go to study the use
53:46
case. Again right, there are some use
53:47
cases Right now that have like high affordability of something
53:53
not quite being right, right. Such that I can I can generate
53:57
an output and as affordance for my my users such that.
54:03
Okay. If I if we get Society wrong or
54:05
very wrong or whatever and that's fine, like I wasn't shown
54:09
the user and the user weren't take a bad action on the back of
54:12
it. There are obviously low
54:14
affordance use cases that we just can't surf today, right?
54:17
Like I don't want to To a medical diagnosis from an llm,
54:21
right because that's very low of wouldn't be right if he gets it
54:25
wrong like it's game over, right?
54:28
So and I think I wonder how long it will take until we start to
54:35
slide down that affordance scale.
54:37
That's kind of as an aside to your question that I think is
54:40
good context. Allah is have those really cool
54:42
a professor like they're basically databases with the
54:44
constant looker and so because we have to compute the weights.
54:49
There's just a fixed number of Weights but as a downside is
54:52
like we lose out the accuracy of their of the results being
54:56
returned trained in. So, whatever would I found,
54:59
really interesting was like, okay, this charity is like, this
55:03
is some submission risk there. And then when I saw the
55:06
infamous, Listen what what genius doing is basically, they
55:10
used like generate me a query that Kriegers the database that
55:14
gives you the top five results in, then it goes back into some
55:17
reason. It summarization application.
55:20
So in that inner, but in the benefit here is like it leaves
55:24
out on the concert look up by the benefit is basically get
55:28
them. All the references, whenever
55:30
they'd State something and maybe in the meantime, we need to
55:33
think about on a system level in such a way, they're like, okay,
55:36
how can we trade? References, or things like this.
55:39
And for the moment, give up the comments and look up concept and
55:43
then work towards that there is maybe it's part of the model as
55:47
well. Fantastic, we are at the hour
55:49
here so I want to be respectful of everybody's time.
55:52
First of all, want to thank all of you.
55:55
David Hunt has changed record. This was awesome.
55:57
This was really, really fun conversation.
56:01
I think like told the Matrix we should make a conference out of
56:04
this. This is a this is sounds like
56:06
there's so much each to talk about such an exciting space.
56:09
I totally agree with what James said, you know, probably eat
56:12
crow. And anything I said today, in
56:14
two months from now just based on you know like the speed at
56:16
which is moving everything and that's the fun part.
56:20
So I always reserve the right to be wrong constantly.
56:23
So with that, I will pass this on to Demetrius.
56:27
Thanks for having us. Yes, dude, this was so good.
56:30
We're going to do a conference. I'm already planning it.
56:34
The last thing, I'll say, if you haven't put your information,
56:37
Nation into this survey. I'm going to release it to
56:39
everyone, so I'm dropping it in the chat right now.
56:42
Again, throw in, if you're using these large language models in
56:46
production or if you're not, we want to hear why?
56:49
And then I will reach out to you with the answers and all the
56:55
data on this. I think there's already been
56:57
some incredible answers that have come through.
57:01
I'd love it. And I also will try and make
57:04
some kind of like tldr port on it so that everybody can see
57:10
what we're all doing and where we're all out with this.
57:12
I thought this conversation was perfect.
57:14
So thank you, everyone. And I may be given out of some
57:17
of these bad things happen to good data shirts.
57:20
For some people that fill out this this survey so bad things
57:26
happen to good data, we know that and if you drop in your
57:30
email, in the survey, you could win one of these shirts.
57:35
All right, everybody. That's it for today.
57:37
See y'all right job? Thanks everyone.
57:40
Thank you.