Latest / Agentic Conversations (formally mlops.community) / DSPy: Transforming Language Model Calls into Smart Pipelines // Omar Khattab // #194

Transcript

DSPy: Transforming Language Model Calls into Smart Pipelines // Omar Khattab // #194 · Agentic Conversations (formally mlops.community)

0:00
I'm Omar. I'm a fifth year pH.
0:03
D candidate at Stanford. I'm also an ALE AIML scholar.
0:09
I do retrieval and what we call foundation model programmes.
0:14
So these are pipelines of language models.
0:17
And I actually don't drink coffee.
0:20
This is the question that Dimitri has asked.
0:21
I love tea, right? Welcome back to their beloved
0:27
community podcast. I am your host Dimitrios and who
0:31
got a doozy today. Oh my God I love talking to
0:34
Omar. He is doing some incredible
0:37
stuff and I just got the full scoop from him.
0:40
We we stopped recording and then he told me the whole story about
0:44
the timeline of his creation DSP and then later on DS π and so
0:49
I'm going to rehash that for you real fast.
0:52
I think it is quite interesting because he is a researcher, but
0:55
he is doing DSI, he is creating DY and really looking at
1:00
retrieval and that is 100% Milos.
1:04
That in a way is exactly what we're working on day in, day
1:08
out. And so it's great to hear his
1:11
stories and how he looks at things and why he felt like
1:15
something like DSP was needed. It is an abstraction layer that
1:19
is a bit different than other ones that I think you see out on
1:22
the market that are doing similar things.
1:26
And so let's talk about this timeline because he broke it
1:30
down for me and he said, listen, we started version zero for DSP,
1:34
which is the precursor to DSI in February 2022, So not February
1:40
2023 when there was all that GPT ChatGPT buzz going on, it was
1:44
February 2022. And in February 2022, what he
1:47
wanted to create was something that was an abstraction layer
1:51
for pipelines using large language models, namely GPT 3,
1:57
the precursor, again through ChatGPT and all the crates.
2:00
O they put it out there and then they realised, hey, wait a
2:04
minute, nobody is really creating pipelines with these
2:09
large language models. Or if they are, it isn't really
2:12
a pipeline, It's just a straight GPT 3 call and they're
2:16
interacting directly with the model, not looking at it as a
2:21
pipeline in the way that we look at different pipelines these
2:25
days. And in traditional mill, you
2:27
would look at pipelines and using different models in
2:30
different pieces of your pipeline.
2:32
And so they continued down the road.
2:36
Nevertheless, they didn't let that discourage them and they
2:39
released DSP version one. And then he realised, you know
2:45
what? I think this is the wrong
2:47
abstraction layer. And so they released the version
2:50
one. But they had to take a long,
2:53
cold, hard look in the mirror and see if this is really what
2:56
they were trying to create and if this was as much of A value
3:00
add as they wanted it to be. And ultimately they said no.
3:05
Since they said no, they went back to the drawing board and
3:09
the reason that they were having trouble was because the D and
3:13
DSP this demonstrate it was really complex for people to
3:18
understand. And later in this conversation
3:20
you'll hear him talk about how this demonstrate what he now
3:25
considers to be compilers. Are there ways that you can
3:29
compile these different modules and really like go and do almost
3:35
this hyperparameter tuning when you're he?
3:37
He makes a lot of analogies with neural Nets, and he was talking
3:42
about how compilers are a way for you to go and you can
3:46
explore the best way of using these pipelines.
3:53
But when DSP came out, they were not using the compilers because
4:01
they were putting it on the user, the end user to create
4:04
this demonstrate and these compilers and they were saying
4:07
you need to demonstrate what you're trying to get from us,
4:09
right. And so people didn't know what
4:12
the hell was going on. They were confused out of their
4:14
mind and DSP wasn't the right abstraction layer.
4:17
So they went back to the drawing board after version one and they
4:20
said you know what, instead of this demonstrate what we're
4:23
going to do is we are going to be the ones that create the
4:26
compilers even though it is open source and if anyone wants to
4:29
create their own compiler they can.
4:31
But it he feels that it is his job to create the compilers and
4:36
then the rest, all these modules for the pipelines and all the
4:40
abstractions the end user can use and leverage.
4:45
And so you do not have to think through that complex piece when
4:50
it comes to the compiler and they put out DS PIE from that
4:55
and DSP I came out in August 2023.
4:58
It feels like it's been out for ages and it is absolutely
5:02
killing it. For anyone that has not used it,
5:04
we're leaving a link in the description.
5:06
I have fallen in love with this framework.
5:09
I absolutely think it is fascinating and especially
5:12
because some of the key things that he pointed out when I was
5:15
talking to him and I literally was holding on to my chair, this
5:18
whole conversation trying to keep up with the guy.
5:20
Omar is brilliant and he was able to bring me on this journey
5:25
and I hope he brings you on this journey and doesn't lose you.
5:28
I had to slow him down at some points just to make sure that I
5:31
was understanding what he was saying.
5:32
But really he talked about being how brittle it is.
5:38
If you're prompt 1st and prompt first is like, hey, there's this
5:43
dark art of prompting. And really if we're looking at
5:46
it as like we're going to somehow find a magic prompt when
5:51
we're interacting with these LM's and when we are creating
5:54
pipelines to go and make LM calls, that's not the way to be
5:58
looking at it. And that's what they're actively
6:00
trying to avoid with DS π. So the final thing I want to say
6:05
is that at the end he talks about fine running and how
6:08
modules with fine tuning or a module in DPI can just be a fine
6:13
tuning of a model. And wow, he brought a whole new
6:19
way of looking at data pipelines.
6:21
Or ML pipelines, LM pipelines, whatever you want to call it, to
6:26
my mind and my view. And I really liked it.
6:32
Hope you enjoy this. Let's get into the conversation.
6:36
But first I gotta get paid. So let's hear a word from our
6:39
sponsors. All right, let's take a minute
6:41
to talk about our sponsors weights and biases.
6:44
It seems like everyone's working on large language models right
6:47
now, but understanding what to build with LMS and how they work
6:52
under the hood can be tricky. Thankfully, weights and biases
6:56
can help. You might know about weights and
6:58
biases Flagship experiment tracking product in the LM
7:02
space. Teams like Open AI and Cohere
7:05
rely on weights and biases to train and debug their models.
7:09
But weights and biases? There's plenty more to offer you
7:14
if you're thinking about training, fine tuning, or prompt
7:17
engineering in LLM on your own Weights and Biases Traces lets
7:22
you uncover granular insights about your LMS, providing a
7:26
visual representation of their execution flows, colour coded by
7:30
each component in the chain. This helps you understand each
7:34
step your LM takes on its way to generating its output, giving
7:38
you a deep, nuanced view of its behaviours and letting you
7:42
iterate and debug much more quickly.
7:45
Weights and Biases, LLM production monitoring enables
7:49
the creation of custom dashboards to monitor your LLM
7:53
activities in real time, including throughput, token
7:58
usage, cost, latency, and errors.
8:01
That sounds so good, and if you need help getting started with
8:05
weights and biases, they can help too.
8:08
On that front, they've released multiple free classes for
8:11
machine learning communities like ours, the ML OPS community,
8:15
teaching them how to build LLM as intelligently approaching
8:21
prompt engineering, and a whole lot more.
8:25
Whether you're a seasoned pro or just getting started on your LM
8:29
journey, you can find what you need at weights and biases.
8:33
You can get started today going to one DB dot ME/ML OPS
8:41
underscore traces, or free today.
8:45
Get after it 1B dot ME/ML OPS, underscore traces.
8:52
Let's get back into the podcast. I was calling it dispy, but you
9:04
said it's not dispute. What is it?
9:06
Yeah, it's the ESPY like numpy. DSP.
9:10
I like numpy. Of course it all makes sense now
9:14
and that's why, yeah, there's the capital.
9:16
And have you ever thought about calling it dispy?
9:22
We did. And the story here is that the
9:28
first version of the framework was called demonstrate, search,
9:30
predict. So that's the and we got too
9:35
many questions and too many people who thought it's just
9:38
about digital signal processing and there's absolutely no
9:42
relation there. And so in the second version, we
9:46
wanted a rebrand that nonetheless keeps the people who
9:50
are familiar with the original version can relate to this.
9:52
So DSPI was it. Then you pivoted.
9:56
Basically, you did the good old pivot and you said, hey, we're
10:01
gonna go with DS pie. I like it.
10:03
Well, dude, Omar, I'm stoked to have you on here, man.
10:06
Give me the quick rundown. I know you're doing stuff at
10:09
Stanford right now, but you've got all kinds of great things
10:13
happening in your life. I would love to start at how you
10:17
got into tech. Well, I started building small
10:24
software, you know, back home, I guess as a teenager and I did
10:29
see as an undergrad. Um, and what I wanted to do in C
10:35
was mostly research oriented stuff.
10:37
Basically I like building systems that have novel
10:40
components, and in undergrad I did mostly research on sort of
10:46
optimising the efficiency of AI applications and from there I
10:52
worked on information retrieval. So making search faster.
10:57
But this was before the AI boom, I imagine.
11:03
This was before, well I I mean I've moved as many many of these
11:08
cycles I guess, but certainly before the the most recent one.
11:11
And this was you know 20/16/2017 and I started doing IR in 2018
11:16
from a system standpoint and then I joined.
11:19
I joined Stanford from my PhD and birth had just come out in
11:24
NLP. So language models, large
11:26
language models by some standards, where kind of
11:29
starting to look like they will be the future and they are the
11:32
present now, but they weren't really sort of mainstreaming in
11:35
search yet. And justice around that time the
11:40
IR community, the information retrieval community started
11:42
looking at birth for ranking, but it was this sort of artefact
11:48
that is 1000 times more expensive but a lot better in
11:53
quality than whatever sort of we had before in in search.
11:57
So I started my PhD thinking about can we make you know
12:01
language models for search stuff or there's of magnitudes faster
12:05
and cheaper while also pushing their quality up and that was
12:09
Colbert. And from there we started
12:12
looking also at you know this is work with my advisors Matesa
12:16
Hadia and Chris Spots. We started looking at building
12:20
these retrieval based NLP systems.
12:22
So people talk about RAG all the time now.
12:25
So that type of thing, building these retrieval based systems,
12:28
especially in cases where the language model actually needs to
12:31
interact in complex ways with a search engine.
12:34
And from there, this is where ultimately the Spy came out as a
12:40
programming framework that generalises years of our work
12:44
into a highly expressive and powerfully optimising
12:51
programming model. So you could build the type of
12:54
stuff that would take us months before in minutes or hours and
12:59
compile them into really high quality null systems.
13:04
Wait, so I gotta ask. Of course.
13:07
Cold air. Is that a shout out to the good
13:10
old Stephen Colbert with somebody, a fan?
13:14
Yeah, I mean it's that is the the, the, the context of the
13:18
name. We saw that birth was all around
13:20
and you know it stands for contextualised late interaction
13:24
over birth. And The funny thing here is the
13:27
paradigm that we introduced is called latent interaction.
13:30
And so it was just right there, you know, late interaction and
13:32
the Late Show nested jokes. Yeah, yeah, yeah.
13:36
It just gets, keeps getting better.
13:38
What did you see as the evolution there?
13:42
Because, as you mentioned, OK, rags are all the rage now, but
13:46
when you were doing it with Burt, what was it and how has it
13:50
changed? Yeah, it hasn't fundamentally
13:52
changed. I say people are doing the very
13:54
basic versions of things that that that's what got popular,
13:57
which, which you know, over time you realise that's really what
14:00
happens, is that the basic thing gets popular because it's very
14:02
easy for people to sort of grasp it.
14:05
Um, so generally back in like 2019-2020 and and around that.
14:11
You know around that time and and and really all the way until
14:14
last year. In fact the second-half of last
14:16
year all of the RAG stuff with very, very few exceptions were
14:20
about fine tuning Retriever and a language model together.
14:26
Usually very small models such that the language model could
14:30
condition its generations or like bases generations on the
14:34
retrieved passages. What we started seeing last year
14:37
when we let a lot of this is using few shot prompting as a
14:42
way to use language models for conditioning on retrieval.
14:46
And this is what people mean when they say ragnell.
14:50
Technically rag is the name of a method, a great work from Meta I
14:55
that is a very specific way of fine tuning A generative model
15:00
with a retriever together. So it's not really what or at
15:03
least generally fine tuning A generative Retriever, you know
15:07
conditioned on a fine tuning A generative model language model
15:11
conditions under Retriever. And what we did basically April
15:16
2022 was for the Natural Language Understanding course at
15:21
Stanford. We released this course project
15:24
as open source and we widely announced that as the first open
15:29
source sort of artefact where you could prompt GPT models and
15:34
local I think they were GPJ models so that they answer open
15:38
domain questions through retrieval.
15:40
And this is basically you know, the foundation of a lot of the
15:44
recent and popular stuff around drag.
15:47
But the whole area of retrieval based NLP more broadly than RAG
15:51
has a lot richer pipelines. So the questions there are
15:55
things like how do you supervise the retriever so that it solves
15:59
complex tasks for your generator Usually.
16:04
You know, maybe you wanna check complex claims or there are sort
16:08
of dependencies of knowledge so you can't actually find the page
16:11
you're looking for until you find some other dependencies
16:14
first. So that's called multihop
16:15
retrieval and and adapting retrievers.
16:19
There is a very significant challenge that is, you know,
16:22
something we worked on for a while and this is kind of
16:25
missing in the recent developments in open source in
16:29
general, besides a few key key artefacts.
16:33
So this is fascinating because it's basically saying like, hey,
16:36
the rags that we know and love today that everybody's talking
16:39
about and writing about are one style and maybe not even the
16:44
style that Meta I put out back in the day when they wrote about
16:48
Rags. But we kind of hijacked that
16:50
name and we call it one thing, which is basically you just like
16:54
go and look for some information in a vector database before you
16:57
give a prompt to a large language model.
17:02
And you're mentioning that's like a very basic method of
17:06
doing retrieval. What are some other.
17:08
Yeah. And you also mentioned how there
17:09
there is much more complicated methods, one of them being you
17:14
gotta get some dependency, so you have to figure out some
17:17
piece of information before you can move on to the next piece of
17:19
information. What are some other ways that
17:22
you've seen or you've been playing around with like super
17:25
complex retrieval? Yeah.
17:28
So there is 2 sides to my work that I think are going to, you
17:36
know, add a lot to the sort of application layer developers
17:41
we're looking at rag things. So these are research ideas
17:45
around what we call late interaction.
17:48
And that is basically at the level of how do you model, how
17:53
do you model retrieval. And the sort of popular approach
17:58
that people are familiar with is you encode documents as a
18:03
vector, so you're producing A dense representation for each
18:06
document. And similarly for the query.
18:08
And late interaction says it's actually really a significant
18:12
burden and really hard for these encoders to produce one vector
18:15
for each document which might be relatively long and that is
18:18
supposed to capture every possible query you can ask about
18:22
that document and nothing else. So late Interaction says we
18:28
should actually be representing documents as a matrix of many
18:31
small vectors, so a very, very tiny sort of representation at
18:35
the level of the tokens, not at the level of the document.
18:37
And if we sort of optimise that, well, with the right
18:41
infrastructure we can get this thing that is as fast and as
18:46
efficient as encoding each document with a vector.
18:50
But that has much higher quality, especially in domains
18:54
without a lot of training data. I saw a recent paper that was
18:57
basically saying if you fine tune cool there with I think it
19:01
was like 1000 queries. It does as well or better than a
19:06
single vector representation fine-tuned with 50,000 queries.
19:09
So there is a 50X sort of data efficiency gap there according
19:14
to that recent work. So that's one I think direction
19:17
that people should increasingly pay attention to.
19:20
And a different line is sort of putting emphasis less on the
19:24
specific models, language models and retrieval models in general
19:29
and more emphasis on your programme.
19:32
So on your pipeline. He liked to draw an analogy to
19:37
neural networks. When you're building, you know,
19:40
we've all sort of now come around to or see Transformers as
19:46
sort of this architecture that works in general and works
19:49
really well. But for the longest time people
19:51
were just exploring all these feedforward neural networks
19:55
where you're composing a lot of layers together.
19:57
Maybe you have like convolution layers, you have recurrent
20:00
layers, you have linear layers and then you have all these
20:04
transformations like dropouts and.
20:07
Other activation functions and all that stuff and and clearly
20:11
the power there in general is not in finding like one really
20:15
great layer or optimising the parameters of a great layer.
20:18
It's about the death and composing a network out of these
20:21
modular pieces. And what we're saying is that
20:24
increasingly in language model land, it's not about using GPT 4
20:28
or the right or fine tuning with the right RMHC or whatever it
20:33
is. It's really about building the
20:34
right architecture for your target task and application and
20:38
maybe down the line a general purpose one.
20:41
And so that is where things like multi hop interactions come in.
20:47
So the idea is we'd like to basically be able to build these
20:53
higher level architectures where language models are simply
20:57
devices that can execute these instructions, where the focus is
21:01
not on, like prompting the model as carefully as possible with
21:07
magical once or or or fine tuning it and you know the exact
21:11
right way. It's really about breaking down
21:13
problems so that this powerful contextual processor that we
21:21
have can make sense of complex tasks.
21:24
And that's where something like ESPY comes in.
21:26
It gives you a programming model for programming these language
21:29
models into pipelines. Dude, a lot to take in here.
21:34
And of course I was stuck when you were talking about I was
21:40
expecting you to have some word for the rag averse or something
21:44
that you were gonna say these rags that we come on and the
21:48
that's kind of the the way that we're looking at rags these days
21:51
is through this one encoding the the whole document as a vector
21:57
as opposed to this matrices that you're talking about.
21:59
And so there, let's just harp on that point real fast.
22:04
It's not, you're not saying that, OK, vector databases
22:07
aren't going to be needed. You're saying that the way that
22:10
we create the vectors and then throw them into the database can
22:13
be fundamentally looked at in a different style.
22:17
Right, that's exactly right. So you still, you still you
22:20
still need some infrastructure to do the search and right now
22:24
Colbert is an open source project that has that
22:28
infrastructure. We've written a lot of papers on
22:30
making it really efficient and both in terms of latency and
22:36
storage costs and otherwise. But this is you know many, much
22:39
of that is actually integrated in vector database called Vespa.
22:44
I believe they are also open source and they have offerings
22:47
that are commercial and you know maybe you can expect more in the
22:51
in the future. In general though, this is this
22:57
is a fundamental redesign of the search stack.
22:59
But at the same time, from an end user standpoint, it's a
23:02
black box. So it's the same.
23:04
It's the same behaviour. You're sending search queries,
23:06
you're getting back results. But the encoding of these
23:12
documents is redesigned in such a way that our encoders, our
23:15
language models, get to focus on representing documents
23:19
contextually, as opposed to doing this combinatorial process
23:23
of just trying to find representations in one fixed
23:26
embedding space for, you know, entire documents.
23:29
Yeah, which ultimately leaves a better end user experience
23:33
because they're going to be getting more relevant and
23:36
retrieved information and faster if I'm understanding correctly.
23:40
It's not faster, but it's it's about as fast it is.
23:43
The quality though that is that is that can be a lot better,
23:46
especially in harder domains and domains with little data.
23:50
No. OK, OK, cool.
23:51
And I gotta ask before we move on to the next point around this
23:55
system, because this ML hub or the like, the hub that you spoke
23:59
about is incredible too with Burt.
24:03
What is the reason for using Burt as opposed to now?
24:08
There's been a tonne of new open source large language models
24:11
that come out. Have you thought about trying to
24:14
do stuff with those? Or is it that Burt is the right
24:17
size and the right the right model because of the way that it
24:22
performs? It's good, it works.
24:24
It's smallish. So it depends which points
24:29
you're asking about. If you're talking about like
24:31
building pipelines of language models, birth birth can't really
24:34
take you further. But if you're talking about
24:37
modelling retrieval, so for example if you're talking about
24:40
DPI, yeah, you gotta use Lama to or Mistral or GPT models.
24:47
If you're talking about modelling search and in
24:50
particular encoding documents for retrieval.
24:52
Up until now, birth is both birth like models.
24:57
I don't mean actual original. Birth are still both the right
25:01
size and the right pre training and the right architecture
25:06
combination. There has been some cool recent
25:09
work that looks at like using lama two models for this.
25:13
I'm not impressed yet. Wait, why not?
25:15
Well, because I mean people release their artefacts and you
25:18
can play with them and sort of see what's the trade off.
25:21
And so you have these models that are.
25:23
I don't know is that like 700 or whatever 70 times bigger, but
25:29
they are like 2 points better on some benchmarks and so retrieval
25:35
is a special problem because the emphasis is really on scale.
25:42
Like you can throw all the kitchen sink at your problem and
25:47
maybe get one extra point of quality in some cases.
25:50
The the real the real deal is in things that could process, you
25:54
know, millions of query queries over millions of documents in
25:58
like 10 milliseconds or 20 milliseconds.
26:00
Because you have a lot of concurrent users, you have a lot
26:03
of documents and so you know, it's not the sort of thing like
26:08
for, you know, reading the documents where you can read 10
26:13
or 20 documents and get really great results.
26:15
So it's a fundamentally different game.
26:17
That said, I do expect that we will see you know more GPT style
26:23
models entering the core. You know encoding and retrieval
26:28
art of IR with higher quality fairly soon.
26:34
And with that will that will that will spur is people who
26:38
will start to think like, hey, how can we actually make that
26:40
efficient, which is that cycle. You know that we see a lot in
26:44
mill but especially in IR. That said, I think my bigger
26:50
message is this specific model you're using for language
26:54
modelling and to a large extent for retrieval isn't actually the
26:57
key focus going forward. You still want to use the best
27:01
components and we build a lot of the retrieval site there.
27:04
So we'd like people to use it. But fundamentally it's really
27:08
about how you bring these things together.
27:09
And so, for example, you might have a limited retriever.
27:12
But what a lot of the research is showing is that if you
27:15
generate good queries for a fairly limited retriever, you
27:19
could get around a lot of these limitations.
27:20
And while you don't want to use a limited retriever, the bigger
27:25
picture here is it's not about the representations, it's really
27:28
about getting the right workflow.
27:31
So if you're trying to maybe Fact Check a complex claim, you
27:36
shouldn't expect this. You know, magical vector that
27:40
can take any string you have and justice find all the right
27:42
sources for you without any additional intent.
27:45
You really should expect this pipeline that breaks this thing
27:48
down, runs multiple queries, aggregates the results, compares
27:51
the documents, and focusing on optimising the quality of that
27:55
sort of thing is what I've been doing for the last year and a
27:57
half or so. Dude, So again, it comes back to
28:02
looking at it as a system, looking at it as a whole.
28:05
And this is a perfect segue into that second part of this hub.
28:09
Break that down more like, what does that look like?
28:12
You kind of let us in. You did a little foreshadowing
28:15
right there, right where you said, OK, let's if you're trying
28:19
to Fact Check something, you might want to look at the
28:21
pipeline this way. What are some other ways that
28:23
you've seen pipelines being built so that they are more
28:26
resilient and more reliable and just more able to do what we
28:31
want them to do? Awesome.
28:34
So maybe. This is a good place for a
28:36
little bigger context O what we've seen over the last, you
28:43
know, maybe seven or eight years or whatever it is that you want
28:47
to start with a pre trained language model and then adapt it
28:50
to whatever task you want to do. And adaptation for the longest
28:54
time has been through fine tuning, which is this data
28:56
intensive, complex process that needs a lot of expertise and it
28:59
is fragile and slow ways. And then prompting came along,
29:04
right. So that's maybe three years old
29:06
and you know especially in the general form and and it really
29:11
only started properly working like two years ago.
29:14
So what we started seeing in 2022, especially an increasingly
29:20
now is this notion of prompting techniques.
29:23
So people have been you know researchers and practitioners
29:26
and everyone interested in this space.
29:28
I've been proposing these sorts of like conceptual ways of
29:32
inducing behaviour from language models by talking to them in the
29:36
right way. And so chain of thought is 1
29:39
building agents React and self refine and all these cool
29:42
things. Retrieval augmented generation
29:44
through prompting is another here in this space.
29:49
But increasingly I think people that are coming to the very
29:51
correct realisation that that can only take you so far because
29:56
the language models don't actually know everything.
29:59
It's not about talking to them in the right way.
30:01
It's a very fragile process. So increasingly what you want is
30:04
to prompt model in the right way and stack them in the right way.
30:08
So we're seeing things like if you want to generate SQL queries
30:12
from your model, you don't just want to ask it with a nice
30:15
prompt. You probably want to structure
30:17
this into a process, and we're seeing research in that in that
30:19
space that will handle different types of queries differently or
30:23
will actually look at the schema and like the, you know, the
30:26
parts of the database that you have and the columns that you
30:29
have more carefully before it generates queries.
30:32
And maybe it could great stuff and revise them or other things
30:37
like that. We're seeing tonnes of this when
30:38
it comes to retrieval. Lots of pipelines that sort of
30:42
generate queries, edit them, find citations and all that
30:46
stuff. So with these pipelines in their
30:51
in the wild, what you actually see is that to implement them,
30:55
researchers and open source solutions alike, what people do
30:58
is they write strings, they sit down and they open up Python
31:02
file or whatever JSON file and they start writing these kind of
31:06
handcrafted prompts. They don't work.
31:08
They edit them a little bit until it starts working for
31:10
their favourite model and then they connect these pieces.
31:13
They chain them to build these pipelines and that is bad.
31:19
You know, that's bad for many reasons.
31:20
First of all, super brittle. The language model changes under
31:23
the hood. You get harder questions or
31:25
different questions, the domain changes.
31:27
Someone was working in a a general domain that wants to do
31:30
biomedical stuff. Your prompts don't work anymore.
31:32
So it's super brittle. But more importantly, it's also
31:36
scalable. So every time you want to change
31:39
this pipeline, are you going to edit all your prompts?
31:44
And that's especially important in pipelines because if you are,
31:50
if you have like 3 or 4 stages of calling the language model,
31:53
you know this third stage doesn't have any labels
31:56
inherently. It's just trying to connect the
31:57
pieces. And so if the other, if the
32:00
pieces around the change, there's no guarantee that
32:03
whatever prompt you wrote initially is actually going to
32:06
be the correct glue in between. So this is super brittle and our
32:10
solution to that is sort of keeping the focus on building
32:14
these architectures and building these pipelines but shifting it
32:18
from kind of a hand tuning game of open-ended three form prompts
32:24
to a more structured and automatically optimizable view
32:28
of the world where give you modules And these modules are
32:33
going to represent you know all of the conceptually clean
32:39
prompting techniques that are out there.
32:40
But they're actually well defined functions that have
32:44
these parameters that can be optimised automatically.
32:46
And so you can say I want a chain of thought module that has
32:50
the following signature. Maybe the signature is it should
32:52
take a question and some paragraphs and give me an answer
32:56
for example. And that's all you specify.
32:58
And then you can reuse that in your pipeline as many times as
33:01
you want, you know in loops or in complex control flow.
33:04
And once you're done, you can give it your data and say all
33:07
right, please optimise my pipeline for my data given my
33:11
language model under the following constraints.
33:14
And that's something we know how to do automatically as opposed
33:19
to just leaving that to the developer who now has to write
33:22
all these strings. So it's just a significant step
33:27
forward, I think, from writing prompts as the main way in which
33:32
we build these systems, which I think is just a really bad idea,
33:36
yeah. It's not prompt first, it's
33:37
really workflow. And again coming back to the
33:40
system first and thinking about it holistically, I'm going to
33:44
just keep repeating myself because I like the way that you
33:46
think about it. And I I see the pain every day
33:50
in the MLS community, community when people are talking about
33:53
how brittle those workflows are. And so that's one of the main
33:57
reasons why I wanted to talk to you and you did gloss over
34:01
something though that I want to go over and it's about stacking.
34:05
Can you explain this this idea of stacking more?
34:10
Yeah, so the general idea here, people call it stacking,
34:14
chaining networks, programmes, but it's fundamentally the same
34:18
thing. It's that you have this language
34:20
model and for the longest time we thought of language models in
34:25
NLP including fine-tuned ones as like these task specific
34:29
Transformers. But now you can think of them as
34:32
these devices that you can programme.
34:34
And so instead of thinking, oh, I have a language model that I
34:39
could use as a chat bot and I can ask the questions you really
34:42
want to think of it as like, Well, I'd like to write a
34:44
programme. I'd like to build a system that
34:46
solves my task or solves my problem.
34:50
What are these little sort of primitives that I'd like this
34:56
model to softly implement? And that any unique thing here
35:01
is that for a lot of things that are fuzzy, for a lot of
35:05
computations that are fuzzy, you want to work with.
35:07
You don't want to work with strings and integers.
35:09
You want to work with questions and answers and messages and
35:12
queries and SQL queries. And as opposed to, I don't know,
35:17
JSON output and all these sorts of things where there is some,
35:22
you know, limited hard constraints on them.
35:25
Like Jason, valid JSON versus invalid JSON is a is a pretty
35:28
hard constraint. But fundamentally though it is
35:31
fairly fuzzy. Like what is the right, Jason to
35:34
have? That's not the thing you could
35:36
just easily automate the check of and so stacking language
35:40
models into fairly complex pipelines, basically arbitrary
35:44
code, maybe recursion or loops or whatever.
35:46
But we're language models are responsible for these fuzzy
35:49
computations that is the world that we're moving towards
35:54
increasingly. And our sort of big message is,
35:59
yes, we should move there, but that shouldn't be done through
36:02
prompting, it should be done through programming model that
36:06
sort of codifies and standardises the way we interact
36:10
with these things such that we can automatically compile them
36:13
or optimise them. Yeah, so if I'm.
36:15
Understanding this correctly, really it's like trying to
36:18
change the chip in my mind or anyone's mind who who's
36:24
listening as to, hey, we're not just going to hit GPT 4 and
36:28
expect it to do everything because we magically came up
36:31
with the perfect prompt that solves all our problems.
36:35
We need to look at how we are going to be trying to.
36:40
Almost constrain these different large language models to where
36:46
they are doing the one thing that we're trying to get done
36:51
and like you said it's it's the building blocks that we can
36:53
stack stack on top of each other.
36:56
The in my getting that right First off and next I guess I'll
37:01
ask you a question just to make sure I'm clarifying that and
37:04
yeah you are you. You're exactly right.
37:06
So the focus now is like, what is the workflow, what is the
37:09
code that can solve this problem, and which are the
37:12
little hard arts that they actually couldn't just easily
37:14
write myself in Python? And how do you know what is the
37:17
right signature there and what is the right module to ask a
37:21
model to do that? But the focus is on your entire
37:24
workflow, so this could be just Python or I don't know,
37:29
TypeScript or whatever it is, and the language model is just
37:31
handling these relatively well defined or much better defined
37:35
now pieces in that bigger workflow O.
37:39
You're not going and saying to the language model, hey do XYZ
37:44
and expecting the auto GBT to bake you a carrot cake because
37:49
it says OK well 1st I'm going to buy carrots and then I'm going
37:52
to get them ordered to your house and then I'm going to have
37:54
the robot machine cut the carrots and put it into with
37:57
some flour and stuff. It's realising that large
38:02
language models have a function that you can give them and they
38:05
are really good at playing with, as you mentioned, fuzzy things
38:11
that aren't so clear and aren't as well defined as something
38:18
that you could just code with Python or any other language.
38:22
And so the more that you're able to just plug it in to your
38:26
pipeline and recognise that we're only going to have it do
38:30
this one thing and that's all we want.
38:32
And then we move on to the pipeline.
38:34
And maybe the next step in the pipeline is something that's
38:37
just hard coded and there's no machine learning or AI or
38:39
anything involved in it. And then the next step in the
38:43
pipeline might be a new set of large language model calls that
38:47
you're doing. And so you get to construct that
38:49
because you're the architect of that pipeline.
38:51
And my one thing is that automatically comes up is how it
38:59
seems like we could get a really expensive bill coming in because
39:03
we start to really like modular lot module.
39:07
I don't know how to say that word, but we could make a lot of
39:11
calls to large language models and then recognise like, oh man,
39:15
I have to be smart about how I'm using these large language
39:19
models calls because it's starting to get expensive.
39:22
Have you seen that? Like have you have you
39:24
recognised like, ohh, yeah. Well, now that I've got this
39:26
pipeline, I could do anything. And then next thing you know in
39:30
one pipeline you're making like 20 calls.
39:33
Yeah, of course this is something that we this is a
39:35
central concern that we tackle. So the key observation here to
39:40
keep in mind is that when you're building these pipelines,
39:44
there's 2 dimensions to this. The 1st is pipelines inherently
39:49
require a lot less complexity from the language model.
39:51
When you're breaking things down the individual steps, they
39:54
almost never need GT4 for the full pipeline.
39:57
There's at most in a typical pipeline, there's at most 1 coal
40:01
that actually needs to go to a really powerful language model.
40:03
And that's because most of the steps are, hey, let's break this
40:06
thing down, let's put these things together, and that's not
40:10
that hard. The second thing which is a lot
40:12
more important, is that these pipelines are optimizable.
40:15
So when you say I want the chain of thought module.
40:18
MSI. What that means under the hood
40:20
is I want this thing to actually teach my language model how to
40:24
solve this step of my pipeline by thinking step by step.
40:28
So this is not like inserting in the prompt.
40:31
Let's think step by step. It's actually hey, can it
40:34
internally come up with a few short examples that show
40:37
whatever model it is, how to think step by step for my task?
40:40
So that could be by few short prompting or it could be through
40:42
fine tuning. And that's all done internally
40:44
in the programming model in the spy.
40:47
And so actually happens is when you're compiling programmes like
40:51
this, you can make them really small in terms of the models
40:54
they use. Like we can go like the FLAN T5
40:57
lawyers for example, which has 700 million parameters, million
41:01
not billion, right. So you're going to things that
41:03
are, you know, maybe a thousandfold smaller than GPT 4
41:06
or even smaller. And and because they're working
41:10
at the granularity of like I want to solve this one step, not
41:13
like this overarching task. And because they're sort of
41:17
compiled on your data with your model and automatically you can
41:22
actually get pipelines that are a lot cheaper than calling a
41:24
model. And this is this is where this
41:27
really ends U connecting all of these pieces.
41:31
Dude, well, we've gotta go into what I brought you on here to
41:35
talk about this beat or DS pie, as you like to call it.
41:38
I'm gonna call it this beat because that's what I had in my
41:41
mind before you showed up. But the give us the breakdown.
41:45
I mean, you've showed a little bit of it, but now like Open the
41:51
Corona, what exactly is it there?
41:53
Are these modules and modules I'm kind of thinking of as
41:57
templates that we can use as a nice starting point and then we
42:02
can tweak those. Is that how we should look at
42:05
them or should it be a little bit different?
42:08
I think it's a little. Different.
42:10
So let me tell you the way things are done now and what
42:12
people mean outside of the spy when they talk about modules,
42:15
just for context. And this is probably familiar to
42:17
many in the audience. So if you go to a, you know a
42:20
another library for building with language models, they'll
42:24
have modules. But these modules are going to
42:26
be things like the module that generates queries for SQLite,
42:31
and there's a different one that generates queries for MySQL.
42:34
For example, there's a module that does self ask.
42:39
There's a, which is a particular kind of workflow for working
42:42
with search engines. There's a module that does.
42:45
Let's see particular type of rag in DSP, that's not the
42:51
granularity of the modules that we have.
42:52
And the reason for that is if you go and look at these modules
42:55
in the libraries that provide this type of thing, that's
42:58
basically a wrapper around the prompt or two and then some kind
43:03
of tool use that allows the language model that they built
43:06
this around to conduct that particular pipeline.
43:12
And so it's really helpful if you just want something off the
43:15
shelf that you can stack with other pieces.
43:18
But if you want to build your own workflow and have a lot of
43:22
control over it, it's not actually that helpful because
43:24
you will have to open it up, start changing that prompt the
43:28
moment things go slightly wrong. In the ESPY our focus is on
43:32
finding the right modules and the right primitives that are
43:36
task independent and in fact task adaptive.
43:39
One way to think of this is how the neural network sort of
43:42
development process and how frameworks around that have have
43:45
sort of developed. We think we take them for
43:47
granted now, but you know this stuff actually took a while to
43:51
reach a consensus. So the idea here is I don't want
43:54
my modules to be. Here is a pre trained resnet and
43:57
here is a pre trained birth. I want my modules to be.
44:01
Here is layer that I can reuse and train for each task
44:06
differently. So in our case, for example,
44:08
prompting techniques, our modules and.
44:10
So if you want to do a chain of thought, that's a module.
44:12
If you want to do a programme of thought, that's a module and
44:16
that is not a task specific thing.
44:18
A programme of thought module just says I want the model to
44:21
solve a particular subtask by generating code and executing
44:25
that code and then interpreting the output of that code.
44:28
That's a very general process, and the way it works in DSPI is
44:31
that you are going to declare whatever modules you're going to
44:35
use. So there's a bunch of built-in
44:36
ones. You can easily add new ones by
44:39
using existing modules to build bigger modules.
44:41
And these modules, what makes them really general is that they
44:44
take this thing called the signature.
44:46
So a signature is just a specification of the input
44:49
output behaviour. You can think of it as a prompt
44:51
that is not written in around hundreds of words, but is
44:54
written around, you know, 3 or 4 words.
44:58
And so that's basically a function signature that says
45:00
something like alright, for this module what I wanted to do is to
45:04
take a long document and give me a summary or and that's
45:07
literally it, like long document summary, or it takes a question
45:11
and maybe some paragraphs and gives me a response or whatever.
45:16
You know gives takes an English thing and gives me its German
45:19
translation. And this is just trying to
45:21
specify the input output behaviour there, because you
45:24
want to be able to interact with this thing as a function.
45:26
So just as with a normal function signature, you might
45:29
say I should give you a list and you give me a string.
45:32
That's the type of signature that we have here, except that
45:34
the types themselves are specified in natural language.
45:38
Once you've done that and you give that to the module, you
45:40
have this function that the module gives you that you can
45:44
call. So if you call the module that's
45:46
a chain of thought that answers questions, you give it a
45:49
question. Internally it does a train of
45:51
thoughts I think, step by step, and it gives you an answer.
45:54
And by declaring several of these modules, you can start
45:58
writing your code in any way you like.
45:59
You could have like recursion and loops and exceptions where
46:03
you are calling these modules in the right places of your code
46:06
and they're acting as these sorts of language model
46:11
computation devices that you can call at the appropriate places
46:18
and that gives you a programme. Now the third and the last.
46:21
And the key thing in DSPI that makes all of this work is that
46:25
once you've written your programme, you can compile it.
46:27
And by then get means, hey, I have, you know, I'm actually
46:31
working in some context. So there is some data I work I
46:34
want to work with or I want to process, you know, my internal
46:37
database or whatever it is. And what the compiler is going
46:40
to do is it's going to say, oh, you have 3 modules, like you
46:42
have this chain of thought module, you have this programme
46:45
of thought module, whatever. You have this other agent that's
46:48
right there and you're using them in this pipeline on this
46:51
data. Each of these modules has some
46:53
parameters that I can optimise. So in particular, each of these
46:56
modules is actually sending a prompt to to a language model
46:59
that Prompter 2 is not something we wrote.
47:01
Like if you open the SPA, there isn't a single prompt in the in
47:03
the code base. These prompts are going to be
47:05
generated. It's through an algorithmic
47:07
process that is going to call the language model on your
47:10
pipeline, like execute your pipeline on your data several
47:13
times, potentially with high temperature, and it will keep
47:16
track of successful traces through each module.
47:20
So every time it goes through a module, it sees.
47:22
Oh, when I called it with these inputs, I got these outputs and
47:25
ultimately it passed all the constraints.
47:29
So I I had a metric that I wanted to maximise and that it
47:32
raised my score. It will cache those and it will
47:35
basically be able to reuse it later after compilation in such
47:38
a way that the whole pipeline maximises its metric.
47:44
O Basically we've taken this prompting technique that is
47:47
right now an idea that you can sort of sit down and type as a
47:51
string. And we've said, well, what are
47:54
the parameters of this idea? Well, there are a few short
47:56
examples. No one is going to give us
47:58
those, but we can generate them through the pipeline itself.
48:01
So we can bootstrap them and maybe some instructions and
48:04
other pieces and we've converted into like OK, how can we
48:07
optimise these pieces automatically so that you're
48:10
just focused on when I use this module with this signature in
48:14
the in this workflow and you know you just compile it with
48:18
your favourite data and language model without sort of hand
48:21
engineering like data labels for all these stages or prompts for
48:25
all these stages etc. And what that gives you is that
48:29
compiling takes the same workflow that you wrote and you
48:32
can actually map it to fundamentally different and
48:34
powerful things. For example, the same code that
48:36
you write could be a zero shot prompt to GPT 4 because you want
48:40
to keep your prompt short or a few short prompt to Loma two
48:43
that has like several samples and it's very careful about how
48:45
to optimise that so that Lamma 2 understands your task.
48:48
Or a fine tune of local language models that are small, like flat
48:53
five that is under a billion arbiters in size.
48:56
And it's the same code because fundamentally you're just saying
48:59
I want the module that asks this language model to solve the task
49:04
that has the following structure by thinking step by step.
49:07
So the key thing here is that with an abstraction like this
49:10
these modules are highly declarative.
49:13
And so when I say they want a chain of thought module, that is
49:18
going to take maybe a document and give me a summary through
49:22
thinking step by step. As far as the compiler is
49:25
concerned, you are free to specify that you want to do this
49:28
by compiling into short zero shot or one shot prompt for GPD
49:35
4 because you want to keep the prompt short because it's
49:37
expensive. Or a long few short prompt for
49:41
llama to, for example, you know 7 billion because that's you
49:45
know, cheaper. But also you want to give it a
49:48
complex enough prompt that shows exactly what you want.
49:51
Or by fine tuning a small model, like a model with hundreds of
49:57
millions of parameters, not billions, so that it follows
49:59
that task. And I think this is something
50:01
that really surprises people a lot and is a key strength.
50:07
And ESPY is that there's no difference between prompting and
50:10
fine tuning when you're not doing either.
50:12
You're just saying, I want this module that has these pieces and
50:16
I want to interact with the language model, you know at
50:19
these intersections of my code to solve the following sub task.
50:23
Doing that through prompting or fine tuning is almost equivalent
50:26
at that point when you leave it to a compiler.
50:28
Because in both cases what the compiler is going to do is it's
50:31
going to simulate your pipeline on a lot of examples and it's
50:34
going to keep track of inputs and output behaviour of each
50:37
module and in particular cases that are actually effective.
50:41
And with that you could either fine tune the model on those
50:44
inputs and outputs or prompt it with them, or you could even do
50:47
like learning based on preferences.
50:49
You could say like this output is better than that one.
50:51
We're not doing this at the moment, this last one, but it's
50:54
a very easy extension. So it is basically an
50:57
abstraction that takes us to a different way of thinking about
51:01
interacting with language models.
51:04
So this compiler idea is fascinating to me because it
51:07
reminds me of basically how you get to a point in time and then
51:11
boom, there's infinite options. Almost.
51:14
I mean not infinite, but you get many different options where it
51:17
explores what is the best route. Here.
51:20
Let's just go and really explore our options and change the
51:24
temperature, change all the turning, all these knobs and
51:28
then you see which ones are coming out and which ones are to
51:33
the liking of the end user I guess.
51:36
Or is it to the liking of the actual model and?
51:41
So these, these ideas are sort of all subsumed here, but the
51:45
way in which we make it, I think this is also a key contribution.
51:49
The way in which we make this whole thing tractable is by
51:52
analogy to neural networks. So when you have a neural
51:55
network, it has a gazillion parameters.
51:58
You don't generally sit down and think.
52:00
It's not even tractable to sit down and think, what should I
52:03
set all of the weights to today, right.
52:05
You say, well, what is my loss function?
52:07
What am I trying to optimise this network for?
52:10
You say what is the architecture?
52:11
Google, I gotta decide the layers and how large they are
52:14
for example. And you also say what is my
52:16
optimizer. So different optimizers can have
52:18
fundamentally different behaviour, like they want to do
52:20
basic SGD, they want to add some momentum or add them or
52:23
whatever. So the same thing is the message
52:26
we're pushing here. You are going to decide your
52:29
workload, your workflow or your programme.
52:32
No one is going to, it's all automated.
52:33
You still need to decide that code, but it's going to be like
52:36
15 lines of code, 20 lines of code.
52:38
And then you are going to have to decide what are you
52:41
optimising, like what are you optimising for.
52:43
So you need a metric, you know, I mean, are you trying to get
52:46
the answers to be correct? What does it mean for the answer
52:48
to be correct? That that itself could be a
52:50
different programme that you just use off the shelf or that
52:53
you write yourself that determines that process.
52:55
And now that you have a programme and a metric, you need
52:58
to pick the right optimizer from what we are, what we provide.
53:01
Or you could write a new one if you want.
53:03
And the combination of these three pieces over your data and
53:06
your language model is going to hill climb in the right way.
53:09
So for example, one particular optimizer, we go, we go we call
53:14
these teleprompters because they do prompting at a distance.
53:17
O1 particular teleprompter is called bootstrap view shot with
53:22
opportuna. O what that's going to do is
53:24
it's going to simulate your pipeline on a bunch of inputs
53:29
and then it's going to keep track of the full traces through
53:32
all the modules whenever the metric, your metric likes the
53:36
output. So likes the output could mean
53:38
hey the answers are right or it could mean I actually don't have
53:41
labels at all. But the answer is fulfil the
53:43
following criteria as tested by other DSM 5 programme, when it
53:48
when it finds those outputs, it'll basically cache them and
53:51
then it will go back in the pipeline and say oh what if I
53:53
used one or more of these as psot examples in each of the
53:57
modules. Does the average score go up now
53:59
as far as my metric is concerned and that is an automatic
54:03
optimization process. In fact it's like a
54:04
hyperparameter search that an off the shelf thing like
54:07
opportunity in this case could run a sophisticated sort of
54:12
discrete search algorithm under the hood, you know, can optimise
54:15
this process, but there's a lot of other teleprompters that are
54:17
already there or things that we're adding O you know there
54:20
are teleprompters that are based on fine tuning.
54:23
So you're not trying to update the few shot examples, you
54:27
actually trying to update or basically update the weights of
54:30
a smaller language model in order to conduct each of these
54:35
steps that are modules. And the idea is not that your
54:40
first attempt will be this perfect programme that is
54:43
infallible. It's really more like we give
54:47
you the the way to iterate, right?
54:49
So you build this programme, you test it prompting.
54:52
It works well, but it's too expensive.
54:53
Try fine tuning. It works well in most cases, but
54:56
it's making a consistent Tye of mistake.
54:58
Well, you have two well defined options, or at least better
55:01
defined options. Do you want add an exception in
55:04
your code that deals with that edge case?
55:06
Or do you want to add you want to change the module that you're
55:10
using here to a more resilient module?
55:12
Or do you want to change the metric so it catches those
55:15
errors and optimises them away? Right now I think what is
55:19
happening in practise is a lot less structured than this.
55:22
Is that oh, I'm seeing a mistake.
55:24
Maybe if I did this prompt in just the right way or wait for
55:27
the next update to GPD 4 maybe things will be better.
55:30
Or if I get the bigger pay for the bigger model, maybe I'll
55:33
just solve my problem and we're saying, you know, you actually
55:37
have a lot of options and we can structure them.
55:39
It's so funny because. Earlier this week I was thinking
55:42
about how fine tuning, I think it's thrown around a lot by
55:45
people because it's like as a last resort, it's like, damn,
55:49
my, my rag isn't working. And so it must mean that I have
55:53
to fine tune this and then maybe what I want to happen will
55:57
happen, you know, and hopefully a lot of people out there.
56:00
That's not the case. But I do feel like I've seen
56:03
that enough to where it's like people are scratching their
56:06
heads and they're trying to do exactly what you're talking
56:09
about, where they're using the framework and they're saying,
56:12
hmm, I guess I gotta like tweak this prompt to our guests like
56:16
they're trying to debug why it's not actually coming out with
56:19
what needs to come out or what they're hoping for.
56:23
And it's really not as granular as you're you're talking about.
56:27
And it's really kind of just like throwing spaghetti at the
56:30
wall, right? And and.
56:31
And I think a lot of that spirit is needed.
56:34
Like that debugging mindset is good, but you want to have some
56:39
structure to it. And I think your example on fine
56:42
tuning is great because maybe fine tuning is the answer for
56:45
you. But do you really want to work
56:47
in an abstraction where fine tuning is a complete revamp of
56:50
everything you've done, where you throw it away and you start
56:53
from scratch to build this fine-tuned that may or may not
56:55
even work? Or do you want to work in an
56:57
abstraction where it's just the same code, just flip out the
57:00
teleprompter from the one that's doing future learning to the one
57:03
that's doing tuning and test it out?
57:05
I mean, maybe it doesn't work right away, but you didn't
57:07
didn't cost you anything. And in terms of complexity.
57:10
And maybe you know what you want to do is a different
57:12
teleprompter that does a different type of fine tuning.
57:15
And so adding structure into this space and thinking about it
57:18
as an optimization problem helps a lot with the problems that you
57:24
just brought up. Dude, it's funny because I was
57:27
on a podcast a while back to and I was talking about how fine
57:31
tuning is a lot more of a headache than most people think.
57:35
And it is something that you probably don't need.
57:39
Because just with the normal and what I understand now is a
57:43
simple rag, it's usually good enough and it's like you can go
57:51
really far with a simple rag, right?
57:54
You can get it really far. But then people again like to
57:58
think that Ohh and I gotta fine tune this and I want to figure
58:02
that out. And so like I'm I just need to
58:05
throw some data at it. But people don't think about how
58:07
much work goes into a like gathering the data and cleaning
58:12
the data and getting all that data, like preparing it for the
58:15
fine tuning and then the cost of actually like fine tuning it,
58:20
right. And so in one way, potentially
58:22
our minds have been tainted because we had stable diffusion
58:27
before we had fine tuning on LM and we were like ohh, stay fine
58:31
tuning. Stable diffusion is just like a
58:33
Google collab, you know, like I can throw a few photos of myself
58:37
in there and then boom, I prompt it with my name and I've got
58:41
this incredibly fine-tuned text to image model and it works
58:45
pretty well. But then when you try and do
58:48
that same fine tuning on a large language model, it's not exactly
58:52
one to one comparison at all, right?
58:54
But what you're talking about here, I think is is fascinating
58:58
because I love the idea of how you want to make sure that fine
59:03
tuning in DSP does not allow you to forget about anything that
59:11
you've already created. In fact it's it's a way to
59:13
augment what you've already created so that you're able to
59:17
say, OK, well it's not really working with my rag.
59:21
I've tried all these different things.
59:23
I've tweaked it. I feel like I could get there.
59:25
But as that next step you say, I'm going to try and fine tune
59:30
it and see if that works and you do understand it's it's much
59:34
more. Again, it's a much more modular
59:36
way of looking at a whole system as opposed to going in and
59:40
trying to figure out if that prompt that's been hard coded
59:43
needs a little tweaking. And then if it does get tweaked
59:46
and you figure out, you have to dig through documentation after
59:50
documentation to find that prompt that is at the bottom of
59:53
it's the needle in the haystack. And then you say, OK, well you
59:56
know what, Now I'm I can tweak that prompt and then it doesn't
1:00:00
actually come out better. Yeah, that's exactly right.
1:00:04
And I think to your point about the hard coded prompt, people
1:00:08
are doing a lot of hardcoded fine tunes too.
1:00:10
And that's also bad when you're fine tuning in D and this is
1:00:14
just a general message, even if someone doesn't use DY, at the
1:00:17
end of the day the key artefact you're producing is your system.
1:00:20
Your system might use one or more fine-tuned models.
1:00:23
These models are not your system, they're just sort of
1:00:26
processing parts of the kind of components or the modules in
1:00:30
your system. So when you're fine tuning in
1:00:32
the ESPY, actually you're not fine tuning the programme.
1:00:35
The programme is still there. It's just that every time it's
1:00:38
going to call the language model, it's going to call an
1:00:40
appropriately fine-tuned language model.
1:00:42
O That's the an important kind of shift from the way you know,
1:00:46
in general, as you've said, people are thinking about these
1:00:48
fine tunes as these kind of end to end solutions, which they're
1:00:52
not. Yeah, well, I I love the
1:00:55
versatility on this too, how you don't get trapped into one
1:01:00
workflow for one model for one use case.
1:01:04
It's really generalizable in that way.
1:01:06
And Speaking of generalizable things, I think I would be
1:01:10
remiss to not ask your opinion on agents and how your
1:01:16
experience with them have been and what you've been up to in
1:01:19
that realm, because it feels like this very much is to help
1:01:24
move the ball forward in that space.
1:01:27
I think agents are a good idea and if they were great, I would
1:01:32
have said great Agents or workflows.
1:01:35
They are just programmes like any other except that they
1:01:38
usually have a loop and they sort of their modules are making
1:01:44
decisions about coding tools. So they're just like any other
1:01:47
type of programme. And so we've written agents in
1:01:49
DSPI. You can compile agents in DSP
1:01:52
because they're just programmes. The thing about agents is that
1:01:56
it's kind of like we just this is an analogy, so hopefully
1:02:00
people can sort of follow along. It's kind of like we just
1:02:02
discovered neural networks and the first thing we wanted to
1:02:05
build is the right Transformer architecture.
1:02:07
But we have no clue what convolutions are, are, what
1:02:10
attention is what anything is. We're just kind of jumping right
1:02:13
away to build the GPS of neural networks.
1:02:17
But in the land of language models, meaning agents are this
1:02:20
general purpose architecture that is supposed to be able to
1:02:24
solve all kinds of tasks in the general purpose way.
1:02:27
But as a community, we still actually haven't fully figured
1:02:30
out how to solve individual, well defined tasks.
1:02:32
First, jumping into agents at this point is someone needs to
1:02:37
explore. I'm not against exploration at
1:02:39
all, but from a deployment standpoint, 90% of things that
1:02:43
you're that people are doing don't need agents and are going
1:02:46
to get better performance if they don't use agents.
1:02:49
That said, I'm all for exploratory science and research
1:02:55
and just development in general. And agents are just programmes,
1:02:59
they're just workflows that you can compile.
1:03:01
So if you're going to build an agent, don't write the prompts,
1:03:05
select your modules, connect them in the right way and
1:03:07
compile appropriately. You might get a fine-tuned
1:03:09
model. You might get a sort of sequence
1:03:12
prompts that optimises your agents for the actual goals that
1:03:16
you care about now. One thing that this will make
1:03:19
clear is for some cases a lot of the exploratory, exploratory
1:03:24
research or just discussion around this is so open-ended
1:03:27
that it's not actually trying to optimise for any metric.
1:03:30
So if you want to build this agent that could do everything,
1:03:33
I think a programming abstraction like DSCI at least
1:03:36
makes it very clear what you are optimising for.
1:03:39
Or if you're not really optimising for anything in
1:03:41
particular. At least that's a decision that
1:03:44
you'd have to be making, which I think is is still valuable as an
1:03:48
exploratory thing, but maybe not really for deployment.
1:03:52
I love this. Conversation that we've been
1:03:54
having. I love that you came on here and
1:03:56
you talked to me about all of this and you explained DS π in a
1:04:00
way that was very understandable for my thick skull.
1:04:05
Right. And I appreciate you doing that.
1:04:07
Yeah. Thanks for thanks for.
1:04:08
Inviting me it was. It was great chatting, chatting
1:04:11
about this with last question I got.
1:04:13
For you is if you weren't doing the work that you're doing now,
1:04:16
where would you be focusing your time and energy?
1:04:19
And I'm mainly thinking about in the AI sphere, like if you
1:04:23
weren't working on DSI, what other or retrieval, right?
1:04:26
Where would you be looking? Yeah, so my time.
1:04:28
Is spent 5050 between these two? If I wasn't doing the spy, I'd
1:04:32
definitely go all in on on building better retrieval
1:04:35
models. If I'm not doing either, I know
1:04:39
what I will not be doing. What I will not be doing is
1:04:41
trying to build better language models because I think I think
1:04:44
it's all in the system architecture.
1:04:46
But yeah, if I'm not doing this I I'd be releasing colder V3
1:04:50
sooner rather than later. Incredible.
1:04:54
And there's no plan right now. I mean to go and make a run at
1:04:59
it with DS Pie and create some company around it, No?
1:05:04
There, there are no such plans. This is open source research we
1:05:07
want. We definitely want to keep it
1:05:09
around. This is, we're looking at this
1:05:10
as like the pie torch of this new language model space.
1:05:13
And so we want to support all kinds of innovative abstractions
1:05:17
and research and development, and we certainly want to keep
1:05:21
this around for years. But no commercialization plan.
1:05:27
Incredible. Well, Omar, man, thank you for
1:05:29
coming on here. We'll end it there, dude.
1:05:31
Yeah. Thanks A.
1:05:32
Lot.