Latest / Agentic Conversations (formally mlops.community) / Treating Prompt Engineering More Like Code // Maxime Beauchemin // #167

Transcript

Treating Prompt Engineering More Like Code // Maxime Beauchemin // #167 · Agentic Conversations (formally mlops.community)

0:01
My name is Maxine Bushmans. I'm French Canadian, but I I go
0:05
by Max is just easier and I am CEO and founder now at a startup
0:10
called Preset. So we do offer a managed service
0:13
around Apache superset, patchy Super set.
0:15
I mean if I'm going to say something that's like non
0:17
commercial about some of the stuff we're building is actually
0:21
Super Set is a very, very strong open source contender in the
0:25
business intelligence space like call it data consumption, data
0:29
exploration, dashboarding. So we're very like.
0:31
Superset is a very strong open source competitor to Tableau and
0:37
Looker and all the business intelligence tools around.
0:40
So if you haven't checked that open source BI or patches 2% in
0:43
a while, you should check it out.
0:44
It's really, it's becoming really, really solid and really
0:47
competitive like and better than these like commercial vendor
0:50
tools in a lot of ways I appreciate we offer a freemium
0:54
type service on top of it. So it's like so, so you can get
0:57
started very quickly and and try it.
1:00
So encourage people to take a look at that.
1:02
And then? The way I prefer my coffee, I
1:07
don't know. I mean, I do espresso with oat
1:10
milk, little honey, that's that's been my jam lately.
1:16
Welcome back to the MLS Community Podcast.
1:19
I am your host Dimitrios flying solo.
1:22
Today I'm talking with Maxine. You all may know him because he
1:26
came on the podcast about a year ago and he is also the creator
1:30
of a little tool that used and he decided go and check it out.
1:43
It's on the last podcast that we did with him.
1:45
But he talked to us about why he did not create a whole company
1:48
around airflow like Astronomer, like a managed airflow
1:52
basically. And he went off and did his
1:54
other project which is around as he mentioned in the intro around
1:59
Apache Superset created preset and he's been killing it with
2:03
that. But today he's here to talk to
2:05
us about promise, which is a little tool.
2:07
That he created for internal use.
2:10
And then he said, you know what, this is actually really useful.
2:13
Maybe I should just open source it to the world.
2:15
So we went through what exactly Promise is, why he decided to
2:19
open source it, why he open source basically every tool that
2:23
he creates and also how it can help.
2:27
And I absolutely loved how Promise can help you, it seems
2:31
like. If you are trying to figure out.
2:35
In a very scientific way, if you're prompts are useful and
2:40
how useful they are when you change prompts.
2:42
When you change different dials, Promise can help you view that
2:47
and see that and create logs. He calls these prompt suites and
2:52
he draws the parallel between test suites in regular software
2:57
engineering and you have all these test use cases and you
3:00
have prompt use cases that you can line your code up against or
3:05
the different prompts and you can.
3:07
Evaluate models and you really have your own valuation and
3:10
benchmark metrics when you're looking at whether or not you
3:14
want to use a new model or if for a certain use case, a
3:20
certain type of prompt is going to do better than other prompts.
3:25
It's a space that is absolutely blowing up right now.
3:28
I know there's a tonne of other companies that are popping up in
3:31
this field because. There's pain around it.
3:35
We don't have clear ways of interacting with the models that
3:39
are, as Max was saying, super scientific and can give us clear
3:45
feedback. So Max dives into it and.
3:50
I think you all are gonna enjoy it.
3:52
Let's get into it. Before we jump in one one ask
3:56
from my side, just one That's all I ask for is that you share
3:59
this with one other friend if you liked it, We are here.
4:03
Like subscribe and leave a review.
4:05
All that fun stuff. But really share this with one
4:08
other friend because it would be great to get all of the cool
4:12
stuff that we talked about today with Max out to the world.
4:16
All right, let's jump into this conversation with Max, the
4:19
creator CEO. And just all around awesome guy
4:24
of great to have you back the first time that you were here.
4:34
You absolutely blew my mind with so much cool stuff.
4:37
We talked about air flow and preset and why you went off and
4:42
did preset. I think that was the coolest
4:44
thing that is forever etched in my brain and the TLDR for people
4:48
who didn't see that first episode.
4:50
But I highly recommend everyone goes and watches it is you kind
4:55
of just said yeah, but I have more fun doing stuff with
4:58
preset. And I really think that's a
5:00
better challenge that I want to sink my teeth into.
5:04
And so I love that it just reminded me to work on things
5:07
that you enjoy and do things that you like.
5:09
It doesn't necessarily mean you have to do things because of
5:14
whatever the external circumstances are really like
5:16
follow what you actually like. So I'm excited to talk to you
5:21
today about your new little thing that you've been getting
5:24
into and you've really been liking.
5:25
And I I really want to talk about like this whole.
5:29
So it's in promise right? Then my saying that correctly?
5:33
Yeah, I think so. I mean really I think we should
5:36
kind of probably go back a little bit to the beginning of
5:40
like how how did that came to be and what what's that set the
5:44
stage for for this. Really the premise is like
5:47
everyone is trying to build to bring AI into the the products
5:51
and the software that they're building today.
5:53
And then how do we do this right.
5:55
We have this new beast like the the LLM's and and they're very
5:59
like you can ask them anything and they can answer.
6:01
Anything there's a lot of power to that, but how we channel
6:04
this, this power, how we bring that into our products is, is
6:07
very, very challenging. Yeah, so true.
6:10
And you mentioned before, we hit record just the whole API.
6:14
And the way that we interact with the API is a whole new
6:17
paradigm too, because you can kind of just say whatever you
6:21
want. You're speaking to the API in
6:22
English in a way, and then good luck getting out something that
6:26
is very consistent each time. And so there's a lot of big
6:31
question marks that I think people are still exploring to
6:34
this day. 100% yeah, if you think about your normal REST API
6:38
and your your general API that we have.
6:40
That we've had access to to build products so far.
6:43
It's very, very structured, super well documented.
6:47
There's a really clear schema as the input.
6:49
There's a clear schema as the output.
6:50
Super deterministic. Correctly asked the same
6:53
question five times to get the same answer five times.
6:57
With these LM and the GPT type interfaces, you're free to
7:02
construct your prompt however you want.
7:04
Now there's some limitation of as to how long the prompt can
7:07
be. Amish context you can pass, and
7:09
you really gotta use that smartly.
7:11
So that's one of the challenges there.
7:13
But it's like provide whatever context you want.
7:17
Or you feel is relevant and the the structure that you feel like
7:21
it's going to be more intelligible to the to the AI
7:24
and then get the result that you.
7:27
Ask for and then you can ask for some structure on the on their
7:32
response. It's possible to say like a
7:33
return using the following format or in less than 200
7:37
words. So there's a lot you can do in
7:39
the prompt to shape the output of what it does.
7:42
But it's still extremely probabilistic and an open field
7:46
kind of problem. Like where do I even start with
7:49
this thing and how do I measure whether my prompts like, is this
7:53
problem better than the previous prompt I had?
7:55
Anecdotally, it might be, but it might not perform well.
7:58
Has some specific sub use cases, so the whole idea would promise
8:03
is to just to bring some of the test driven development
8:06
mentality to be able to say hey if I switch this if I switch my
8:12
prompt a little bit. If I ask a slightly different
8:16
way or if I use GPT 5 turbo instead of GT4, how How is it
8:23
gonna do at my use cases or my test set?
8:26
Did you said something there that you just kind of glossed
8:28
over it? But it is very important to
8:32
highlight which is. The way that we have been doing
8:36
things when we are. Hitting APIs that it's been very
8:40
well documented, but now when you are basically given infinite
8:45
possibilities of how you're going to interact with the API,
8:48
how can you? Even think about documenting all
8:51
these different edge cases and so it's so hard to think about
8:55
it you get. Companies like Open AI.
8:58
They're coming out with, hey, here's some prompt guidelines.
9:01
Here's how we think. We've seen the best ways that
9:04
you can use it, but it is. It's not like you can just say,
9:09
all right, cool. I have.
9:10
I noticed this when I when I'm playing with the tools.
9:13
Sometimes I think is it because the model isn't there yet or is
9:17
it because my prompt isn't good enough?
9:20
And it's and it's probably a combination, an intricate
9:23
combination of the two, right? And without a scientific method
9:27
around your prompt or or or or rigorous method around like
9:32
trying different version of your prompt.
9:34
You're not going to be able to to to really know, right?
9:39
That's the whole point there is like, OK, how do I?
9:42
So let's say in in my specific case and scratching my own itch
9:46
there, we were working on text to SQL, right?
9:49
So I think that's really it's an interesting problem when you
9:53
think about text to SQL and let me state like try to explain
9:56
what I mean by that. But the general idea, you have a
10:00
large database with a bunch of models and tables inside this
10:05
database. And then you wanna ask it a a
10:07
text question, human language question, as in what are my in
10:12
which countries are my my sales growing fastest this year,
10:17
right? Or you want to ask like all
10:18
these generic questions and you want for the AI in this case to
10:22
generate SQL that would answer this question with a result set?
10:28
So we count this problem is pretty open-ended problem and
10:31
there's like an infinite amount of question that people could
10:33
ask. And there's different database
10:35
schemas. It's pretty open-ended, but it's
10:37
nicely deterministic in a sense that for a question you're able
10:41
to say how did the AI perform? Did it write SQL that answers
10:46
the question accurately? Yes or no.
10:49
I think that's the perfect kind of problem.
10:52
I think for for a well, it's not necessarily the perfect, but it
10:55
is a a good kind of problem. For say product builder and
11:01
being able to assess to do some prompt engineering where you can
11:04
assess is it answering the question well or not so.
11:09
Because it's black and white. Yes, it's it is like not
11:12
necessarily perfectly black and white, but it is It is fairly
11:16
deterministic. You can say it's exceeded or
11:19
mostly succeeded or completely failed.
11:21
Right Used. Did the sequel run?
11:23
Did it have the answer and it as opposed to write a 200 word
11:28
essay, there are thousand word essay on the.
11:31
Napoleon's conquests in Europe. There it's it's a lot harder to
11:36
evaluate like did it do well or not right.
11:39
Like you need a human type interpreter that an expert to to
11:43
read this stuff or the SQL you can a little bit more say like
11:47
it did well or a little bit more black or white.
11:50
Yeah, that's a good use case. And then the first thing I was
11:53
like, OK, what kind of this problem?
11:54
I need to build a Bank of test cases that are deterministic,
12:00
right? The first thing I got to do is
12:01
like find a test set of say 105 hundred 1000 different prompts
12:08
on a various set of different databases that have different
12:12
maybe nomenclature, naming conventions, level complexity,
12:18
like something might be normalised in different ways I
12:20
need to find. This test set, so I can actually
12:23
evaluate whether the AI is doing well or not, and then whether it
12:28
might prompt is working is doing what it what it should be doing
12:33
so I can iterate on it. Ohh, I love that.
12:36
Yeah, yeah, yeah, yeah. I see exactly what you're
12:39
saying. So basically you you were like,
12:42
OK, if we can figure out all of these different edge cases, then
12:47
it's going to be a much more robust system when I am passing
12:50
it into the L. Right.
12:53
Or if I can have say a test set of say let's say I have 1000
12:58
different prompts with 1000 corresponding queries that I
13:01
believe are are believe are right or answer the question.
13:05
Well now I can go and. Iterate on my prompt first,
13:10
right? I can generate a prompt that's
13:12
like OK given these table schemas, right?
13:15
Like given this database structure and given these
13:20
constraint. Can you answer the following
13:22
question right like what are my sales by department's year over
13:27
year growth? And and then I can.
13:32
I can edit my prompt. I can run this test suite
13:35
systematically. And I can get a report that says
13:39
like ohh this this version of your prompt is 72% accurate.
13:44
I can point it to say a different.
13:48
A a different model, maybe a different GT model or the latest
13:51
whatever Facebook is coming out with or data bricks of these
13:54
open source model and I can run the same test suite.
13:58
With the same prompt and say which one performs better or or
14:02
even more interestingly, to which?
14:04
Where does this one fail and this one succeed and where this
14:08
the opposite is true too and try to understand why and iterate
14:12
over that front and the selection of model.
14:16
So the first thing that comes to my mind is how does this
14:19
interact with a vector database? Because it's like vector
14:22
databases have become the unsung hero of prompting and the LLM
14:28
movement, right? Right.
14:29
That's what's interesting is that.
14:32
A little bit the same way that you think of it testing
14:35
libraries. If you think of like what does
14:37
this this solution that I call promise that I created.
14:41
It's really just a place where you can express your your test
14:45
cases called your. I call them prompt cases, but
14:48
input from. And eval function, right?
14:52
So so you'd say OK for this question.
14:55
You know for for this input question that the user may ask,
15:00
here's the eval function. So you provide this testing
15:03
library with a with your own prompt engineering toolkit,
15:07
right? So you might use Lang chain if
15:09
you're familiar with that. That's a comprehensive little.
15:13
Super super useful little Python library to interact with open AI
15:17
and craft intricate prompts, right?
15:20
So you have these templates, you can do chains of prompts, you
15:23
can create agents so you can a lot you can do with laying
15:26
chain. Might a library.
15:28
All it does is allow you to express your test cases a little
15:33
bit like π tests right? Or these like very simple
15:37
library where you just write your unit tests and then you're
15:40
able to run your suite of tests. So.
15:44
My library doesn't is not a prompt engineering library per
15:47
se as much as just like a prompt engineering testing toolkit and
15:50
evaluation. Ohh, I get it.
15:55
Yeah. OK.
15:56
And that's where. Yeah, now it makes complete
15:58
sense where you're saying, alright, this one is 75% useful
16:02
or this prompt is, is this good. But then when you point it at a
16:06
different model, it holds up much better or much worse
16:09
etcetera, etcetera. And so you can effectively know
16:13
these things in a much. Clearer way than if you're just
16:18
kind of evaluating from looking at the answer and that's really
16:23
hard to scale if you're saying OK now let's see if I want to do
16:27
these 200 how can I evaluate all of these?
16:30
Well if you set up a this promised test suite then you can
16:35
know that right away that. Yes, you can do it.
16:38
It it brings them the mindset of if you're familiar with
16:40
hyperparameter tuning ML, I'm sure certain you are in the
16:44
audience is really often you want to train.
16:47
A whole like matrix like multidimensional matrix of model
16:50
and then get compare how they're performing against one another
16:55
right So so if you you you can kind of think of like this
16:59
mindset of I'm going to try different prompts maybe against
17:02
different models and with different parameters.
17:06
Alright so you can do that easily with promises cause it's
17:09
all test cases at define as code.
17:12
So if you want to say I want to run this this test suite against
17:15
these five different models with these five different parameters
17:19
for temperature with some variation on on the prompt, it
17:24
might might get a little expensive on that API cost.
17:26
If you're 1000 tests then you do multivariate big matrix of test
17:31
but enables you to to do that and to produce these.
17:35
Monies for. What was that, Sir?
17:38
That's what VC money's for. That's it exactly.
17:42
But yeah, so so you can produce these reports for each test, for
17:46
each prompt suite that you run, and you'd be able to say, OK,
17:51
this is this combination of parameters is the one that
17:54
performed the best. And here's where maybe another
17:58
model really succeeded in this one failed, right?
18:00
Like, give me some examples of that.
18:03
Compare these two diff these two reports and tell me how GPT say,
18:08
when it comes out, GTA5 is performing against four, right?
18:12
Is it better at? What is it worse at what is the
18:15
cost? Right.
18:16
So we'll log some of the costs and performance parameter.
18:20
Which one is faster? Which one is more expensive?
18:22
Which one is more accurate? This, this sort of nice, right?
18:26
Allows to bring this more deterministic scientific method
18:30
to something that's super mushy, right?
18:32
We talked about how these APIs are.
18:34
Are like there's no structure you can do.
18:36
It's open playground and do whatever the heck you want.
18:40
But then you're like OK, if I write important and caps in my
18:43
prompt, how does that really does that affect?
18:45
Does that change anything? What does it change?
18:48
Yes, Ohh, I love it. And that is, I completely see
18:52
the vision of, hey, let's make this much more scientific so we
18:57
can at least measure one against the other, because we're not
19:00
quite sure right now what each word is affecting when it comes
19:05
down to it. Is it actually helping my
19:07
prompt? How much is it helping the prom
19:08
door? Is it hindering it?
19:10
And so with this, effectively, you're saying, yeah, you can see
19:14
that in a very clear way. Yeah you can have a full report
19:18
of like for each test case, how much, how it performed versus
19:22
the other one. You can get the summary result.
19:24
You can push your reports into database and run SQL against it
19:28
to get really intricate. So it's really brings rigour to
19:33
testing your your your prompts producing reports.
19:37
And then and then allowing you to to evolve and iterate on
19:40
that. No.
19:41
You talked about like say vector databases and I dodged a
19:44
question a little bit. I'd love to get into that too.
19:46
But the first thing is like, should you, should you use a
19:49
vector database? Yeah, probably if you're prompts
19:53
if you're context window is limited for the kind of use
19:57
cases you have. But how would you let's say
20:00
you're like, alright what I'm going to do the work I'm going
20:01
to set up my. My embeddings and I'm gonna hook
20:05
up like pine cone DB or whatever it is right and then push my
20:09
documents and do some proximity search on these vectors and you
20:14
do all this work you like wait this that perform better than my
20:17
simple prompts that I just had before and in what way right
20:22
where does it perform better than the other one.
20:25
Well promise would allow you to compare like how your your your
20:29
prompt that uses a vector database might be performing
20:33
against your prompt that doesn't.
20:35
And as you iterate on these embeddings and these document
20:37
formats. You can.
20:41
You can measure whether you're doing better or not.
20:46
Man, I love this dude. And so it's worth saying for
20:49
everyone that this is fully open sourced.
20:52
I I guess you just have open source in your blood, huh?
20:55
You can't release something without making it open source,
20:58
can you? I mean, let's go over it.
21:00
What airflow? Fully open source.
21:03
What? Then you have Superset.
21:04
Fully open source. Yeah, and now that Trump
21:08
promised fully open source. Yeah.
21:09
When you think about this discovery promises like I feel
21:11
like that what's more interesting is the idea or just
21:14
the scope of the project is much more limited the same way that
21:18
you would think like a testing library.
21:21
And I'm not saying like that those are unambitious or or
21:25
there's a lot of virtue to them. But the problem space of like a
21:28
little toolkit to to run some tests and produce reports it's
21:32
much simpler than the the air flow problem space and quite
21:37
frankly is like. As a founder, thou as proud of
21:39
the kind of time I can actually afford, This bed is working on a
21:43
small little problem. But yeah, I mean the whole thing
21:46
would open sources. If I what, what what?
21:49
Am I gonna sell this thing or just keep it for myself?
21:51
It's a lot more fun to build something relevant.
21:54
Come on the show, talk about it and see a see some impacts, see
21:58
some people using it. Building a small community of
22:00
people get value and I get a kick out of that.
22:03
That's why I do it. Yeah, I fully understand.
22:05
I'm huge proponent in open source.
22:07
I mean, I think that's how we drive the industry forward
22:10
because the masses creating things and hacking on things is
22:14
always going to be better than just a few people in a room
22:17
doing it and then coming out a few years later.
22:19
And saying, look what we have and it could hit or miss and
22:24
sometimes it hits, but I really think that the the open source
22:27
idea and having the community. Obviously I'm a community guy,
22:30
so I love when things are open source.
22:35
But how do how do you feel if we take the tangent on the whole
22:38
like, should we open source the full power of AI, right.
22:42
There's been a lot of recent debate on like do we we think
22:45
that I forgot what the stat was, but it's like. 80 Like as more
22:49
than half of the the ML practitioners or the the
22:53
specialists in the space thing, there is a 20% chance or greater
22:56
that AI will destroy humanity and then clearly.
23:00
Open source in an area like that where people compare it to
23:05
nuclear, nuclear kind of technology, they say like OHH,
23:08
should we open source nuclear technology or?
23:11
I don't know. I haven't really formed an idea
23:13
on as a pro open source person, but I certainly think there's
23:16
some. Seems like there's some some
23:18
potential being. There are some lines.
23:19
I think if you take it like down the negative spiral, then yeah,
23:24
it's true. I personally am like, but we
23:27
are. There are open source models out
23:30
there already. They're not as good.
23:33
It's pretty obvious that like what's out there on the open
23:36
source market is not as good as ChatGPT or GPT for that.
23:41
Pretty clear right now. Whether or not that's going to
23:44
change by the time we release this episode, that's yet to be
23:47
on scene. I think that it's coming up
23:49
pretty fast and there's a lot of people that are trying to create
23:53
things that are just as good. I am a huge proponent, proponent
23:57
of open source, and so I'm gonna say yes.
24:00
And the way that people use it, I think you can still use
24:03
ChatGPT in a very nefarious way also.
24:07
It's just that. For some reason people think
24:11
that they the. Overlords that oversee ChatGPT
24:17
and open AI. They can fix things when they
24:20
break or when they see like Ohh, there's a loophole that someone
24:23
discovered and now they're going to fix it.
24:25
But potentially a lot of harm can be done before that loophole
24:29
is discovered. So I'm.
24:31
I. AM and.
24:35
Let us get into danger everywhere on on open source.
24:37
I'm not open source. One scenario that we dodge
24:40
somewhat is, well it's one of those like Dodge, but like if
24:44
open AI was called called Closed AI and they found this this
24:48
amazing technology and instead of you know creating ChatGPT and
24:52
making it accessible to the world with guardrails, they
24:56
could have done like, hey, we've got this secret internal
24:58
technology and we're going to hire a bunch of people to that
25:01
are twice as productive given our proprietary tools.
25:04
Right. And we're not.
25:05
We're just going to use. Their own technology to better
25:10
the cause of this company and not the world and instead they
25:14
took the the the other decision of kind of getting it, getting
25:17
the the animal outside the box and forcing everybody else to
25:21
catch up. Yeah, kinda interesting.
25:25
Well, I mean, yeah, you can look at Google.
25:26
They kind of did that, right? They've had the technology
25:30
behind closed doors and they didn't really release it to the
25:35
rest of the world in a way that Open AI did until they had to
25:39
and. I mean, yeah, the, I guess the.
25:44
The thing goes back to like giving it to the masses and
25:48
creating a community around it, whether or not it's open source
25:51
or not. They saw how many different
25:54
applications could be used with this.
25:56
I think it was a I know, a friend of mine who worked at
26:00
Google, and his main job was figuring out how they could use
26:04
these large language models for what use cases they could use
26:07
them internally on different Google products for.
26:09
And this was in like 2018. And so that was his whole job,
26:13
is like talking with the researchers, being like, oh,
26:15
you've got this, OK, this does that.
26:16
OK, cool, let me go. Talk to product managers and
26:20
figure out if we can inject this feature into the product.
26:24
But then when Open AI came out with the ChatGPT revolution, all
26:29
of a sudden we see people hacking on top of it.
26:31
And you see there's a million different use cases that people
26:34
hadn't thought about, right? Yeah, can assist pretty much
26:37
anyone doing anything. That's what we're releasing any
26:39
anything that is like encodable and language like structure.
26:43
It can be useful in assisting people with which is like
26:47
infinite and like use cases. It's just insane, so.
26:52
Where this can be applied even even for for you and I don't
26:56
know if you had the same experience over the past like
26:57
since December but for me it's just like I've throwing
27:00
different things at it like now it's becoming almost first
27:03
reflex to be like all let's see if you know GP TRGT can help me
27:08
with this and it's certainly it's going to fail at this and
27:11
it's like Oh my God actually like really really great at
27:14
assisting me at all sorts of things which has been like.
27:19
It's it's been kind of it's been kind of crazy to experience
27:22
that. Be like all that's how good it
27:24
is at writing SQL or answering this question.
27:27
Or. Or debugging code, yeah.
27:30
ID 8 and complex concepts. And and and it's yeah, or teach
27:34
me Spanish. Shit, yeah, like.
27:36
Anything from that? Yeah, I do like that
27:38
Generalisation, generalizability of it, but getting back to like
27:43
promise, I think that's one of the things where.
27:46
It is fascinating because you you talked for a minute about
27:51
this. Test driven design that we
27:54
traditionally have in software and now you kind of came out
27:57
with a different approach and you're calling it it's like Rome
28:01
driven design I think or what it what is it called that you I
28:04
loved this idea. Yeah.
28:07
So it's like test driven development as applied to our
28:11
prompt cases, I think. Yeah, yeah, so did we have test
28:13
cases? We have prompt cases.
28:15
It's interesting because test cases are there's a lot of
28:19
parallel and invite people to check out the blog post talking
28:22
about all this and I'm trying to draw parallels.
28:25
Between Test Driven Development and Unit Testing and Prompt
28:29
Engineering. Then there's that hyperparameter
28:32
tuning angle too. So it's it's some.
28:34
It's not exactly the same, but it prompt I in a blog post I
28:38
list out the similarities and the differences between a prompt
28:42
case and a test case, and one thing is like a prompt case.
28:46
A test case is clearly like black or white like 0 or one.
28:49
It's a boolean like did you test the did, did you succeed at the
28:53
test and if the test fails you break the build.
28:57
You don't let people merge into master right like that.
29:00
Just like very much determined what happens if you fail the
29:05
test and prompted during is like 0 to one that it could be like
29:09
are you 50% succeeded the tests, are 75% succeeded the test and a
29:14
failure. You don't expect perfection
29:16
also. Right like you expect that all
29:20
you want is for your prompts to perform better than the previous
29:23
prompts. Yeah, for for you to release
29:27
them into production. So there's some parallels, some
29:30
differences, but you know you you need some reporting, you
29:34
need some sort of accounting on like how are your tests
29:38
succeeding, failing, so. So yeah, interesting,
29:41
interesting set of parallels, but a lot of differences too.
29:46
And I'm assuming that you're also using things like the
29:50
temperature and you're you're tracking everything right?
29:53
Like everything that goes into the prompt is being tracked.
29:56
Yeah, pretty much. And you can you can augment
29:59
these the reports and what right?
30:02
So so you derive a class that is expected to take some input and
30:06
generate a prompt and you attach evaluation function to this.
30:13
I've got a pretty good example on the repo to under example
30:18
build a little toolkit that does that.
30:21
Generates prompt to write simple Python functions, right?
30:25
So a prompt might be like general function that gives the
30:29
end number of the Fibonacci suite.
30:32
Or find the like write a function that finds the lowest
30:36
common denominator of two numbers and then so for each one
30:40
of these. Text prompt.
30:42
There will be an eval function that assesses whether it might
30:48
assess. Like if I pass these numbers, do
30:50
I get this number out? And all the mechanics of like
30:53
getting this function parsing into an interpreter is off the
30:58
library in somewhere like that's your own logic that you bring to
31:01
the library. That the promise prompt cases
31:06
would be maybe 100 simple function prompts with little
31:11
eval function that makes sure that it's running well.
31:14
And then when you when you derive that class it's pretty
31:18
easy for you for you to to log more things.
31:21
So by default we're going to log all of the the open AI type
31:25
parameters like temperature and how long it took to answer how
31:29
many tokens were used in the prompt right?
31:33
So we could we keep track of all of these.
31:35
Statistics and put them in a BLOB.
31:37
But you can augment that BLOB with whatever is relevant to
31:40
your prompt, right? So you can log your own type of
31:45
information you want to keep track of in your reports.
31:48
Uh-huh. Man.
31:51
Here's an interesting thought. I'll I'll take out like
31:55
something I've been thinking about.
31:57
Is a is the following. You tell me what you think about
32:00
about this idea, but. Everything is changing so fast,
32:03
right? Like in in terms of AI
32:05
development, LMS, there's gonna be like new.
32:08
We talked about vector databases as being a key part of the
32:12
solution. So that's new and emerging and
32:15
in the prompts are getting longer.
32:16
So the limitations are changing, the costs are getting cheaper.
32:21
Now there's all these open source models emerging that have
32:24
different constraints, so everything is changing really
32:27
fast. Maybe the only thing whatever
32:57
thinking is like, the only thing that is valuable in the world
33:01
that's changing constantly might be your test set.
33:04
It's right if you think about your like the the assertions
33:07
that you're doing to evaluate This is the only anchor in a
33:13
very kind of. Ohh, wow.
33:16
Yeah. Storm, right?
33:17
Looks like you're on this boat. The technology is changing very
33:21
quickly. And and it's really hard to
33:24
evaluate what you should use and how.
33:26
How you should do things, which model you should use.
33:29
And and maybe the most valuable thing in the in those times
33:31
where things are changing so fast is the anchor that tests
33:36
how well you're doing. Right.
33:37
Your tests. Your prompt case is your test
33:39
cases. Well, I love this because a lot
33:42
of people, it's not like we were just talking about earlier how,
33:45
yeah, you can do so many different things with these
33:49
models and that's part of the magic of it is that
33:51
generalizability, but. For the most part, for most use
33:54
cases, it's not like you in your company.
33:58
You're asking for five completely different things,
34:02
right? Maybe you have your one use case
34:04
that you're using, and you're heavily optimising for that one
34:07
use case, and then you may find another use case, and you may
34:10
find that. Open AI is not the right API for
34:13
that. You can actually do it with an
34:15
open source model, whatever it may be, and so you're figuring
34:19
that out. But in these?
34:21
Different use cases if you can create.
34:24
That test suite that is highly optimised for that specific use
34:28
case. Then you can swap out the models
34:31
so much easier because you know what the best model is for that
34:35
use case. Or again, what you're talking
34:38
about. Yeah, let's say like anthropic
34:40
says, all right, now you can just throw a whole book at it,
34:44
you can throw the Bible at it and not even worry, it doesn't
34:46
matter. We'll give you back whatever you
34:48
need and you can analyse, is it worth it?
34:52
Is it more expensive? Is it faster?
34:55
Is it slower? All that stuff that you really
34:58
want to know if you're trying to make a battle hardened use case
35:02
as opposed to just a Twitter demo, it's super useful and it
35:06
just shows like yeah, I really like that.
35:09
Idea is how if you build out your test suite.
35:14
You can know really quickly and you don't have to spend a tonne
35:18
of time evaluating which model is the best for this use case,
35:22
which isn't. Yeah.
35:23
And this, this, maybe this, this test library becomes, or your
35:28
test cases, your collection of like prompt cases, become your
35:32
single most important asset in a world where everything else is
35:36
evolving too fast, right? That's the only thing that
35:39
doesn't erode over time. You have to rewrite next week
35:42
when the next model comes in. And one thing we haven't talked
35:45
about. We talked a lot about prompt
35:47
engineering and how the key, I think the key is there's always
35:50
going to be a huge portion of key that's in the prompt itself.
35:54
But there's there's a lot in terms of like potentially fine
35:57
tuning these models, right. So I would take a base model
36:00
with a set of weights and then instead of passing a lot of
36:03
context and the prompt which is limited but you know arguably
36:07
growing the prompt limitation are growing.
36:09
Maybe the the real solution is to train further these models
36:12
down the line instead of like trying to like yeah here's the
36:15
whole Bible is like. How about I just train you once
36:18
in in terms of like reading this book, really understanding it
36:21
and then that's compiled in your in your weights and then their
36:24
neural net. So it could it could well be
36:27
that. Dissolution weeks or months or
36:30
years ahead will be all in the direction of like custom fine
36:35
tuning, custom training. You pick your base model and
36:39
you're still in a world where you need to assess these
36:41
problems and now they're doing too right.
36:43
So that library is still useful regardless of your your
36:47
approach. In the end, don't you?
36:50
Don't you feel like though, if you are trying to figure out
36:53
these new models and where they excel and where they don't, you
36:57
have these prompt cases, but you also want to add new prompt
37:02
cases because there are new capabilities.
37:05
Surely it's I think you're you're pro said I talked about
37:08
it like a like a super important asset if you're solving like
37:14
called call it a subset. You you take these generalised
37:16
model you take a more limited problem like generating Python
37:21
functions or doing text to SQL and that test library.
37:26
You should you can totally argument to be better and more
37:30
representative of your user your your actual user cases.
37:34
And what I realised is. Actually a I is really good at
37:38
generating these yeah from cases.
37:41
So one thing is like ohh should they be defined dynamically by a
37:46
nation And a I evaluate the AI as a question a different for
37:51
me, I like to have these test cases like defined as code and
37:56
static and being very deterministic because we're
37:59
trying to create some like some deterministic.
38:03
More deterministic outcome in this super probabilistic world.
38:06
So if he's like Ohh this AI is gonna evaluate this AI and then
38:10
you're still in that realm of like I don't really know what's
38:13
happening. So so there I like the idea of
38:17
like writing those as code right?
38:20
And putting them in a repository and evolving this this code base
38:24
or this this library of prompt cases.
38:27
What I realise is you can take say ten example, you do a few
38:31
shot example of like here's 10 test cases not getting you
38:36
generate 100 more and they will do that pretty well too so that
38:39
AI can help you come up with this library and the example I
38:44
had for the Python think for the for the Python use case which is
38:49
a prompt, a toolkit that allows you to ask for the AI to
38:56
generate. Small Python function are just
39:00
like deterministic simple Python functions I wrote like four or
39:03
five. Can you write a function that
39:05
the Germans whether it's a prime number, the number is prime or
39:09
not. So with four or five and I was
39:10
like here is four or five can you generate like a dozen more
39:14
And it did a really really good job at at that including the
39:18
evaluations methods. They generated promised code
39:22
using promise code and created these really great prompts that
39:26
then I put in the repository and.
39:29
I was able to use in my test suite.
39:32
So there is something that it's kind of changing gears, but I do
39:36
want to mention because we've kind of been dancing around this
39:40
for the whole last. 30 minutes that we've been talking and it
39:44
is all around how. What you've created.
39:49
If you can monitor the A speed and B cost, those are some of
39:56
the biggest things that people talked about when we did the
40:00
ALMS in production survey that we sent out to a tonne of people
40:05
and it was like basically speed, cost and trust in output or how
40:11
can I know that it's seriously? Yeah, accuracy in quotation
40:16
marks. Exactly hand quotes, because
40:17
it's a weird accuracy 1. And so.
40:22
You effectively are with this test suite and the the prompt
40:27
suite. You can help measure all of this
40:30
and you mentioned before that you can kind of bring UA
40:33
dashboard and see what it is. Is that with.
40:37
Let me talk about that a little bit.
40:39
So what the library does that is at its core is they will run
40:43
your test, your prompt cases. For you it's, it offers a little
40:46
bit of a DSL like how do you define a prompt case.
40:49
So it's something as simple as like what's the input and what's
40:53
the. What are the eval functions for
40:56
that input? So that's.
40:59
So that's a way to express in a very nice and syntactic sugar
41:06
away, like the same way that you have these test suites.
41:08
So you can like very easily represent your test cases as
41:12
code and and then it will run them for you, run the eval
41:17
function and produce a report. And that report is in the form
41:19
of a. It's a little CLI application,
41:22
so it's a little CLI. You're like, OK, promise.
41:25
Run, find all the tests in in this folder and run all the
41:29
tests and put the output report in some sort of temp file, some
41:34
sort of yaml file somewhere. So I'll put a yaml file for each
41:39
prompt case with a bunch of statistics in it.
41:42
Right? So these statistics are not even
41:45
statistic but just like logging. Essentially of like how each
41:48
case did. You can add some categorization
41:52
of your prompt cases, right? So you could add like complex
41:55
function. Simple.
41:56
Functions in Python. So you come up with your own
41:59
categorization of things. You come up with your own
42:02
weights. You say disc test.
42:04
There's like 5 times more important than the rest of the
42:06
test. Or this one is less.
42:07
Important. There's ways.
42:08
There's categories. Ohh, fuck this one up.
42:12
Whatever you do if you don't. Yeah that this one is like 100
42:16
times more important. If this one fails, you could add
42:19
more semantics eventually too. You could have a category that
42:21
says like if it fails, never, right, never publish the
42:26
library, call it like the kill switch.
42:29
But yeah, so you can so it produced this big yamil report.
42:32
And then giving these like YAML reports you can do with the CLI
42:37
compile some some high level statistics, things like what's
42:41
the percentage of success. Where the weights but but better
42:46
you can push. You can push all these
42:48
statistics to a database if you want to.
42:51
So that's pretty easy to just push these yaml reports with.
42:56
What is the prompt case? What was the input prompt?
42:58
What was the output prompt? Did it succeed or fail?
43:02
How much does it cost? How much did it cost to run?
43:04
How long did it take to run And open AI, How many tokens did it
43:08
use? So all these tokens are in a
43:10
YAML report. You can put it in the database
43:12
if you want to do more complex analytics.
43:14
On it, but I think the the CLI you can write promise reports,
43:21
pick and a yaml file as input. And ask for which statistics you
43:27
want out. And right now we just compile
43:29
and tell you like this test suite ran better than the
43:32
success rate is higher than this other one.
43:35
And I think there's more work to be done there.
43:38
So right now I pretty much just logs the the into yaml.
43:44
But it could do more things like diffing to outputs, right?
43:47
Maybe you run this week with the prompt or with say GBT 4, GPT
43:51
3.5. Now you want to, say give me
43:54
examples where one succeeded and the other one failed and or vice
43:59
versa to start understanding like, oh, it does much better at
44:03
this kind of tests, and here's why.
44:07
Yeah, well, it's we see it all over social media, right?
44:10
There's all these benchmarks on this model did this.
44:13
It performed better than GPT on X amount of questions or
44:18
whatever. But for most people, that
44:21
doesn't really mean anything because you want to test it out
44:24
against your use case. So it's very much like what
44:27
you're saying is, hey, we can basically give you your own
44:31
benchmarks and then you can figure out what works for you.
44:36
Yeah, it's a methodology for you to create.
44:38
Their own benchmark, really, right for the things that you're
44:43
interested in. And clearly different people are
44:47
building different products and are thinking of different ways
44:50
to leverage AI, right? But this story is always
44:54
similar. Where?
44:57
So in our case, we want to help people do their own analysis of
45:01
data and explore their data more on their own.
45:05
And. So so our use cases are around
45:08
generating queries and making sense of of data.
45:12
But in general you always have like OK my product as some
45:14
context. In our case that's that might be
45:16
the database of the user and the database schema of the user.
45:21
And the user has a question. And if we take this user
45:24
question what they want along with the context that's in our
45:27
app, we can output something that is useful in the context of
45:31
that app. And I think for every screen and
45:36
every product, there's potentially a place where AI can
45:40
can be leveraged. And she's like the questions
45:43
like, how do you evaluate how well it's doing?
45:45
You definitely don't want to be shipping a product with an
45:47
untested AI. You don't know whether it's
45:49
gonna hallucinate things or make shit up or just have biassed to
45:54
or or like to upgrade like this, upgrade to the newest version of
45:57
GPT. Maybe it performs better at
46:00
certain things but worse at others.
46:01
You need to handle. If you're going to ship product,
46:04
you need a handle on every every API that you use.
46:08
And this one API that we started using now that's in the LM just
46:12
as no structure. So like that maybe that's the
46:15
way to put some structure and evaluation on top of it.
46:19
Yeah. And it's funny to another thing
46:21
from the product perspective, which people pointed out in this
46:25
report that we did, it's and I'm just bringing it up because I've
46:28
literally been. My head has been buried in this
46:31
data for the last three months and so I feel like I've come to
46:37
some conclusions. They're probably completely
46:39
biassed, but you know, just reading the responses because I
46:42
don't know anything about doing reports, I did not get that
46:45
consultant training. I never went to McKinsey for my
46:49
report writing skills and I left every single answer or the
46:55
questions were all with open freeform text box because I
46:59
didn't want to assume. Anything.
47:01
Since it's so new I didn't want to push anyone down any path.
47:05
That made it really hard to try and write a report from.
47:09
And because you have to just go through all this data, there's
47:12
no like clear. I can't create graphs or
47:14
anything. There's not like Ohh people, 10%
47:17
of the people chose this. And so getting back a little bit
47:21
of a tangent there, but getting back to this, one thing that
47:26
people were noting is that, yeah, I can add AI to my
47:30
product, but at the end of the day is this AI feature.
47:35
Sufficiently going to add more value to the end user so that we
47:40
can charge more and cover the cost of how much it costs us to
47:46
implement this AI feature. Yes.
47:49
It's an uncharted territory in a lot of ways, right.
47:52
Like, so people are using your product in a certain way today.
47:56
They're kind of trained to do that.
47:57
Then you wanna add a new, I don't know, set of tools and
48:00
shortcuts that enable them to to do much, much more work, maybe
48:04
work they wouldn't do otherwise or are things that they would do
48:09
somewhere else. And then you're assuming that
48:12
this is going to be useful and how do you measure whether it's
48:15
going to be? I mean it's it's a tough one.
48:17
I think the only way to know is to.
48:20
Add a feature flag in your product bake it in and then do
48:23
some user interviews after to launch it and then data private
48:27
beta and then look at and mix it user feedback from humans like
48:32
sit down with them and be like oh so you've used this feature
48:35
quite a bit do you like it And then there's like emerging usage
48:38
pattern too that you can't predict like to your point
48:41
earlier we're talking about we launched that GPT are they
48:44
launched ChatGPT didn't no what to use it for and and all these
48:48
use cases are popping out. Of in the wild as people are
48:53
trying to figure out how to use this thing and they find
48:56
intricate ways to get value that you and I can never think about.
49:00
Like a dentist using chat, GPD or neural surgeon asking
49:04
questions that we could never come up with.
49:07
And but but yeah, so so there. Do you do user research.
49:12
You can also log a bunch of data.
49:13
So for us, like we're gonna be. Logging like OK, what's the user
49:17
prompt? What's SQL generated?
49:20
I know you had a little thumbs up and thumbs down and try to
49:23
get some call it some more quantitative metrics on that and
49:27
you can sit down and interview people and be like, hey, is this
49:31
AI assist feature that we've built, is that useful to you
49:35
more your normal kind of user research type approach to to it.
49:39
Yeah. And I think the interesting
49:41
piece there is like you from your perspective, alright, now
49:44
you see that people actually like it.
49:47
How much more are they willing to pay for it, right?
49:49
Because then if it's not more than your API calls are costing,
49:54
it is not, you're not like actually generating an ROI on
49:57
that. Yeah, so, so there in terms like
50:00
the perceived value of the feature and that's gonna be
50:04
really specific to every product and whatever you wanna gate that
50:08
our charge people more. For it than exactly.
50:11
What they see out of it is it's going to tough to evaluate, but
50:14
yeah, you probably need to put in some sort of like open beta,
50:18
get some feedback, try to figure out before you even think about
50:21
pricing and packaging or ROI. But the intuition is that, yeah,
50:25
I think everyone's intuition is like there should be ROI.
50:28
Here. And and then we can't afford to
50:32
miss that boat as a pretty big feeling in the industry.
50:36
Like I would say, like every PM was not thinking about this
50:40
right now is. Probably gonna potentially
50:43
missing a boat and and it's it would be unreasonable to not
50:47
think about what how LMS can augment your products today.
50:50
Yeah, yeah, yeah, yeah, 100%. And so a friend of mine told me
50:55
when I was talking to him about this very thing, he was like,
50:58
yeah, I mean, at the end of the day, you're just going to have
51:01
to tally up the. Cost of using LMS to cogs.
51:09
So your cost of goods sold basically.
51:11
And so if if it works out, if the math works out then it works
51:16
out, but if it doesn't then you've got to figure out how to
51:18
make it work but in like. Sheeps though, right?
51:21
Like if you look at, I mean, maybe if you use like GT4 with
51:25
the longest prompt possible or if you run your own
51:27
infrastructure exists, it's expensive.
51:29
But like this stuff is like pennies a a prompt, right?
51:34
So unless you're doing like crazy.
51:36
Chains or recursive prompting. Are very very large input and
51:42
output. I think it's it's like you you
51:46
could never run and that put me on this but it would be hard to
51:50
run a bill on GPD 535 turbo right Like it's like pennies per
51:55
call. So unless you're a product is
51:58
doing if there's a user interaction the user clicks a
52:01
button gets a prompt at a penny or call or sub penny or call,
52:06
it's pretty it seems like commodity.
52:09
Yeah, yeah, it's hard to. Run up a gigantic bill?
52:13
I guess, but. I mean, you know, I wouldn't put
52:16
it past some people in teams to make that from Fort.
52:20
Ohh yeah. People will build fat bills of
52:23
open air is gonna be making some money off that API, that's for
52:25
sure. Yep, Yep.
52:27
So there is one kind of elephant in the room that I want to get
52:31
to, which I've heard other people talk about and seems like
52:34
you're very bullish on prompts. But I do know people that talk
52:38
about how they feel like prompts are an artefact and they
52:43
hopefully are not the end state on how we interact with.
52:47
I cannot think of another way that we would do it, but it
52:51
feels like. What we've been talking about
52:54
English or like speaking or language in general, a written
53:00
language is a lot more difficult to.
53:05
It just has a lot of downsides that coding and the
53:10
deterministic ways that we interact with computers.
53:13
Don't have. That's not to say that we're not
53:16
necessarily going to go back to everything being coding after X
53:20
amount of time because we realise that this is so hard but
53:23
when it comes to prompts. And interacting with large
53:27
language models through prompts. Is that the end state?
53:30
Have you thought about what potentially is next?
53:33
I mean the way I think about this question maybe is like
53:36
between like specialised model versus generalised model and
53:40
then like these LM's are they're proving the power of these
53:44
generalised model and the power of unsupervised turned
53:48
unsupervised training. Using text and language.
53:53
And that's a new API that we're trying to tame and control with
53:58
prompts and prompt engineering. Look at more traditional machine
54:03
learning type model. You have these very, very
54:05
specific use cases, right? You prepare a data set that says
54:09
like is this. I alright, classic use case.
54:13
You take the examples of like given the features of a flower
54:17
or tell me what kind of flower it is.
54:19
And you you train this model on this specific use case of like
54:23
given these like 6 input parameters, predict which kind
54:27
of flower that is. And in some ways that's where we
54:30
came from from these like they call it specialised models.
54:33
So these specialised models I think are still extremely
54:37
powerful and whether they're neural Nets, trees, forest,
54:40
linear regression, like whatever.
54:42
They are there. They're useful, and they are.
54:47
You know, they have kind of their very clear API, right?
54:51
Like when you think about those, they have very clear API.
54:54
We just have this new world of like generalised AI.
54:58
Not AG about like you know generalised AI that are just
55:00
good with language. It's a new beast for us to to
55:04
tame and then clearly like the. You know, Prompted Jang.
55:07
Is is. The way to do that though, I
55:10
don't think people realise just how much you can do with prompts
55:13
right? Like so.
55:14
One thing I discovered is you can ask for structured output.
55:18
You can say like hey I'm about to ask you a question to
55:21
generate SQL, but I would like for your output to be a JSON
55:24
BLOB with one and one key that this is a SQL query.
55:28
Another key that is your confidence level from zero to 1
55:31
on whether this SQL answer is the question.
55:33
Another key is some text feedback as to hence to improve
55:37
my prompt to get a better answer from you.
55:41
That need for ask for clarification.
55:43
Type input and you can ask this AI through a prompt.
55:49
To to provide a structure structure and its answer and
55:52
that's been working very well for me and GPT 4 right with few
55:56
shots examples of like when I ask you a question like this, I
55:59
want an answer like this formatted like JSON with those
56:02
keys. Here's 3 examples.
56:04
Now here's the question you tell me provide this similarly format
56:10
JSON BLOB and so you can ask for structure and the answer and
56:15
which is great. Then how do you structure the
56:18
question? How do you best provide the
56:20
context to get. On the AI to help with your use
56:23
case, that's I guess the other that's a big part of that prompt
56:28
engineering too. But the interface for LLM is
56:31
going to be language is just gotta be it.
56:34
Yeah. Then then there's a question.
56:36
And maybe a question going back to you is like, which real life
56:40
use cases? Percentage of real life use
56:43
cases are gonna be solved by specialised model versus
56:45
generalised model. Prince perspective on that
56:48
change over the past 12 months from like.
56:52
It's all gonna be specialised and I was like, it's all gonna
56:54
be generalised, like there's probably a medium there
56:56
somewhere. Yeah, exactly.
56:58
That's the funny part, right? Like it feels like every day.
57:01
Then we move forward. We're going into this, yeah,
57:05
it's generalised, but they're smaller models and so and we
57:09
are, we went to the generalise, but then we're saying yeah, but
57:12
now it's probably going to be a lot of generalised models.
57:14
And to your point earlier about the fine tuning, if you can fine
57:19
tune a large language model that has a certain ability, but then
57:24
you can add a little boost on top for your specific use case.
57:29
And then you have other use cases that you can fine tune
57:31
for. It's probably a lot of those
57:33
fine-tuned models as opposed to one big one that you're trying
57:37
to fine tune with everything, right?
57:39
Right. So so yeah, I talked to some
57:42
some folks around this specific topic because I was interested
57:45
in. Well, instead of like doing a
57:48
lot of prompt engineering, I'll get a little bit deeper into the
57:52
the text to sequel challenge that we're trying to solve.
57:55
But thanks to SQL, you can have a database that has 10s of
57:58
thousands of tables in it, right?
58:00
And people might ask a question and generate question on data.
58:04
Then the challenge becomes, OK, what are there's a lot of
58:07
private contexts here, which is like all the data schemas inside
58:11
the user database and clearly that doesn't fit in a prompt,
58:15
right? I cannot say like here's 10,000
58:16
tables. The schema of 10,000 tables with
58:19
their simple data, usage, statistics, descriptions and
58:23
comments like 10,000 tables. Now can you generate SQL that
58:26
answer the simple question what is my net sales this year?
58:30
So then with vector database we try to do document retrieval to
58:35
best construct the prompt. To best construct the prompt
58:39
that is likely to have the context for the AI to provide
58:44
the answer. Then the other approach like how
58:47
can we just fine tune? Just train the model I I create
58:51
a corpus of 10,000 tables and I'll throw my DBT pipelines, my
58:55
airflow Dags, my my data notion pages and whatever else into
59:00
this corpus. I'll find you in a model that
59:03
will have the full context of all my data catalogue and more
59:07
right data catalogue data pipelines.
59:11
Then as I OK can I do that? I spoke to someone at Mosaic
59:16
they they work on something called GPT for All, which is
59:18
like one of the open source model probably great person to
59:22
talk to and shared the contact info.
59:26
But they were saying that. If you try to take a model
59:30
that's fully trained, so you bring the model topology and the
59:34
weights, and then you throw your own corpus at it, you're like,
59:38
Oh well, it should learn this stuff on top that I'm trying to
59:41
teach it. But what this person was talking
59:45
about is the sequencing of of information is really important.
59:49
So if you train a model with a bunch of information, so I'd
59:51
open a I. They have this sequence of
59:54
training and the SQL teaching portion of it, a Co teaching and
59:58
then multiple language teaching is fed in a very specific
1:00:01
sequence. So if you take that model and
1:00:04
then you train it all the way and then you like, OK now here's
1:00:08
a bunch of like random information on it can really
1:00:12
confuse the model. So then.
1:00:14
That means you would have to go back to fitting your your
1:00:18
training in the right sequence and then train from scratch.
1:00:22
And then that's hundreds of thousands of dollars of GPU
1:00:24
cost. And and otherwise you deal with
1:00:28
these issues of like kind of memory scrambling.
1:00:30
I forgot what what they're called officially but like if
1:00:34
you try to take a model is fully trained and like areas.
1:00:36
By the way here's a bunch of extra information that is
1:00:38
relevant to me and I forget things and I kind of just like
1:00:41
mess up the the weights and the neural Nets so.
1:00:45
So there's we're looking for some improvements in those areas
1:00:48
too. Can you fully train a model and
1:00:50
then and then specialise it, tune it?
1:00:53
Later in the process without confusing it up.
1:00:57
Hmm, yeah. I think that there's so many
1:01:00
open questions on that, like which is better and for what use
1:01:03
cases is it better for and so it makes complete.
1:01:06
Here's the thing too, Like for me, if I'm like, OK, well, I
1:01:09
should build a fine tune, a model for each one of my
1:01:13
customer at present. Yeah, played.
1:01:15
You want isolation too. You don't want for it.
1:01:17
Yeah, one customer to be trained.
1:01:18
Through compliance, there you go.
1:01:20
That's how we're getting to compliance and privacy.
1:01:24
And then that means I need to fine tune a model.
1:01:28
For each one of my customers and serve it serve a model.
1:01:32
A different model for each one of my customers.
1:01:35
And the air gapped way. Is rough right?
1:01:38
Like if you wanna do that today, I think you can do fine tuning,
1:01:43
ask it and open AI. But but fine tuning the way it
1:01:46
works might not be the way that you want it to work and it's not
1:01:49
quite there yet, right? So maybe the year or two from
1:01:52
now then it's it becomes really easy to.
1:01:56
Fork models and fine-tuned models in different ways.
1:01:59
Like today. It just makes a lot of sense to
1:02:01
do prompt engineering. If you're a small and mighty
1:02:02
organisation moving fast, you probably just want to spend some
1:02:06
cycles doing some prompt engineering and then hitting the
1:02:10
the Open AI API. The spider.
1:02:13
The most reasonable approach if you start getting into training
1:02:16
your own fine tuning your own models and you're entering
1:02:20
potentially a world of of trouble.
1:02:23
Yeah well the trouble the world of like increasing investment
1:02:26
like right to get to value expected to do a lot more and.
1:02:30
Yeah, totally. It is.
1:02:31
It's the classic engineers dilemma, right?
1:02:35
Like, we want to sink our teeth into things because it's like,
1:02:38
oh, that's fun. Let's add a little more
1:02:39
complexity onto it. And really, you could have done
1:02:44
everything from a much easier standpoint and so.
1:02:48
Yeah, and then the. Thing that I will.
1:02:50
Write to you right. Like I said, there's a timing
1:02:52
issue like I do. I wanna try to build an iPhone
1:02:55
before LCD screen, Touch screens are good enough and chips are
1:02:59
signature enough. So there's a a time like if you
1:03:02
try to do the right thing too early, it's very expensive and
1:03:07
costly. Where if you want to invest into
1:03:10
fine tuning, maybe the best time to do it is when you know you
1:03:14
you keep monitoring the the advances in that domain and once
1:03:18
the right API, the right cost. Comes out, that's when you start
1:03:21
investing in that. But I think we're entering,
1:03:23
we're in that era. It might be a short era where
1:03:26
prompt engineering and vector databases is the name of the
1:03:31
game right now, if you want to get value out of LMS, is is
1:03:35
really where you should spend your cycles from my own personal
1:03:38
perspective. Right, fine tuning.
1:03:40
Think we might enter the era fine tuning and a month or in
1:03:44
six months? In 12 months, who knows?
1:03:45
Yeah, it's hard to gauge that, right?
1:03:47
I know people are excited about the what is it, the Q Lauras
1:03:50
that are showing promise, But it is true like, hey, build for
1:03:55
what is here today and I love what you said, which is.
1:04:00
Such great wisdom. It's like look at where you are
1:04:02
and take a cold hard look in the mirror and say if you are this
1:04:08
small and scrappy startup and you've got to move fast, go with
1:04:12
what is going to be the easiest in the lowest hanging fruit.
1:04:15
You don't need to overcomplicate things just because you read
1:04:19
some research paper and you saw. I made it might be more fun too,
1:04:24
but to me, I don't know. I'm super pragmatic too, so
1:04:27
entering the whole like, I'm gonna figure out how to load up
1:04:30
a bunch of. Weights and get access to a
1:04:33
bunch of GPUs and rake up these bills to try to fine tune my
1:04:37
stuff which I might just throw away after the first iteration
1:04:41
and start over at seems prohibitive and and this phase
1:04:45
it's much more like the pragmatic approach of like I'm
1:04:47
trying to fix myself a meal. Like what's in the fridge.
1:04:50
I'm not going to go to the store.
1:04:51
I'm not going to go to to specialise grocery shop or are
1:04:56
there stuff online. I'm just really trying to cook
1:04:59
something what's what's in my kitchen.
1:05:02
Yeah, for sure. I mean, if it's possible, right?
1:05:04
Because I know there are people that are sitting at companies
1:05:08
and whether it's because of GDPR issues or because of company
1:05:13
policy or regulatory issues, they just can't say, all right,
1:05:17
now I'm going to hit Open AI API and so they got to figure out
1:05:20
workarounds. But I will say this, you
1:05:22
mentioned something before that I thought was hilarious because
1:05:25
I've been meaning to say it for a while now and it's what did
1:05:29
you call it like random scrambling or something?
1:05:32
Terms I don't know yet. Yeah, I don't know what the
1:05:36
terms are, but I gotta give it to the researchers because every
1:05:39
time I hear a new term that comes out of the LM's or just
1:05:44
the machine learning researcher community, I'm like, man, that
1:05:48
is such a good term. First of all, we've got
1:05:50
hallucinating, right? That is epic.
1:05:53
And I created a shirt. I'll have to show you sometime.
1:05:55
It says I hallucinate more than ChatGPT and.
1:06:01
I thought you would like that one.
1:06:02
Yeah, I thought you would. But then the other term that I
1:06:05
absolutely love, and I'm trying to figure out if there's a meme
1:06:08
or a shirt to grate around it, is the whole idea of
1:06:12
catastrophic forgetting. And it's like, Oh my God, can
1:06:16
you make something sound so epic?
1:06:18
I mean, catastrophic forgetting. Give me a break.
1:06:22
Yeah, that's right. That's the term when you teach
1:06:24
new things and it forget like so much.
1:06:26
Should I forget something that's more important, like you would
1:06:30
exact new language on top and it would.
1:06:32
Get Spanish or like, scramble it with some other language.
1:06:35
Forever. Yeah, yeah, exactly.
1:06:37
Can't recall it anymore because of the sequence of event or
1:06:41
something like that. But you know, another thing that
1:06:43
I think that a term I've been using to describe LM is like
1:06:46
amnesiac. To us it's related to get this
1:06:49
through before getting but they these these models and and I
1:06:53
thought people I figured out early on when they came out and
1:06:56
like how did these things work but they're immutable models
1:06:59
right They they they have no they're trained ones.
1:07:02
Are you taking you take like GTA5 Turbo right like this train
1:07:06
once it is completely immutable. It is trained as of like 2021
1:07:11
the fall of 2021 it knows nothing about what happened
1:07:14
after and then that is like well, but it recalls like Avis
1:07:18
I'm having a conversation. With it and it remembers what we
1:07:21
were talking about because tell me more about that.
1:07:23
And it knows. That's just because at every
1:07:25
request you pass the context, the jabot receives the context
1:07:29
of the current session, but it has no recall if you go past
1:07:35
that prompt window too. So if you have a long enough
1:07:37
conversation and you ask, hey, what did I first talk about when
1:07:40
we started this session and it's outside the token window, it's
1:07:45
like, I don't know. There's no way for me to know.
1:07:48
Of course, like nowadays. Looking to train.
1:07:51
They're going to train and fine tune their models with the chat
1:07:54
sessions that we're having today outside the API, at least at L
1:07:59
and AI. They're doing that, but.
1:08:02
These models are amnesiac. They they have no recall, no
1:08:06
memories. They have to bolt that on top
1:08:08
through context and prompting. Yeah, that's actually that is
1:08:12
something that when we had Matteo on here from Databricks,
1:08:17
he was talking about how much of A crapshoot it is.
1:08:19
When you do decide, hey, I'm going to go and train my own
1:08:22
model because you are training for 40 days or 60 days or
1:08:28
however long you're spending a tonne of cash and then you're
1:08:32
kind of like left. With whatever comes out, you
1:08:35
can't just go and say, alright, now we've got this basic
1:08:39
addition and we're gonna make it like 100% better.
1:08:42
You have a a model that comes out and there's not a lot that
1:08:47
you can do if it's a shitty model.
1:08:49
I think you can load it back in memory.
1:08:51
So right you can extract the topology of the network and then
1:08:54
the waits and then you can. Fine tune and I guess is what
1:08:58
they call it. You can feed it more corpus but
1:09:01
then the I think the sequencing matters and there is this like
1:09:04
issues with catastrophic forgetting and just scrambling,
1:09:07
catastrophic scrambling, memory scrambling.
1:09:10
So it will like it's likely to really weigh the stuff it I mean
1:09:17
like you and I right like what we last heard about the last
1:09:20
input. We recency bias, yeah.
1:09:22
See bias. Say we'll have that in the
1:09:23
training people are looking to. Don't kind of fix that because
1:09:28
otherwise it's true that if that team I think this point like you
1:09:31
have to go back and retrying the whole thing on on a cluster of
1:09:34
GPU's for for six days or for 40 days then that's that's reminds
1:09:40
me the world of like very slow data pipelines on map readers.
1:09:44
It's like fire and like you run the job and then it's going to
1:09:48
take 24 hours to run and you just hope it's not gonna fail
1:09:51
towards the end it does like. The cognitive weight of that
1:09:55
right you You start many of these jobs because they take
1:09:57
forever. Then you forgot what you had
1:10:00
done. Like if you're training 66
1:10:02
version of model at once, then you forgot like why you trained
1:10:06
model one in the first place by the time it's ready.
1:10:11
Yeah. So yeah, for me, I don't like to
1:10:12
iterate on these like super slut.
1:10:15
Like anything that takes a lot of compute and time is hard at
1:10:18
work on, man. It's just like, it's no fun.
1:10:22
What problem is said? You can like exactly get direct
1:10:26
feedback. Yep, and you can use promise to
1:10:31
get your your prompt suite. I love that term and really go
1:10:35
with this. And the TD or it's what is it
1:10:39
the it's not TD you were calling it the PDI.
1:10:42
Ohh test driven development. I guess it's like Trump cases,
1:10:45
it could be prompt driven. I'm driven to driven.
1:10:48
I like. Or like test prompt development,
1:10:51
I don't know, but to me the idea it's gonna reason why I come out
1:10:56
to talk about these things to share my experience working with
1:10:59
these things and share some things out long I learned along
1:11:02
the way. I'm not sure if it's to me
1:11:04
promises less about the tool the Python.
1:11:08
Little toolkit and CLI but more the idea of like how do we bring
1:11:12
more and more deterministic approach to interacting with
1:11:18
these MLM's. And that's to me the blog post
1:11:21
is really if people are interested in the topic, go
1:11:23
read. The blog post is much more
1:11:24
structured than I can be in free form like this but it's maybe
1:11:27
less entertaining but more structured.
1:11:29
And then like this is the the big that's the big ideas and the
1:11:34
blog post and then prompt amazes just like at a reference
1:11:37
implementation of how to do it. And if the toolkit works for
1:11:40
you, use it. But if you want to write your
1:11:42
own test suite and reporting suite, you can do that as well.
1:11:47
Excellent, man. Well, it's been a blast.
1:11:50
Absolute blast talking to you. I knew it was gonna be fun and
1:11:54
I'm happy that we did this. I'm super excited for what you
1:11:57
created with promise. And it's true, like you're doing
1:12:00
it so that you can just share your learnings and you recognise
1:12:02
that this is probably valuable. So get it out there and thank
1:12:06
you for doing that. Yeah, it's it's great to share
1:12:09
the enables these scans of conversation to do it right.
1:12:11
So it's like I was able to write a blog post, create a little
1:12:14
toolkit and then that gives me an opportunity to connect with.
1:12:19
People and and talk about this this topic from from experience
1:12:22
get input from others and compare notes So it's been it's
1:12:26
been super exciting too. I gotta find a community of like
1:12:29
the community of people that are building if you have any input
1:12:32
on not like people trying to wiggle and jam AI into the
1:12:36
product they're building. I don't know if those
1:12:39
communities exist but if anyone in a in a show notes have been
1:12:42
put on that I'm looking to get closer to other people that are
1:12:46
trying to figure out how to bring the power of AI and.
1:12:50
Problems inside the product you're building.
1:12:52
Yeah. Like the AI product community.
1:12:55
Yeah. And yeah, yeah, I love it.
1:12:58
I mean. By the time we released this
1:13:01
episode, your talk from the LM and production conference that
1:13:06
we're about to have next week. So this will date when we're
1:13:10
actually having this conversation.
1:13:11
That'll be out too. So if anybody wants more of you,
1:13:15
Maxine, they can go find that, we'll link to that too in the
1:13:19
show notes. So it has been awesome, man.
1:13:21
I really appreciate you coming on me.
1:13:23
Love to connect, Yeah. I don't know what we're going to
1:13:26
talk about next year. Last time we talked about data
1:13:28
modelling and how. You.
1:13:31
Say it's like cross NL and data engineering or analytics in
1:13:35
general. Now we're talking about pump
1:13:37
engineering, so who knows what it's going to be about next
1:13:40
time. Yeah, exactly.
1:13:42
And that wasn't that long ago. That was literally like nine
1:13:45
months ago or yeah, a year probably.
1:13:47
So it's funny how things move.