Latest / Agentic Conversations (formally mlops.community) / The Real E2E RAG Stack // Sam Bean, Rewind AI // #217

Transcript

The Real E2E RAG Stack // Sam Bean, Rewind AI // #217 · Agentic Conversations (formally mlops.community)

0:00
I'm Sam Bean. I work at Rewind AI and Applied
0:05
AI and take my coffee black with a lot of caffeine in it.
0:11
What is up? Good people of the Internet,
0:14
welcome back to the ML OPS Community Podcast.
0:16
I am your host Dimitri Ose and today what a conversation with
0:22
Sam. Wow, coming away from this
0:25
having goosebumps. I don't say this all the time,
0:28
but when I do say it, hopefully you take me seriously.
0:32
This is a conversation that I would recommend to everyone.
0:37
Sam has had experience dealing with LM's since before they were
0:42
cool and building systems insearch and with LMS since
0:49
before the whole chat debt craze.
0:51
And he talked about some of his huge learnings, this collection
0:55
of learnings that he's had. And oftentimes what he mentioned
0:59
is there are simple things that you can do to help you progress
1:06
your LLM systems. He worked at U Com, he's now
1:10
working at Rewind. We get into all of that.
1:13
But really, I think the key mode of the key factor that I'm going
1:19
to take away from this is when he talked about overcomplicating
1:24
your LLM systems for no good reason and how we can all fall
1:30
into that trap for some reason or another.
1:33
And then you don't recognise what metrics are crucial and you
1:39
don't know how to read your metrics if your system is too
1:42
complicated from the get go. So he walked us through how to
1:47
build from 1:00 piece of the puzzle to the next and make sure
1:54
that you're able to get those metrics and understand and
1:58
interpret those metrics very well before you decide to add
2:04
any new pieces to your system. I think this is a little
2:08
different than the talk that I've been hearing.
2:13
Quite a bit of it. It's very easy to say, OK, we
2:16
need everything. Now we need to make this the
2:20
most robust system possible because these LM could go off
2:24
the rails and we got to make sure that we are following the
2:29
most up to date system design that I've seen on LinkedIn or on
2:35
Twitter. And Sam is not about that at
2:38
all. He is very pragmatic in the way
2:42
that he does things and he is down to the metrics, making sure
2:46
that before you do anything, exhaust the simple stuff,
2:53
exhaust your ability to get very far with what you have and
2:59
really crush the metrics that you have.
3:01
And once you can't go any further with those metrics, then
3:04
bring on more. Seems simple.
3:08
For some reason it's not so easy to do in practise.
3:13
We like to over engineer things and that's just I think in our
3:17
nature. So enjoy this conversation with
3:20
Sam and if you liked it, as always would mean the world to
3:25
me. If you tell one friend or leave
3:28
us a review, leave us some stars.
3:31
We've got almost 120 stars now on Spotify.
3:35
That's incredible to see. I am appreciative of every
3:41
single one of you that are listening and I thought I would
3:43
say it before we get into this conversation. only one place
3:54
that we can start, and that is with the story behind your
3:59
competitive pinball playing. Ohh, yes, of course.
4:04
You want to start there? So yeah.
4:08
So people don't know I'm a competitive pinball player,
4:10
actually in my weekly league tonight I'm gonna go competing.
4:13
I'm playing for a trophy. I'd a friend of mine, Jack, who
4:19
I went to computer science university with at University of
4:23
Michigan, and out of out of school he went and he went and
4:27
programmed pinball machines. He wrote game code for Stern
4:31
Pinball, which is one of the manufacturing companies out of
4:34
Chicago. And he wrote game code for, I
4:37
think the likes of Mustang World Poker Tour and some stuff on the
4:41
Game of Thrones. And you know, he was, you know,
4:45
playing coding. He was like living and breathing
4:48
this stuff. And it just kind of like got
4:50
into our friend group was like, hey, this is something we can
4:52
kind of go do. And I'm a competitor.
4:55
And so it went from fun thing to do with my friends to
4:58
competitive thing that I can find people to, you know, go
5:02
play against. And if I ever find something
5:04
like that, I I can't help myself.
5:07
What does it mean to code Pinball?
5:10
Because I thought it was just the ball goes and bounces off of
5:14
shit. Is it that you're deciding how
5:17
many points you get when it bounces off of things?
5:20
Yeah. So I think it's it's it's easy
5:22
to think of it as kind of like a strips down traditional game,
5:27
similar game mechanics. Like it's mostly like a state
5:30
management system that's like, hey this is the current state,
5:34
this is the next input switch. The ball is mostly rolling over
5:39
switches or heading targets and that like creates more inputs
5:43
for the basically state machine to figure out which state to go
5:47
to next. The only interesting thing is
5:49
that the inputs are physical. It's a ball rolling over a
5:53
switch, or it's hitting a target or a pop bumper or something
5:58
like that versus virtual inputs or inputs from like a PS5
6:04
controller. The inputs are coming from a
6:07
ball rolling around on a wooden field.
6:08
It's it's interesting, but past that, the input source, it's
6:12
mostly like your traditional game engine.
6:17
So where do you find these pinball arcades?
6:20
Or did you buy your own? Or did your buddy vision?
6:24
Yeah, so I've got a few that I practise on.
6:26
Those are the ones that if you want to have a bad time playing
6:29
pinball and come play in my basement throughout the Dojo, My
6:33
machines are set up to make you a better pinball player.
6:36
And they are not so much set up to have fun playing.
6:39
So they're mostly set up to not have fun playing.
6:43
They're they're extremely difficult.
6:44
They're very fast. You can lose the ball very
6:48
easily because of the way that I've removed certain physical
6:51
objects that usually would impede the ball leaving the
6:54
playfield. Those posts are all gone.
6:57
It's brutal, but if you wanna get a pen ball, I got I got the
7:03
collection for you. Yep, that is it.
7:06
This is a really cool hobby, and this is a first that I'm hearing
7:09
anybody on this podcast talk about, so I like digging into
7:12
it. I had to when I saw that that
7:14
was possibility there and so you've had quite the experiences
7:22
dealing with ML OPS and particularly LLM's.
7:26
And I want to go through your journey a bit because you've
7:30
worked at u.com, you are currently working at rewind, and
7:35
I think there's something interesting in and about this
7:38
space of dealing with search and dealing with large language
7:43
models that it feels like interests you and maybe you can
7:47
breakdown what gets you excited and why.
7:51
How did you decide to start working on these MLM's at such a
7:57
time where it was like before the hype almost?
8:00
Yeah, yeah. So.
8:02
My journey with ML OPS I think really starts at really starts
8:07
at stock X doing things like simple drift detection,
8:11
retraining routines based on performance degradation like
8:16
you're like standard initial MVP.
8:20
I'm all up setup. I think that going from like a
8:25
forecasting problem domain, which is a pretty simple just
8:30
you know each point has a feature set and then you're
8:35
mostly just creating cross features or better feature
8:38
representations and then you're putting that through a big, you
8:41
know decision tree. I think the complexity with
8:46
observing these systems increases a lot when you move to
8:50
search. I think that there's a lot of
8:52
different failure modes for search.
8:55
And also search is one of those really, really special problems
8:58
where you're never actually going.
9:01
There's going to be questions that there are no right answers
9:04
to in the world, right? That's impossible to get right
9:07
because it's either someone's asking something that doesn't
9:09
exist, it doesn't make sense, or a variety of other reasons.
9:14
And then there's questions where there could be millions of right
9:18
answers. Any of a million web pages could
9:20
be served to this user and any of them will solve their query.
9:23
And then there's a tonne that is in the middle.
9:26
Most of it's in the middle. And so search is one of those
9:28
problems where you're never really quite 100% sure if you're
9:33
doing it right without the behavioural data.
9:36
And behavioural data is so expensive to go out and get via
9:40
AB testing or, you know, user surveys or however you're going
9:45
to go. Get real people's feedback on
9:48
your software that is just so expensive to get.
9:52
And so it feels like the machine learning operations just
9:58
paradigm and that tool set, the importance is amplified in
10:02
search and in looms, I think because of the ambiguity that is
10:09
just inherent to those systems. I think that they can handle
10:12
more ambiguous problems and inputs and can do more for the
10:15
user. But because of that ambiguity
10:18
and because of those capabilities, I think they're
10:20
more challenging to monitor and to create like really robust
10:24
life cycles around those systems.
10:27
So it made sense to like, take those skills and level them up
10:29
in the search. Spaceandyou.com was like the
10:31
perfect place to do that. And so it went from being at
10:36
stock X, working on more, as you mentioned, more traditional mill
10:41
we could say. And that's not without its
10:44
problems. That's not without its
10:45
difficulties, right. And then you jump into search
10:48
and there's so many more neurons as you're mentioning.
10:52
Just thinking about how many right answers there can be for
10:56
certain questions or how there could be no answer for certain
11:00
questions, and how you make sure that when you're serving up
11:04
these results they're hitting on the most relevant topics is such
11:10
a fascinating problem and it almost is like.
11:13
I think one thing for me with search, which is really hard to
11:17
think about, is like most of the time when we think about search,
11:24
nine times out of 10 it's like, ohh, but that's a solved
11:27
problem, right? That was like the 90s with
11:30
Google. Didn't they figure that out
11:32
already? You know the that's all done.
11:35
How is it still a difficult problem?
11:37
And I imagine you were in the weeds figuring out day in day
11:41
out what you're doing with search.
11:43
But then also adding LM's to the mix makes it even more fun.
11:48
It does. I think it actually, yeah.
11:52
I think it actually makes the problem simpler and harder at
11:56
the same time. It's one of those kind of you
11:59
have another. The thing about search is there
12:02
isn't like a model, you know, like I've got my search model.
12:05
Search systems are usually comprised of multiple models,
12:07
right? You have an intent detection
12:09
model, a query rewriting model, You have your actual like
12:13
retrieval models. If you're using semantic search,
12:15
you have ranking models. You may have filtering models to
12:18
remove irrelevant result, like you're talking about 5 to 10
12:23
models to run a search system and then you add on top of that
12:27
and LM and those are have their nuances and subtleties around
12:33
around training and deploying. But what they kind of do is they
12:37
kind of give you a lens to how the user is going to use these
12:43
search results. And so while it might be really,
12:47
really hard to figure out the right or wrong answer for a
12:50
search query, it's actually significantly easier to figure
12:55
out the right or wrong answer for just a question answer pair
12:57
out of your traditional Squad V2 hotpot.
13:01
Because they come with the right answer.
13:03
And there is a you can guess this right, you can guess it
13:06
wrong, there might be different versions of right or wrong, and
13:09
we can totally get into all of that.
13:11
But if you have this kind of end to end question answering set up
13:16
at the very end of your pipeline and you have metrics for that,
13:19
like how how many of these questions can I actually answer
13:22
correctly with by Reg setup. Then you can actually start to
13:26
work your way backwards towards the more dot dot dot sub models
13:33
or subsystems within your search.
13:35
And then you can figure out like, hey, if I tweak this a
13:37
little bit, can I answer more? Does my LLM actually answer more
13:41
questions right or wrong? That's a very easy thing to
13:44
understand as a human being and to actually track your progress.
13:47
And so yes, you have this new model, and yes, it has its own
13:51
kind of special latency requirements, and it has
13:54
streaming tokens, which is a different kind of paradigm than
13:57
maybe some request response servers were equipped to handle.
14:02
But at the same time, again, there's this lens that's exposed
14:06
that kind of gives you this insight into the kind of the
14:10
motivation of your search system, which I think is really
14:12
powerful. And for people who aren't trying
14:16
to tie those dots together, I think you're really missing out
14:20
on an ability to give yourself a really good vantage point to
14:23
kind of see get some some customer, some customer sympathy
14:30
because you're actually like seeing the end result of what
14:33
your search system is going to do.
14:35
Will this actually help a person if it can help an answer
14:39
questions better? You would expect it to be able
14:41
to help a person answer questions better.
14:42
And that's a really like being able to make progress on that is
14:45
really rewarding and it kind of gives you a really good
14:48
grounding to borrow that term from MLMS of how you're going to
14:53
measure your progress. Very clean, very clear, easy to
14:58
communicate. Yeah, that that I found to work
15:02
really, really well. It's like if this information
15:06
that I'm gathering is helping me understand my question, nine
15:11
times out of 10, it's going to help the LM that you're feeding
15:14
it to. Yeah, that that's a great
15:16
insight. So the other thing that you've
15:19
mentioned, and I don't want to skip over it too much before
15:22
moving on, is how expensive it is to get human feedback and get
15:27
like human. Almost like evaluation on this
15:32
or just humans telling you anything.
15:36
I know from a survey that we conduct in the Mill OPS
15:41
community, there's thousands of people in the Mill OPS community
15:45
ecosystem. You know, like I think we
15:47
tallied it up and there's over like 6070 thousand people that
15:50
have in one way shape or form interacted with the ML OPS
15:53
community For the survey that we do, we get like 150 responses
15:59
and I'm stoked. And so you're talking about how
16:03
expensive it is and how like resource intensive it is to get
16:07
that kind of information back from your users.
16:11
O What are some workarounds that you've found that you don't have
16:15
to almost lean on the goodwill of the users, so that you can
16:20
still make sure that you're doing the right thing, but
16:23
without having to ask for that goodwill?
16:27
Yeah, I think it's a really. I think it's a really good
16:30
question. I I don't know if there is a
16:35
shortcut to get your first batch.
16:38
I think the first batch of feedback is I don't think
16:44
there's shortcuts around going and sourcing it from people
16:47
rolling your sleeves up, talking to your users, talking to your
16:50
customers and trying to put different answers in front of
16:57
them for for the questions that they're trying to solve in their
17:00
day-to-day lives. And just collecting, just
17:04
directly collecting it, to your point, it's extremely expensive.
17:08
And So what I think the methodology that I found worked
17:10
really well is, is that you create this like it's, I almost
17:15
think of how like a planet, your core is your human labelled
17:20
data. That's like what's powering
17:21
everything else. And again, there's no shortcuts
17:25
around that. I will go into a spreadsheet and
17:28
I will label my data. And so for some like this, the
17:32
sad truth that people may or may not be trying to avoid is that
17:36
you just gotta go start and talking about how like you can,
17:41
you can kind of spin yourself out a little bit and talk
17:44
yourself out of going and getting feedback.
17:47
Because there's all this stuff around synthetic data, there's
17:50
all these bootstrapping kind of self improving systems that are
17:54
coming out. But I think going and getting an
17:58
actual labelled data set for yourself, we can use like an
18:02
intent detection system for as an example, just go in and
18:07
getting 100 queries, 200 queries and then hand labelling them.
18:13
And then you sort of use that as your bootstrap set to either
18:17
align another model, maybe a small model or you just use it
18:24
to prompt like a a larger Oracle model to generate more things
18:28
that are of this ilk. And so I think that you begin to
18:32
insulate your core of your your planet with some synthetic data.
18:36
And then depending on if the synthetic data are hurts or
18:39
helps that'll kind of tell you whether or not you have a
18:42
sufficient amount of your actual real golden data to get started.
18:48
And if you don't, then you can go and you can increase that
18:51
initial set a little bit until you start to see that your
18:54
synthetic data is close enough in distribution such that using
18:58
it to train a model is still improving you, improving your
19:02
eval scores, whatever you're using, if one, or perplexity or
19:07
whatever. And so I think that you you
19:11
really there's no way around getting hand labelled table.
19:14
Nothing beats it. Nothing's ever going to beat a
19:18
human being coming and saying what the right answer is.
19:24
But you can use that to create much more sophisticated and much
19:29
more close to distributions, synthetic data.
19:31
Once you have that starting point and then as you start to
19:34
see that you can create smaller and smaller models using more
19:37
and more data, then you can kind of go on this trajectory where
19:41
you start off with this very big Oracle model and this very small
19:44
amount of hand labelled data. And then as you kind of train
19:48
smaller and smaller models, your data is increasing and
19:51
increasing and so you kind of you you have this kind of like
19:55
trade off, whereas your data increases, you can get away with
19:58
smaller models which means you can create more synthetic data,
20:01
which means you can use even more smaller models.
20:03
But you have to make sure that as you're generating more and
20:06
more of that data that you're not generating a bunch of stuff
20:08
that's out of distribution. This is the less is more
20:11
alignment paper right? And paper is not about.
20:15
Data set size. That paper is about data set
20:18
quality and I think that that's something that's largely
20:21
overlooked. Ohh, you only need 100 examples.
20:23
We'll know you need a hundred really good examples.
20:27
And if you have 100 really good examples, you shouldn't show
20:29
your LLM anything else because that's enough to align it.
20:33
But what I've seen is that once you've aligned that it's really,
20:37
it gets a lot better at synthesising new data, which you
20:40
can then make an even smaller model good at capturing that
20:44
distribution. So that's I think the journey
20:47
that I've seen be really successful.
20:49
I wish I had better news for the people who were like I am never
20:53
going to hand label data, but I think that's just kind of like
20:56
it comes with the territory. If you want to work in machine
20:58
learning, you want to be a professional machine learning
21:00
engineer, you're going to label some data.
21:01
Sometimes it's just comes with the.
21:03
Gig. Aren't you that you say that
21:04
too? Because one of our first virtual
21:07
meetups that we had back in 2021 of the guests talked about how
21:12
they would have labelling parties at work.
21:16
So like at 5:00 on a Thursday, instead of everybody going and
21:21
having a beer, they would have the party at the office, they
21:25
would have their beers or whatever it was their social
21:27
hour at the office. But they would all be labelling
21:29
data. And so they would get the power
21:33
of numbers and just be like all right here's how we're going to
21:36
label data and and everyone from the receptionist to the ML
21:40
engineer was labelling the data and I thought that was an
21:45
interesting way of going about it because you you make it fun
21:49
and you also get the power of numbers to.
21:52
So now one person doesn't have to label 100 things maybe they
21:55
just have to label 20 while they're chatting with friends
21:58
and I don't know how the quality is on these.
22:01
If you're like the you might want to check as the night goes
22:05
on what the label data turns into.
22:08
But that was one. And then on the other thing that
22:11
I wanted to just mention was that it feels like what you're
22:14
talking about reminds me of the supply and demand curve.
22:17
And you know, like the price. It's just right there in the
22:20
middle where like supply equals demand.
22:22
It's like what you're showing is, yeah, in the beginning
22:25
you're gonna have a lot of, you're gonna have this big model
22:28
and a little bit of data. But then eventually it can
22:31
switch and you can have a lot of data in a smaller model and you
22:36
can just kind of like watch as one goes down and the other goes
22:39
up in a way. It's it's funny because I I our
22:42
initial intent model at u.com was this you know distilled Burt
22:47
standard transformer model, 100 million parameters, nothing to
22:51
write home about. But as soon as we started trying
22:55
large language models, we were able to beat that model just
23:00
with a few shot prompt and 10 lines of Python.
23:04
I was like OK, well there it goes that six months that we
23:08
built that fully featured intense training pipeline and
23:12
deployment and stuff. But then we were like OK, but
23:16
now we can actually make this model, this large language
23:19
model, this few shotted model much smaller because now we're
23:24
going to use that to label data. But if you think about that
23:28
towards its logical conclusion, you're going to converge back to
23:32
the distil Burt model that we started with.
23:36
Like the logical end of that is that you're going to end up back
23:39
where you started almost, but you're probably going to have a
23:43
large amount of synthetic data that you've created and that's
23:46
going to allow you to hopefully capture that same performance
23:49
that you've got with a few shot, few shot in the Oracle model
23:54
with the model that you were originally using.
23:56
So hopefully you end up ahead still, but you're still like
24:01
it's this very strange kind of acrobatics that that we ended up
24:06
doing. Interesting times.
24:09
Did you ever hear that like paradigm where I remember it
24:13
was, it was like, OK, if you have like the finish line and
24:17
then the car, each time that you move the car, you just like half
24:22
its distance to the finish line. And then you never actually go
24:28
across the finish line, right? Because you're just having the
24:32
distance to the finish line, so you can never fully go through
24:36
it. And it reminds me of that in
24:37
some way. I think there's like a, there's
24:41
like a a Greek fable about that. It's like Archimedes arrow or
24:46
something like that. One of those philosophers is
24:48
like, if I shoot this arrow, it's never going to hit someone.
24:50
That's it always has to go half the distance.
24:53
Surprise. It'll hit the guy and it'll
24:56
hurt. But yeah, it's umm.
24:59
It is one of those things where it feels like the yeah, the
25:04
finish line is um, the official line ends up being where we
25:09
started. Almost like a like a like a
25:11
loop. But it was an interesting
25:14
journey and I think it illustrates kind of just the
25:17
type of step function that we were seeing at the end of of
25:20
2022. Going into 2023, things had just
25:24
fundamentally changed in a way that, you know, I still don't,
25:28
I'm still not confident that we are appreciate how much things
25:31
are going to be different in five to 10 years.
25:34
But you know what? We should all enjoy this time
25:36
because it's probably gonna look back on and be like that was a
25:38
really special time to be in machine learning like that was
25:41
pretty sweet. So anyone out there listening?
25:44
It's just find ways to enjoy this time and history, cuz
25:48
you're never gonna get it back, right?
25:50
Umm, so talk to me a bit about Rewind and what you're doing
25:55
there, because I think there's a lot of stuff that we can dig
25:58
into with how you are now leveraging AI and also how
26:08
you've built rag systems. I know that we wanted to like
26:11
fully dive into production rag and how that's very different
26:15
than what we see on all the social media networks and how we
26:20
have to be very aware of some caveats before jumping into like
26:25
creating these production rag systems.
26:27
So let's just start with like Rewind.
26:30
I know what it is, but it's probably worth telling people
26:33
out there what it is. Yeah, sure.
26:36
So Rewind is fundamentally a a company that is focusing on
26:44
giving people superpowers via augmenting memory.
26:48
What this means in practise is it's an extremely powerful data
26:52
collection platform for those people who are willing to, you
26:57
know, fork over a lot of their data, a lot of their everything
27:03
they see and hear. We have full microphone audio
27:06
capture. We have full OCR on top of your
27:09
screen capture and we convert everything to text and we index
27:13
all of that and try and put it to use for people.
27:17
And I think for those out there who are in the camp that they're
27:22
just like take all my data, I don't care, just make me, if you
27:28
can give me like 30 minutes back, like an hour back, just
27:32
take the data, I don't care. I just want the time back for
27:36
those people and there they are out there.
27:38
There's a good amount of them. We are offering a solution where
27:43
we're going to try and make all everything you see and hear
27:47
accessible by an LM. We're going to try and create,
27:51
you know, these proactive flows that are going to one day
27:55
hopefully even interrupt your flow and be like, hey, you know,
27:59
I did some work over here and this is what I think about the
28:02
thing you're working on and maybe even give you like just
28:05
someone to kind of look over your shoulder and check your
28:07
math. And very similar in spirit I
28:11
think to what I expect out of like a pair programmer.
28:14
You know, someone to go look things up for me while I'm while
28:18
I'm typing code and and maybe talk things through with me, go
28:22
do some research, maybe go write some tests, exploratory code.
28:26
Those are the types of things that I traditionally think of as
28:29
like a passenger. If you're thinking of driver,
28:32
passenger and pair programming and being able to create
28:36
something like that, that is never gets tired, never has to
28:40
go, like take a bio is just like they're with you until the end
28:45
of the night. It's gonna be a really powerful
28:49
experience. I think for people who are used
28:53
to working collaboratively with humans, and especially if you're
28:57
getting used to working collaboratively with an AI in
28:59
your day-to-day kind of workflows, this is going to be
29:03
something that is going to, I think, really be a step function
29:05
in productivity for you. And so right now it's taking
29:09
everything that you're doing and taking everything that comes in
29:14
or you interact with out on the Internet or that you speak with
29:18
whatever it may be, whatever form you interact with your
29:22
device and it's able to index the index that so the search is
29:27
on point and then it's also able to give you back answers.
29:30
And so your like, First off, I think the burning question on my
29:39
mind is multimodal rag. Are you all doing that already?
29:44
Because that's like the buzzword of the year, right.
29:46
And tell me about it. What is going on there?
29:50
Ohh, I think that I think that I I can't say too much but I if
29:56
you were, we're talking about logical conclusions before.
30:00
I think the the logical conclusion of a OCR and ASR on
30:04
top of you know those mediums and then converting it all to
30:09
text and then having some text model that operates on those as
30:14
kind of a proxy. I think most people could see
30:16
that any that a model that can operate directly on speech and
30:20
directly on images, instead of having these two models have to
30:24
kind of translate for the text model, is going to be a more
30:29
performance system because you don't have that extra
30:32
translation layer in there. You're operating directly on the
30:35
input signals and therefore you can fit features directly to
30:39
them instead of these like proxy features on top of the text.
30:43
So if you're keeping track at home, I think it's pretty clear
30:47
that a multimodal approach is going to be really beneficial
30:51
for the likes of rewind. Can't say where we are in
30:53
development, but I think it makes a lot of sense for us to
30:57
invest in that technology, yeah. All right, real quick, some
31:01
words from our sponsors, Ellis, and then we'll be right back
31:03
into the programme. Are you struggling with building
31:07
your retrieval augmented generation, AKA RAG
31:10
applications? You are not alone.
31:13
We all know tracking unstructured data conversion and
31:17
retrieval is no small feat. But there's a solution available
31:21
now. Introducing zelizer Cloud
31:24
pipelines from the team behind the open source vector database
31:28
mill this This robust tool transforms unstructured data
31:32
into searchable vectors in an easy manner.
31:36
It's designed for performance. Zellis Cloud Platforms Pipelines
31:42
Zelis Cloud pipelines also offer scalability and reliability.
31:47
They simplify your workflow, eliminating the need for complex
31:51
customizations or infrastructure changes.
31:54
Elevate your rag with high quality vector pipelines.
31:58
Try Zelis Cloud Pipelines for free at soliz.com.
32:03
You can find the link into the description.
32:05
Let's get back into this podcast.
32:07
Yeah, yeah, 100%. And so again, like today, you
32:10
get to take in all of this information.
32:12
You get to index it and be able to search it really easy.
32:16
And I know, I think the thing that I loved when whenever I
32:20
look at rewind, it's like, ohh, remember that time that I was
32:25
reading that thing that had that stuff in there?
32:28
Like, I remember this emoji that was in it, but I don't remember
32:32
anything else. And I remember vaguely that it
32:35
was like, about this. And I can just give all that
32:38
information to rewind. Or ideally one day, as you're
32:42
mentioning, rewind will be able to ask me like, yeah, so tell me
32:46
more about it. And what else was it?
32:49
Something along these lines. And this happened to me just the
32:52
other day because I was been quoting this paper from these
32:57
folks from the University of Prague on how the, the data that
33:03
you send to an LM, it inadvertently leaks that data.
33:07
And so these folks that at Rogue did this whole paper and they
33:13
put it out and I was like, cool, I need to really, like
33:15
synthesise this paper and understand what's going on here
33:18
because I'm quoting it a bunch, but I don't really know what is
33:21
going on. And so I tried to like, find
33:26
that paper again, impossible. Impossible.
33:29
And I remember it had leaks in the word and it had an emoji.
33:33
But I was like, was it a leak emoji?
33:35
So I'm going back and like looking for this leak emoji.
33:38
There is no leak emoji. First of all, it was like, it's
33:41
either cabbage or broccoli. So I'm looking through my Google
33:45
search history for like, cabbage or broccoli trying to philtre.
33:49
I spent way too long doing this dude, literally way too long.
33:52
And then I'm searching like archive for like University of
33:56
Prague papers leak in the title and and then I'm like is it LEK
34:01
or is it LEAK disaster? I can't find that damn paper for
34:05
the life of me. If I had rewind I assume that
34:09
would be much easier. And if I could just ask rewind
34:13
the questions directly like so. What was this paper about?
34:17
Yeah, data leakage. Yeah, I think that creating
34:20
those linkages automatically is somewhere where we're gonna
34:26
start to really shine. And it's it's it's not all there
34:31
yet. But if you imagine even having
34:34
this conversation, imagine being able to have this conversation
34:38
at the paper that you forgot and describing it.
34:40
And having it be able to go into your memory, your virtual
34:45
augmented memory, and be like this is the paper.
34:49
And then automatically create a linkage through time for you
34:53
that will actually create this ability to trace your steps back
34:58
through time and your thought process.
35:01
Those are the types of things that today human beings just
35:03
forget, right? Like there's no tool that's
35:06
going to actually tell you unless you have really great
35:09
note taking, which I mean not many of us do anymore.
35:14
We just were gonna forget and it's just gonna be like ohh you
35:17
know how many times on the day you're like what was that?
35:19
I must not have been important. Well imagine if every single
35:22
time rewind was able to go back in time and find those things
35:26
for you and be like and then that actually related to this
35:29
thing even further in the past. It's related to these two or
35:32
three events which creates this like lineage of almost thinking
35:35
and cognition for you, which is I think for people who have a
35:40
lot of knowledge work to do and they are trying to keep a lot of
35:43
stuff in between their ears. That type of like, you know,
35:49
read only memory so to speak is gonna be really, really valuable
35:53
to them because there's just too much for us to to to remember
35:57
every single day you know this you run the MLS podcast.
36:02
How about this stuff comes out every single day about machine
36:04
learning nowadays. Dude, seriously?
36:07
And just thinking about all the people that I interact with on
36:12
the Internet itself too. And so many times, it's like
36:15
somebody was just telling me about that.
36:17
And I feel kind of bad sometimes these days because I'm like, who
36:22
just told me about that? I swear.
36:24
Ah, and it's it's gone like you're saying.
36:27
And so to be able to get that back is at least a win for
36:32
attribution. So I don't go around
36:34
plagiarising other people's thoughts as much as I am now.
36:38
But one thing that I was also wondering with is like, are you
36:41
looking at signals behind the scenes as far as I've had this
36:47
tab open? Like for example, this paper
36:51
that I wanted to read never got around to reading it for some
36:54
reason. I updated Chrome or something
36:57
and then it didn't come back up and then I lost it forever,
37:00
right? Are you also looking at signals
37:02
as far as, like this paper has been open as a tab for five days
37:07
or 10 days and so that gives you a bigger window or it gives you
37:11
a higher weight on things breakdown like how that looks
37:16
too. Are there those types of
37:17
features that you're taking into account?
37:20
So I think that in an in effect I think we are because we do
37:27
have ways to kind of handle time specific queries.
37:35
But I think that if you're asking like hey, are we using
37:39
like behavioural signals for like re ranking results and
37:42
we're going to use like are we actively like using metadata
37:47
about tab life cycle or whatever as features.
37:53
I don't think we're quite there yet.
37:55
I think that there's a lot to master in text and and one of
37:59
the things that I see as a as a mistake.
38:04
Um, in industry today is like trying to convince yourself that
38:09
you don't have to master text for XYZ reasons.
38:13
There's lots of companies with a vested financial interests and
38:15
convincing us of that. But I think that at the end of
38:20
the day that is something that you really want to master via
38:26
the text medium. And I think that trying to go
38:32
straight towards, we're going to use text and we're going to mix
38:35
in these behavioural signals as well and we're going to create
38:37
this composite semantic and behavioural feature set.
38:42
I think that if you're doing that and you've got a team of
38:45
people, big team of people, go for it.
38:48
I think for most people, if you are not comfortable that you've
38:53
mastered text, that's what you should be doing.
38:55
Master Text get really good at that.
38:58
And once you feel like you've mastered that and you can train
39:01
and deploy your own text models, then I think you start to create
39:05
your ensemble models, right. But I wouldn't start with an
39:09
ensemble until life mastered the the models that make up the
39:13
ensemble first. Otherwise, it's going to be
39:15
tough to explain it's you're going to end up with something
39:18
that isn't quite what you're looking for.
39:20
It doesn't do quite what you're looking for if you don't
39:22
understand how the ensemble is actually compositing those
39:26
predictions. Yeah.
39:28
In other words, it's like the content inside of the document
39:34
is more important than the metadata around that document.
39:39
In a way, like, I don't care if you've looked at it five times
39:42
in the last days, is it relevant to the question that you're
39:45
asking me right now? Is the actual content relevant?
39:50
And then later, if I can, without a shadow of a doubt,
39:55
figured that out, then I can start adding in these features
39:59
where it's like, OK, cool, it's. The question that you're asking,
40:06
this document has relevance, and it's what you most recently have
40:10
been viewing, so that's probably where you're going with this
40:12
question. And I think that it's going to
40:15
be different for people who are operating in small domain search
40:19
versus web domain search for EU dot coms and the Googles and
40:24
things of the world. Once you're operating at web
40:27
scale search, there are popularity signals.
40:31
You know, domains that in the top 100 cloud flare collect
40:35
less. Things like that are extremely
40:38
important because you search for the word attention and you can
40:41
get stuff from a variety of different subjects.
40:44
There's different ways to use the word attention nowadays.
40:47
And then I think if you're bringing in like behavioural
40:49
signals and even personalization signals like, hey, Dimitrios is
40:54
a machine learning person, you probably means attention
40:57
mechanisms. Yeah, that's when I think that
41:00
those types of signals, the the behavioural signals really come
41:04
in and start to shine. But if you do not have 10s of
41:07
billions of documents you're searching over, you probably
41:10
don't have the problems that warrant bringing those signals
41:14
in unless you're really good at them and you have really clean
41:17
signals. But then you're probably at a
41:19
point where you don't even need a machine learning model.
41:21
You could probably put business logic on top of that
41:23
representation of the data and beyond your way, right?
41:27
No, Yeah, it's it's like trying to over engineer it.
41:32
And that's what you see all around the industry.
41:35
It's everything is so complicated.
41:37
Everything is you know, requires 5 different systems and it's
41:43
it's become really hard to figure out what an MVP LLM
41:47
system looks like nowadays because if you naively look
41:50
around you're like wow, that's I need like 10,000 lines of code
41:54
to run one of these and that's like it.
41:55
It's so diverges from the true like spirit of the LM from late
42:02
2022 to early 2023, which was ohh my gosh, you have to do so
42:07
vanishingly little work to get these things to work.
42:11
Like the whole point of the anecdote around the intent model
42:16
was, you know, ohh my gosh, we spent six months on this fully
42:20
featured NLP training system and then we really just blow it away
42:25
with 10 lines of Python And a prompt.
42:27
Oops. If you are creating more work
42:33
for yourself that you are saving using an LLM, then you're
42:35
probably using it wrong. Like LMS are supposed to make
42:38
things simpler. They're supposed to lower the
42:40
barrier of entry and they're supposed to short circuit U to
42:42
being able to solve more complicated problems.
42:45
You are not getting those benefits then you are not
42:49
getting what you are legitimately paying for, and
42:51
they are not very cheap things to pay for.
42:54
So, like, make sure you're getting what you like.
42:57
Make sure you're getting what you're paying for, and what
43:00
you're paying for is not a huge, convoluted milot system that's
43:06
required to operate these things.
43:08
Start simple, start getting business value with it, and then
43:12
you'll start to figure out where your life cycle breaks down and
43:16
where maintenance and operations are going to start to seep in
43:19
and require you know some thinking.
43:22
Well, let's let's go into this because I know that you had
43:25
spoken to me about this very thing before we hit record,
43:29
which is like optionality around when you're doing evaluation.
43:34
When you're doing search, you don't necessarily need to chase
43:38
the height and you don't. Just because one design pattern
43:43
is using looms as a judge these days doesn't mean by any means
43:49
that you should feel like you have to do that and you're not
43:54
going to be able to get business value out of your application if
43:58
you're not doing that. Or the same goes for LIKE using
44:02
neural networks for search, right?
44:04
And so give me the, like, give me your take on this, because I
44:10
think there's some really interesting facts here that
44:13
you're bringing up. Yeah, I I think that if you are.
44:19
One thing I think is that if you are ever in a situation where
44:23
someone is trying to make the case that add this neural
44:25
network to your stack, it'll make things so much simpler.
44:28
Like they are selling you something, you are being sold a
44:32
bill of goods at that point. Um and there might be, there
44:36
might be heat to that smoke and there might not.
44:38
You know I would take all of that with a grain of salt.
44:43
I think that your your tried and true kind of counting algorithms
44:47
or where I would start with any of these things.
44:50
You know, word frequencies, They can get you pretty far in
44:53
search. TFIDF get you pretty far
44:57
standing up like a standard single node opens, or like open
45:01
search or elastic search, like one database, and then using
45:05
that instead of a neural network and delegating everything into
45:09
BM25 inside of the scene. That'll get you really far.
45:14
It's also simpler than operating your own neural network.
45:17
And I think the same goes for evaluation.
45:20
It's the same thing, right? Add this neural network, it'll
45:22
make evals easier. Well, it's is it going to
45:26
really? Is it going to be simpler than
45:28
me just converting both the predicted answer and the ground
45:33
truth answer into a bag of words and then just counting the
45:36
number of words? Similar.
45:38
Probably not. I think that I've largely gotten
45:41
pushback on those approaches because of, you know, it's not
45:46
quite what you're looking for or it's going to be a floor on the
45:50
performance because you're technically actually penalising
45:52
the models for certain failure modes that they like an LLM may
45:56
do better at. But I think that that's OK.
46:00
Like as long as you have a floor and you don't have a lane, like
46:03
you're not like overestimating your performance.
46:05
As long as you're underestimating your
46:07
performance, you're OK. And I think that if you're in a
46:10
situation where it's like I should use this neural network,
46:12
or I should use this very simple counting algorithm that will
46:15
give me a lower bound on my performance, just start with the
46:19
simple thing. Start with a lower bound.
46:21
You can make progress on a lower bound you don't need.
46:23
An exact thing is, if your lower bound is increasing your F1
46:27
scores, the number of words that you're getting correct is
46:31
increasing that you're probably making progress.
46:34
It may not be progress on the perfect metric, but you don't
46:38
need the perfect metric. You need a metric, something
46:41
that you can understand and that you can actually synthesise a
46:45
technical strategy around. And if you have a very
46:47
complicated evaluation system, it becomes challenging to
46:52
synthesise a technical strategy and be like, well, this says
46:54
that this is unfactual. And this is like there's these
46:58
metrics that are kind of human understandable but also like
47:01
they don't quite like correlate across data points, if you know
47:07
what I mean. Like the factuality of a 10 here
47:10
might be an 8 here, when in reality they're both pretty much
47:13
totally correct. And I think that getting away
47:17
from these very simple baselines is where I see most people going
47:22
wrong. Your first baseline should be
47:25
super easy to beat because it should be very simple.
47:28
That's why it should be your first baseline.
47:30
You can go get a little dopamine hit.
47:31
You're going to beat your baseline, but at least you know
47:34
you're making progress and at least you have some metrics and
47:36
some guardrails to tell you that.
47:38
Like this is, this is what progress looks like.
47:42
And I think that if you aren't starting there, then you are
47:47
running the risk of ending up with a system that kind of
47:52
escapes your ability to understand because of the
47:55
complexity that you're adding on piece by piece versus just
48:01
having a very, very simple system that when you can't make
48:04
progress anymore, Then you may be in a position where your
48:08
metrics need a little bit more of a nuanced definition that you
48:11
may need to find more other metrics to measure besides a
48:17
simple word counting. You know, intersection over
48:20
union counts, but you should do that then, not at the beginning
48:28
that that. That would be my main takeaway
48:31
if you could, if I was starting today.
48:35
Well, if I'm understanding this correctly, and I think this is
48:38
fascinating because basically you're saying like, we can get
48:45
so caught up in trying to make the system robust that because
48:52
of that goal of having a very robust system, we get lost and
48:58
we can't debug and we don't know where things are going wrong or
49:02
we can't really even interpret what is going on.
49:05
So start with simple metrics, Push those to the limits.
49:09
Once we feel like we've gotten to the natural limit on that,
49:12
it's like, great. This is our very clear, very
49:15
understandable metric that we can go off of.
49:19
And now let's find, let's add a new piece to the puzzle of this
49:24
system that we're creating, and let's figure out more metrics
49:28
that we can go off of. And each step along the way,
49:31
you're adding that robustness to it and you're adding the
49:34
complexity. But you're adding in the
49:36
complexity in a way that you can understand.
49:39
So when you're building this big Lego base, you know which piece
49:47
is which and what it does and what metrics it's it's tied to.
49:51
And you can always, in effect, rollback if you need to, as
49:57
opposed to like we build this gigantic system and now we gotta
50:02
debug it, but we're not really sure what is causing the problem
50:05
and how do you rollback from there It it's like if you go
50:08
from zero to 100 then you can only roll back to zero, as
50:11
opposed to going from zero to 1010 to 20, then 20 to 50, that
50:17
type of thing. I I couldn't.
50:18
I could never have put it better myself.
50:21
I think that motivation is a really keyword here.
50:24
If you're actually making those changes in a stepwise manner,
50:28
you have the motivation, you understand the context of adding
50:31
them, You know what it's actually doing for your system
50:34
and what the capabilities are added in.
50:36
If you're just adding all that stuff from the beginning, I
50:39
think what you're fundamentally lacking is the motivation for a
50:42
bunch of that complexity. And so you may end up with
50:45
things that you don't entirely understand their purpose or what
50:48
their place is. Because you don't.
50:50
You didn't have a real motivational thrust to add it.
50:55
Your motivational thrust was Twitter, and that's not where
50:59
you should be getting your motivation for adding complexity
51:01
to your software. I don't know if I'm bursting any
51:05
bubbles here, but sorry. Yeah, it's like Ohh, I heard I
51:12
would be able to get my next job if I just add this gigantic
51:15
system in there and. I love vector stores.
51:19
Can I get a job now? It's like, yeah, no, solve real
51:22
problems with natural language processing and you will get a
51:25
job, no problem. But it's really hard to solve
51:29
those real problems if you don't understand what you're building.
51:32
Yeah, yeah, it's that understand the Oh my God, I don't even
51:37
think that's a word. Understandability is where I was
51:39
going to say. But just the understanding of
51:42
what you're building is so crucial because you're starting
51:46
with that foundation and you're making sure that you are clear
51:50
on that foundation and from there everything else can be
51:54
built just like we say this a tonne like you can't have ML OPS
51:58
without a very strong data culture.
52:02
You you can't like just jump straight from having nothing to
52:06
like, OK, now we're gonna do AI and think that you're going to
52:10
have success and so you want to have that, you want to have
52:13
strong data and then you can build your ML on top of it and
52:17
then you'll see that success after.
52:21
But this is very much in that vein of you want to make sure
52:27
that your foundations are strong before branching out and
52:31
complicating things and dare we say, over engineering it.
52:36
There, I say, yeah, and I to be fair, I'm a very simplistic
52:41
engineer. I'm a simple guy.
52:43
I like simple systems. And so this is obviously my
52:46
bias. You can take it all with a grain
52:47
of salt, but I don't think that most people will tell you that
52:52
they've gone wrong in their career from being too simple and
52:56
simplifying problems too much. I think people probably will
52:59
tell you the opposite quite a bit, but I don't know.
53:02
A lot of engineers would be like, ah, I just made it too
53:04
simple. Dang it.
53:06
It was just too easy to understand, debug, Gosh, you
53:12
know, I don't hear that a lot. And I think it's really
53:15
seductive to do it and machine learning because it's so hot.
53:19
But if you keep it simple, I think that people will have more
53:22
success in real life. And I think that that will
53:25
translate to success in their careers because that's what
53:28
companies are looking for. They're not looking for someone
53:31
who can be really smart and say all of the smart AI things that
53:36
are looking for people who can solve real user problems.
53:41
And now talk to me about like production rags.
53:45
In this same vein of trying to make it as simple as possible.
53:50
I imagine you've seen on the socials that there are new terms
53:55
coming out of like naive rag. And that's like where it's just
53:59
ohh, it's naive because it's only these three steps and you
54:04
get your embeddings, you have your vector database, you throw
54:07
it at the LM and then you're good and.
54:11
I. On one hand, I see the point of
54:15
saying that's cool and all, but that's not really production
54:19
ready. Like, that's not exactly you can
54:23
go places with it, but if you're trying to go and set up a really
54:27
productionized system, you have a few places where that can fall
54:32
flat. And so I'm wondering from your
54:35
experience building rags, how do you see this?
54:39
And like, how does it all fit in with?
54:41
OK, the ethos is start simple, then go complex step by step.
54:47
So naive rag is a great place to start.
54:50
Even though I imagine with that name, nobody wants to start with
54:56
a naive rag. Now it's like we just killed it.
54:58
Nobody wants to say that they're doing naive rag.
55:00
I'm doing like most advanced complex rag you can think of.
55:04
But then how do how do rags look in your eyes?
55:08
And what are some gotchas that you found over the years?
55:13
Yeah, I think. I think that, umm, you you you
55:18
kinda you took the words right out of my head.
55:21
That or right out of my mouth. You know, I think that it it
55:25
goes back to that motivation thing, right?
55:29
I think that a naive solution, you can call it naive.
55:32
I think a naive solution is a perfectly good place to start.
55:36
And I think again, you want to. If you're going to add
55:39
complexity, you really want to have that motivation.
55:42
And the way that you get motivation is by having metrics
55:46
and benchmarks that could actually create that motivation
55:49
for you. And so again, we used this
55:52
example of trying to make progress on very simple metrics.
55:55
I think that you can kind of translate that even to
55:57
architectures. I think that if you have a RAG
56:01
system and then you have a more advanced RAG system, if you
56:05
cannot be very demonstrable with what the improvements are
56:10
outside of cherry picking, And I know it's so tempting to cherry
56:14
pick, but they are they. Those are selected examples that
56:19
you're seeing regularly. You you need to have set up and
56:26
your setup could like you're naive RAD could be generate
56:30
random text and that's your that's your generate function,
56:34
right? Just output random text.
56:36
It's gonna be a really easy baseline to be with a naive for
56:39
access to because you're competing against random texts
56:42
like you're competing against. Noise is in effect.
56:46
And so I think that it's perfectly reasonable to move up
56:50
the complexity and add you know knowledge graphs, add these very
56:54
complicated trees and graphs of thought and all of those agentic
56:58
search systems. I mean I've built systems where
57:01
you know I had agents and those agents had actually access to my
57:04
individual indices inside my database.
57:07
And it would be like I'm going to search this index, I'm going
57:09
to take the results, I'm going to feed those into this and
57:12
build multi hop reasoning agents.
57:14
Like you could get truly complex.
57:19
But I think that if you aren't motivating those examples and
57:23
you aren't actually able to show, these are my metrics
57:27
before and after. And here are a really good
57:30
representative bunch of queries that we were really bad at, but
57:35
now we're actually really good at.
57:37
And luxury doesn't actually degrade performance in these
57:40
other spots. I think then you can motivate
57:44
your more advanced techniques and then I think you can
57:46
actually go and figure out which of them.
57:49
The really nice thing about them is it becomes very easy to vet
57:53
what smoke and what's fire, because the things that are fire
57:57
will make your metrics move. And they're like, ah, that's
57:59
actually legit. Let's keep that.
58:01
And then you'll add this other thing you like that didn't do
58:04
anything. We'll delete those 100 lines of
58:06
code, right? All of this needs to be
58:08
motivated. And I think that if your
58:10
motivation is the name of the system, then I mean that's one
58:14
way to do it. But don't you don't even need
58:18
rag to begin with as your first system is is my only points.
58:21
Like have something you can make progress on and have something
58:25
that's easy to beat to begin with and then motivate your
58:28
changes in your additional complexities through that.
58:31
You may find that you end up at a system that is not represented
58:36
at all on social media or anything because you actually
58:40
did the work to create a system that solves your businesses
58:43
problems and only your businesses problems because
58:45
you're using your businesses data to actually judge your
58:48
systems and improve your systems.
58:51
And if you do that, then you're going to build models that are
58:54
very special for your for your business, that solve your
58:57
businesses problems better than anyone else's models.
59:01
And you're going to have the simplest system that does it,
59:03
and you're not going to have any craft.
59:06
And then you're going to be our, like, everyone's gonna love you.
59:08
People would be like, Oh my gosh, give me your autograph.
59:12
Trust me, they'll love it. Then when you organise your
59:16
labelling parties in the office, people will show up.
59:19
Yeah, that is how people show up.
59:21
They sure will. They will understand the beauty
59:24
of it and that will make total sense.
59:27
Yeah, it going back to the motivation, it's just like be
59:32
very clear at why you're doing things.
59:34
Don't just do it for the sake of doing it.
59:36
Or, as you were, as one friend Flavio put it.
59:41
He called it a resume driven development.
59:44
And so that one. Yeah, just because I need to
59:48
make sure that my next job is going to be a step up.
59:53
I'm going to implement this crazy in depth system here and
59:56
then I'm going to go and talk about it.
59:58
Maybe I'll submit a paper or something so that I can show off
1:00:02
how badass I am. So I I can see that for sure.
1:00:05
I wanted to switch gears a little bit and talk about your
1:00:09
work with. I know you've contributed to
1:00:11
DSP. Or as I like to call it, Dipsy
1:00:15
and what what are your thoughts on that?
1:00:18
It seems like you've had a few of your contributions actually
1:00:22
like merged. You've done some cool stuff with
1:00:26
them. We've had Omar on here.
1:00:28
I am huge fan of what he's building actually it's been
1:00:34
really cool to see how much attention it's been getting
1:00:37
lately and but but I also know that you've done stuff with Lang
1:00:41
chain too. So I just kind of like want to
1:00:44
set the groundwork with that. You done a little bit of, I
1:00:50
guess you could say you've played around with both and
1:00:54
you're familiarised with both and I want to get your feel on
1:00:59
and your take on these different tools that are out there.
1:01:02
Yeah, I think the thing that caught my eye about DS Pie or
1:01:07
Dispy was that I I felt that lying chain and llama index and
1:01:14
some of those abstractions, while they were useful I think,
1:01:19
for organising people's thoughts and organising the discourse
1:01:25
around alms really, really well. Um, I always felt that despite
1:01:32
brought a little bit more to the table in terms of what I was
1:01:35
trying to accomplish. I think that and that I think is
1:01:39
by design in some respects. I think that Lang chain is
1:01:43
fundamentally about representing these processes in a way that is
1:01:48
you don't understand and very translatable and makes it so
1:01:53
that we can build a community around this and increase mind
1:01:55
share, which is like a very, very important thing to do in
1:01:58
this field. I think DSP is motivation is
1:02:03
slightly different. I think that the motivation
1:02:05
around self improving systems is what really caught my eye and
1:02:11
there are these even these these kind of self improving RAG
1:02:16
search agents that you know deep mind has put out papers on in
1:02:20
the in the very recently DSPI really came and said hey I have
1:02:26
an opinion on what a self improving LLM is.
1:02:29
My opinion is, is that we're going to create these student
1:02:32
teacher models and we're going to try and like create a
1:02:36
framework for this distillation process.
1:02:39
And that distillation process is I think something that is
1:02:42
actually very challenging to understand and challenging to
1:02:51
represent. I guess any DSP does a really
1:02:54
good job of giving you code sort of as a blueprint for what a
1:03:00
system would look like that could.
1:03:03
Can you take a very large model and automatically search through
1:03:07
prompt space to optimise that in the few shot setting?
1:03:11
Can I use that to actually synthesise data?
1:03:14
Can I use that data to then actually train a smaller model
1:03:17
that can do that very similar things to what we were talking
1:03:21
about before, Things that I had actually seen work for a company
1:03:25
this this distillation and really taking a shot at creating
1:03:30
a very rigorous way of representing that and
1:03:34
orchestrating it. And that was, I think, what I
1:03:38
saw as the future. These systems eventually will
1:03:43
require very little input data. You know, we have our core at
1:03:48
the centre of our planet that's heating everything.
1:03:51
We start with that and then we basically create frameworks for
1:03:55
these systems to self improve and there isn't much work that
1:03:59
the human does after that. This work that we're doing to
1:04:01
systematically create more and more data sets with smaller and
1:04:04
smaller models, that kind of supply demand curve that we were
1:04:08
going over, that will all just be orchestrated by a system,
1:04:12
right? Like that whole process which
1:04:14
was done by a human being will be done by a programme and that
1:04:17
programme, one of them could be DSP I.
1:04:21
Ohh. Fascinating.
1:04:23
Yeah. And that's one thing that I
1:04:25
liked about Omari's vision for this was talking about, like
1:04:30
from prompts to pipelines. And it's almost like you're
1:04:33
echoing that in different words, like saying, hey, it's not just
1:04:38
the prompts that we're fucking with here.
1:04:40
We're trying to create the system so that we can put a
1:04:44
little bit of data in on one side and get our distilled model
1:04:47
out on the other side. And that whole pipeline is done
1:04:51
in a way that we have confidence that it's optimised.
1:04:56
Yeah I think that's that's a lot of what I've seen be really
1:05:01
really powerful and and when I was experimenting with it I I my
1:05:06
state-of-the-art numbers were came from DSP I.
1:05:10
And so I saw again this is one of those things where if you
1:05:14
have a really good metrics, you know what smoke and you know
1:05:17
what's fire because this system gave me a state-of-the-art
1:05:21
performance and this other one did it and so on.
1:05:25
That's a cop out answer. But you know when you have
1:05:29
metrics, they're all cop out answers, aren't they?
1:05:31
Because you're just like do the numbers told me to I I I'm
1:05:35
really easy to be peer pressured by data.
1:05:38
Made it better, dude. Well, yeah, That brings me to
1:05:41
the next probably the biggest question that I've heard people
1:05:45
talk about when bringing up DSPI is do you know anyone using it
1:05:51
in production? I don't know anyone myself.
1:05:57
The work that I had been doing was still in R&D.
1:06:02
Well, I was still at Ucom, so I don't, I think that there are
1:06:08
maybe a few more ergonomic things that are stopping that.
1:06:12
I don't think that it's anything to do with them.
1:06:16
I don't think it's anything to do with the the paradigm.
1:06:20
There is some things that are difficult to do right now, like
1:06:25
deep speed distributed training. There's a branch on DS Pie which
1:06:30
kind of does deep speed, but you also would have to be running
1:06:34
that DPI on like array cluster or some sort of torch cluster
1:06:39
that's actually running that and can run the deep speed plugin.
1:06:44
And so I think that there's a, there's a little bit of a
1:06:48
disconnect there from where I was.
1:06:50
I was able to get DSI working very, very well all on kind of
1:06:54
third party alms, but then it was time to move in house and
1:06:58
actually train our own. And I think if for people who
1:07:02
are on that journey, they're going to find that distributed
1:07:07
training is going to be their larger upfront blocker to being
1:07:11
able to do what they want to do, especially if they're trying to
1:07:14
train very large context length models.
1:07:16
If you're trying to actually train on 8K tokens of context,
1:07:20
you're probably only going to fit one example per GPU, even if
1:07:24
you're using not the biggest model in the world.
1:07:28
And so that's all to say that I think that for people who are on
1:07:30
that journey I would figure out your your data and your
1:07:34
infrastructure first. And once you have those set up
1:07:39
that's when I think you can bring something like a DSP I and
1:07:42
to really supercharge those efforts and give you a framework
1:07:45
to kind of couch all of your discussions and and it just
1:07:49
gives you this really, really good lens of of looking through
1:07:52
optimising looms. But there's just that little
1:07:56
ergonomic kind of hole there and and Shadow Omar he built some
1:08:01
really amazing things. I think that the the vision he
1:08:04
brings to bear is really, really differentiating.
1:08:09
And in the yellow kind of like framework space, like he's got
1:08:13
ideas and they're good. So that that's probably how I
1:08:17
would do things. Um Tata Omar.
1:08:21
Yes, that is very, very well said, dude.
1:08:26
Sam, thanks so much, man. From this is Paul.
1:08:30
You're humble beginnings of this podcast talking about your
1:08:34
pinball health that you have at home.
1:08:38
And if anyone wants to go through the equivalent of the
1:08:42
Navy Seal's Hell Week and pinball, I guess it is hit up
1:08:45
Sam and you can talk more about how to become the pinball
1:08:50
champion of the world. Dude, I had so many great
1:08:53
insights from this. I think the biggest thing
1:08:56
though, walking away is just like an all be parroting this
1:09:00
back and I'll probably be saying this.
1:09:02
So I'm going to just tell you right now, I will steal it.
1:09:06
Hopefully it's OK with you. I'll plagiarise it, try and give
1:09:09
you credit as much as possible, but the motivation factor like
1:09:14
no. Why you are creating each line
1:09:17
of code. Know why you are creating each
1:09:20
extra building block and what motivates that is so crucial.
1:09:26
Code is baggage, right? Code is where bugs live, so you
1:09:30
want less code. It's one of the things that like
1:09:33
as you go from junior to senior, like I don't want more code, I
1:09:36
want no code. How do I do this job with no
1:09:40
code? And that's like the journey that
1:09:42
you go on, I feel like. So if there's anyone out there
1:09:45
who I've Cavanaugh, I'll push and they're going to maybe get
1:09:50
to the the simple way of thinking and keeping it simple
1:09:54
then I'm gonna be a happy I'm gonna be happy.
1:09:56
So I appreciate you preaching the good word.