Latest / Agentic Conversations (formally mlops.community) / Python Power: How Daft Embeds Models and Revolutionizes Data Processing // Sammy Sidhu // #165

Transcript

Python Power: How Daft Embeds Models and Revolutionizes Data Processing // Sammy Sidhu // #165 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Sammy. I'm the CEO of Angel and I drink
0:04
espresso at the splash of almond milk.
0:07
Come back to another ML OPS community podcast.
0:10
I'm your host Demetrios, and today I am flying solo for the
0:15
Once in a Blue Moon Times that I do this.
0:18
I got the pleasure of talking to Sammy Sidhu.
0:21
This was an incredible podcast. We go through all of Samy's
0:26
history. It was an incredible stint that
0:28
he went through in the autonomous driving, autonomous
0:31
vehicle. Era and he went through actually
0:35
2 acquisitions of these startups that got bought out by bigger
0:39
companies. The first one got bought by
0:41
Tesla and then the second one got bought by Toyota.
0:44
And everything that you learned from that time doing this very
0:49
low level stuff with tiny ML and optimising, distilling these
0:54
models really making things work fast and work reliably he put
0:59
into his new company eventual and eventual has a.
1:04
Offering that is open source and I encourage anyone to check it
1:08
out. It's called DAF.
1:10
We'll leave the link to in the description to that for anyone
1:13
who wants to look at it and. Let's just jump into this
1:17
conversation with him because I think you all are going to love
1:22
what he talks about when it comes to the future of MLM's and
1:27
machine learning and just data and how we store the data, how
1:31
we query the data. Especially his specialty has
1:36
been the unstructured data world and the what he likes to refer
1:41
to as the complex data world and how you can make that much
1:45
easier than just having some metadata.
1:48
That points to an S3 bucket, and that's really what they're
1:50
trying to do at Daft. From my understanding of it,
1:53
it's much faster and it's much easier and it's much more
1:56
robust. O, let's jump into the
1:58
conversation with Amy right now. By the way, if you enjoy this,
2:04
it would mean the world to me if you send this episode to one
2:08
friend so they can share in this joy with all of us.
2:12
Let's get into it. Here's Sammy Sidhu. you here.
2:24
Of course you didn't think you were going to come on here
2:27
without me talking about California raisins, huh?
2:31
No, that's true. Yeah.
2:35
So I'm Sammy. I grew up on a Raisin farm and
2:39
now I work in complex data. Not just any Raisin farm though.
2:43
Let's talk about this because there's a lot of people who
2:45
probably that are listening that don't understand the cultural
2:49
significance of the Raisin farm that you grew up on, which is,
2:53
I'm pretty sure a lot of millennials and people our age.
2:59
Had your raisins in their lunch box at lunchtime and it was just
3:04
a little box of raisins that probably came with like 20
3:07
raisins and they called, they said on the box, I remember
3:11
distinctly, right, it was this small box, I think it was red
3:14
and it said California raisins on them.
3:17
It is. So I grew up on a farm that was
3:19
one of the producers first said made it.
3:21
And the box I think is best for to most people where it's the
3:25
lady putting the tray of raisins out into the sun that was and
3:29
that's exactly what we did. We would grow grapes on a
3:31
vineyard and cut them down and then make them into raisins.
3:34
And the cool thing is that like no matter where I go in the
3:37
world, I owe them connecting with people over submit reasons.
3:41
I was in Korea for a conference and I mentioned it at a dinner
3:45
and everyone's like, oh, I had that all the time.
3:47
I was a little kid. Two.
3:48
Ohh. So it's always something that
3:50
connects with people. They monopolised the Raisin
3:53
industry, huh? No, it's interesting because
3:55
it's actually a cooperative. So there's actually a bunch of
3:58
farms in California that get together at solar raisins
4:00
together. Ohh So what is the process of
4:04
creating raisins? I know it's dried grapes right?
4:07
It's dried grapes, but the thing was sunrise insides made by the
4:11
sun. So generally what you do is you
4:14
have a table grapes that you grow on a vineyard and then
4:17
would you end up doing is when they're ripe, when the sugar
4:19
level. At a certain level you cut them
4:22
down from the vine and you throw out a piece of paper on the
4:25
ground. And then what you end up doing
4:27
is you kind of just like rotate it every few days until it's
4:30
into raisins and then you throw it into a big hopper.
4:33
So you grew U doing that. You would know when the Raisin
4:36
season was and when the harvest was and all that, and I imagine
4:40
you were cutting stuff down. Yeah, my parents were at work,
4:44
so my summers were cutting raisins and rebuilding tractors.
4:49
So how did that take you into complex data?
4:53
What the hell was that trajectory like?
4:56
Yeah, it's a good one. Let's see.
4:59
Gravida Farm. I obviously like getting my
5:01
hands dirty, so I worked on a lot of tractors with my dad.
5:06
Fixed everything around the farm and got into like electronics,
5:09
computers. Very early cause of that.
5:12
My parents noticed this and got me out of the farm because, you
5:15
know, I was pretty much doing nothing at school at that point
5:18
and the Central Valley. And so I moved to the Bay Area
5:21
and over here I just got hooked on computers, ended up staying
5:25
local, going to Berkeley, and ended up focusing a lot on.
5:31
High performance computing, low level programming stuff and then
5:34
get bit by the math bug. God's machine learning and my
5:38
professor who I worked under in college started a company.
5:43
And that company worked essentially on building
5:46
autopilot for everyone else. And I ended up being the first
5:49
hire there and eventually the CTO.
5:51
And that's kind of how my journey got started working.
5:56
Yeah, you said autopilot for everyone else.
5:59
What does that mean? Yes.
6:01
Well, at the time. While we were trying to build
6:06
it. So if you think about a toddler
6:08
strive, there's like multiple meanings. 1 is what we call like
6:11
level 2 Level 3 driving which is stuff like hands off, hands off
6:16
driving where like the car will drive itself on like a highway
6:19
and then you have like things that are like level 4 level 5
6:22
which will be called as eyes offer minds off, mind off which
6:25
is essentially like you don't have to pay attention or even
6:27
look at the road. So well we were building for
6:29
autopilot was kind of the level 2 Level 3 which is I can go onto
6:33
a highway it will drive itself and do its thing.
6:36
And the way the market we were targeting is for everyone else.
6:40
One thing we notice was that. Around 2018-2019 is a most
6:47
manufacturers already bundling a camera in their vehicle.
6:52
To do things like parking assessed or like automated
6:55
cruise control already and I think that we could do is by
6:59
making models that we are really good at making small putting
7:02
them into in the in the car that and essentially running it on
7:06
potato hardware. We could actually unlock all
7:09
this functionality for self driving without actually adding
7:12
any hardware for the customer. Or some challenges that you had
7:16
to deal with? Ohh.
7:18
You're essentially writing these really complex deep learning
7:21
models on potatoes. It's really hard to do and think
7:25
about. The computing power was
7:26
essentially like, let's say a pen or half 1/10 or 1/4 of a
7:31
Raspberry Pi, and what we had to do is get all the data that we
7:34
would collect. We would have to then transform
7:37
it into something that we can train models on and then we
7:40
would train these like mega models and then shrink them down
7:43
to run real time on tiny hardware.
7:47
How would you shrink them down? What were some techniques and
7:50
tricks that you learned from years of practise, I imagine?
7:54
Yeah, it's interesting stuff. There's things like, you know.
8:00
1st is data quality. If you are creating the model on
8:05
a lot unloaded data where some of them disagree or not are not
8:09
consistent, the model has to be bigger to kind of like see
8:12
through that. But well one thing we notice is
8:14
that if I actually just get data and clean it very well, you
8:17
actually need less model capacity to represent your data
8:20
set really well. So just really clean data was
8:22
one Ave the other ones were a little bit more exotic which one
8:27
of the things we did is I have a paper on this.
8:30
I was going to ask. Knew there was gonna be a taper
8:32
somewhere in this story. Yeah, and it is it, it is.
8:37
We did have to. A lot of the technology wasn't
8:39
there yet, and so we had to develop a lot of it ourselves.
8:42
But I had a paper called Squeezed Ass and the idea is
8:45
that we use a network assisted search to essentially find the
8:50
best network for the hardware. So what we did is we used the
8:54
neural network to then find the optimal network for the maximum
8:58
accuracy and the lowest latency for a given hardware set.
9:02
And that helped a lot. I I wanna know a little bit
9:06
before we move on about you became CTO of this company and
9:11
you were doing the very much ML stuff.
9:13
You also had that low level. Distributed computing aspect of
9:18
it, I guess. And so when you were looking at
9:22
the systems, what were some of the other pieces that you were
9:26
factoring in when you were architecting these smaller
9:30
models on very very dumb or tiny?
9:33
I guess the proper word is tiny ML and the half Raspberry Pi or
9:38
the 10th of a Raspberry Pi. What else were you thinking
9:43
about as you were deploying these onto the cars for everyone
9:47
else? Yeah.
9:50
There's, there's a lot of different things I I would say
9:52
like the important thing with the whole process, right, you
9:55
have these models and the thing that we are optimising for was
9:58
iteration speed. So what that means is producing
10:02
new models quickly dependent on orders that happened.
10:06
And so we think about that One of things we need to be able to
10:08
do is produce these models quickly on these large training
10:13
clusters. So we were one of the early
10:14
people who did distributed training and the thing that was
10:18
really interesting there is when you start making these models.
10:21
Dollar. You end up.
10:24
Stress the system in weird ways. So one thing that would happen
10:27
is because we're training these small models, our GPU.
10:32
We're running this pretty efficiently, but the amount of
10:34
data you had to ingest for these models increase exponentially.
10:39
Because think about if you have like a large model, the amount
10:42
of GPU to CPU work is quite balanced, but once you start
10:46
freaking that model, the CPU work doesn't go away.
10:49
But the GPU work is now getting faster and faster and so we
10:52
ended up getting bottleneck in all these weird places for
10:55
training and we had to like kind of reinvent the wheel to to kind
10:58
of make that work. And then they just kept on going
11:03
like we got to just running into weird issues like when it came
11:06
to model deployment. But we noticed that some of
11:08
these platforms that we were targeting were invented like a
11:12
10 years ago as an accelerator. And what we had to do is that if
11:17
we ran it on just the CPU for this, you know these hardware
11:20
that you put in the car and these hardwares or you could
11:24
think of them as like these platforms that are made by like
11:27
a company called Renaissance or NXP, these companies that you
11:30
pretty much have never heard of, but they have.
11:33
Monopoly in the car Computer chip space, yeah.
11:39
But The thing is that sucked about them is that they're kind
11:41
of designed by committee. And so they just have a bunch of
11:44
random shit in there. Like they'll be like, oh, we
11:46
have this one accelerator for these operations, and we have
11:50
this other accelerator for these other operations.
11:53
And so we ended up building a compiler to be like, OK, here's
11:56
an input model and then kind of like divvy up your model onto
12:00
these other chips. So we had to do design models
12:03
that had layers that were easily mappable to multiple parts of
12:07
the hardware. Him.
12:10
So you had to do all kinds of janky shit?
12:13
Yeah, it it was weird. It was weird.
12:15
And then I think the worst thing we had to do is one of the
12:19
letters we worked with had a chip that the compiler was only
12:24
in Windows. And that was probably like the
12:26
worst month of my life. And I'm guessing this is a lot
12:30
of. Cutting through regulation too
12:33
because or the chips were like this because they had to go
12:36
through rigorous regulation red tape.
12:40
It is true, yeah. And that is the case.
12:42
So typically when we talk about self driving hardware, they have
12:46
different levels of what we call ASAL.
12:48
And the idea is like how safe is this chip?
12:52
And so a lot of these chips have like failovers and redundancies
12:56
such that like if something breaks then it can still
12:58
operating as normal. Yeah, makes sense so.
13:02
Inevitably you probably created a lot of stuff while you were
13:06
there and is that what made you see the light and realise that
13:10
hey, you know what, with this complex data stuff there's
13:13
probably a few businesses that you could create around it.
13:15
I know that you you did this and then you went and you had
13:19
another stint in self driving cars, right?
13:21
And then you started daft. Yeah, it's a it's an interesting
13:27
history. So I I was at this company
13:29
called Deep Scale and then we actually got acquired by Tesla.
13:33
And so I worked through that whole acquisition process and
13:36
after Tesla I then went to left to kind of lead a lot of their
13:41
self driving efforts around perception.
13:43
And then that team got a card by Toyota, so with the Toyota for a
13:47
bit and then after that, you know after four companies of
13:50
essentially building complex data systems and being the
13:54
builder of these systems and user of them, I was like, OK.
13:58
Other people need tools that I kind of had, I I hadn't were
14:02
great, but once you leave you don't really have anything.
14:05
And So what we wanted to do is kind of bring the functionality
14:09
that you're familiar with like a database or data warehouse for
14:11
Talbot data, but to the complex domain.
14:14
So you inevitably thought deeply about if the world needs another
14:19
database. What made you decide and explain
14:22
more about what you're creating with DAF?
14:25
Yeah, for sure. I I guess like.
14:29
I'll talk a little bit about what we see as the the common
14:33
issue right now. What what if you are dealing
14:35
with complex setup. So if you're a company dealing
14:37
with complex data, well I most often saw and the systems that I
14:41
initially built were kind of bringing together the best of
14:46
relational with these kind of batch processing engines.
14:48
So imagine you are a self driving company or a biotech
14:52
company. You have a bunch of things like
14:55
images, videos, 3D scans. What?
14:59
You would intuitively build to kind of build an engine end to
15:01
end to process these images is getting a traditional database
15:05
or data warehouse, putting all the metadata in there and when
15:09
it came to the assets like the video or image, you would just
15:13
have like a pointer to some remote storage like streaming.
15:16
So you would put a bunch of images and three and have a
15:19
database with all the metadata. And what would then happen is
15:23
when I wanted to process this I would have to do three different
15:26
steps. I would first have a data
15:28
analyst SQL query to be like what data should I process them?
15:32
Then you would run that query and you get a list of files and
15:36
then you do something like spark to then process them and then
15:39
give you the results where you then dump the three or dump the
15:41
results back into a SQL table. Then you would finally get
15:44
another data analyst to actually process the end results and give
15:49
you the analytics that you cared about.
15:51
And what we saw was like these really nasty systems that had
15:55
talked to three different teams to actually do this end to end.
15:58
And I was always like OK first, if we're running a query we
16:01
can't optimise the end to end and #2.
16:04
Dealing with Spark with images or videos or 3D scans was like a
16:09
nightmare. Ohh what would happen is that
16:12
Spark doesn't have the right abstractions here to represent
16:14
the data natively and what would happen is that.
16:18
You would have one. Machine get too much memory from
16:23
having all these images and memory and it would boom or out
16:26
of memory and then now would die and they would rebalance and
16:30
they would just knock out the rest of your cluster.
16:32
Like Domino's. Domino's.
16:34
Yeah yeah. And I always just like killed me
16:38
going through like 10,000 lion log files of like Java umm
16:42
errors and like fighting what was the one line that caused my
16:45
bug. So I think I could spend
16:47
collectively weeks of my life on this.
16:49
So painful. Yeah, it's so painful.
16:53
And I always thought like, hey, you know what?
16:54
Like we had a system that was like natively understood that
16:57
this is a 2 gigabyte file and don't just load it like dumbly
17:01
in memory. That would make my life so much
17:03
easier. So after all my senses are
17:06
driving, I kind of talked to a few company.
17:08
It's about like, hey, if I work here, what do you want me to
17:11
work on? And four of the companies that I
17:13
talked to out of five said, hey, we would want you to handle this
17:17
whole side of unstructured data processing.
17:20
And I was like, oh, this sounds like a good startup idea.
17:23
And then I I paired up with someone I worked with at that
17:26
Level 5. I left Level 5 and he was also
17:29
very passionate in space and we started the company and
17:31
eventual. Isn't, and our first product
17:35
that we worked on, our we're working on is called Daft, which
17:38
is an open source query engine for all those types of data
17:41
where it understands it very intimately.
17:44
Hmm, and the idea is that you can load in all of this data and
17:48
then you can query it easily and it's the metadata is with the
17:54
unstructured images and. That is my understanding.
17:59
Is that correct? Yeah, yeah.
18:01
So it it's kind of like I would say.
18:04
It's a distributed data frame that's all Pythonic.
18:07
But the entire engines ran out in Rust.
18:09
And what it looks like to the user is something like a pandas
18:12
data frame where I can say I'm gonna read all these, read these
18:16
millions of files in S3 as like read files.
18:19
I want to then process them as an image.
18:21
I want to crop them, resize them, and then I want to load
18:24
them into a machine learning model for either inference or
18:28
training. And what it looks like to you is
18:30
you're just building this query. Using a lazy API and when it
18:33
comes to execute it will actually run it on a cluster
18:37
using this query plan that it develops, but then also
18:39
intimately understands the data that's running on.
18:41
So no more umm errors and very optimal queries for complex
18:45
data. That's awesome.
18:48
So then. This is daft.
18:51
Eventual is what? So eventual, so we want to be a
18:57
lot more than managed staffed. So what we're working on is we
19:02
have plenty, we have users, so most of the users of data right
19:04
now are actually in the enterprise space.
19:06
So they're right now using open source staff to do a lot of
19:10
query processing a bunch structured data.
19:13
Part that the question we get after how do I how do I run
19:18
these queries on complex data or a structured data is how should
19:21
I store it for better retrieval. So the next product that we're
19:25
working on is a so will be open source, but a managed enterprise
19:29
managed version of this which is how do I store my data
19:33
efficiently for complex data. So think of it as a complex
19:36
catalogue for everything unstructured.
19:39
And it differs from just that three bucket in what ways?
19:45
So if you're familiar with like a data warehouse or a database,
19:48
you get all these amazing features like governance, like
19:52
schema evolution, like time travel.
19:55
But there is none of that for, for example, a bucket of images
19:59
in S3. And So what we're trying to do
20:01
is give you the things that you're familiar with Bigquery,
20:04
Ohh or or dinner or large data warehouses, but for unstructured
20:08
data. So think of it as I want these
20:10
teams to have these permissions and I wanna add this new column
20:13
or I want to delete this column or I want to have a retainment
20:16
policy. These are things that we're
20:18
adding on top of. Just plain old S3.
20:22
Also much faster loading. Rather than just a bunch of
20:24
single images, we can actually compact it into something that's
20:27
really easy to load in. Ohh yeah how's that?
20:31
So there's some interesting formats you can do.
20:33
So what we see right now is people stuffing a bunch of
20:38
images and Parquet which isn't really the best, it's great for
20:41
tabular but isn't necessarily the best format for images.
20:44
So for a lot of us, we're all we're developing our own
20:47
container formats to to be able to load data really efficiently
20:51
for images or you know seek to certain parts of the image from
20:54
cloud storage directly. Ohh nice.
20:56
So inevitably man, I'm sure everyone is asking you about.
21:02
How you fit in in this large language model world and
21:07
unstructured data is so hot right now.
21:11
What is your thesis or how do you look at it when it comes to
21:16
whether it's whatever text to image or?
21:21
Text to animation, text to video?
21:24
Any of that, or just straight generative AI with a open AI
21:30
call? How are you seeing?
21:34
What you're doing at eventual laying into this like new
21:38
paradigm of machine learning and if at all, because maybe you're
21:42
like, yeah, right now we're focusing on this slice and it's
21:46
not actually that important to us to get distracted with the
21:49
shiny new toys. Yeah.
21:53
I mean that that's a good question.
21:54
I would say a lot of what we're building is compatible with the
21:58
future of LMS. So some of the things, some of the risks we're
22:02
working on with some of our early users are around LLM's and
22:08
generative AI. So a big one is actually
22:12
retrieval. And so are you familiar with
22:15
chain of thought for alms? Yeah, dude.
22:18
So we talked about the exhibit example earlier.
22:22
There's this. Great.
22:24
Paper from Alex Radner and a lot of other people that I can't
22:29
remember who, but he was talking about, distilled step by step,
22:33
and it's basically distilling them all.
22:35
Have you seen that paper? Yeah, yeah.
22:38
OK, you know about it, but I'll explain it for the listeners in
22:41
case they missed it. It's distilled step-by-step.
22:44
Basically, is asking, it's distilling the models, but when
22:48
they distil it through the large language model, they're asking
22:51
for this chain of thought reasoning.
22:53
And so it makes it much easier for you to get that distilled
22:58
model and train it with less data, because the metadata is so
23:02
rich from the chain of thought, reasoning prompting, or chain of
23:07
thought prompting that you get. From the large language model
23:09
when you're creating that training data for the distilled
23:14
model. Yeah, it's it's super cool
23:17
stuff. It's it's crazy to me how
23:19
different distillation is now compared to distillation when we
23:22
used to do it. Back in your day.
23:25
Back in my day, distillation was like kind of was kind of dumb
23:28
like what you would do is you would trade a big model and then
23:32
you would just train the essentially like you would use
23:36
that output as the ground truth for a smaller model.
23:39
And that was it. And then, yeah, I don't know, it
23:44
was just like, Ohh, maybe the big model understands the labels
23:47
better than the the ground truth.
23:48
And then I don't know. But anyway.
23:51
Come a long way from there. Anyway, I distracted you.
23:55
Tell me about why you were talking about distillation in
23:57
the first place. Yeah, so the way I think about
24:01
train of thought reasoning is it's quite compatible with this.
24:03
So one of the use cases we're doing now is.
24:07
Being able to process data like the rebajas data set, being able
24:11
to load it in philtre it and then do things like
24:13
tokenization, do things like run and open source model on it and
24:17
then doing things like I have a set of embeddings and I wanna do
24:20
this hybrid search over a large data set of find me.
24:26
You know, sources of text that were published in this date have
24:30
this type of style and then have similarity closer to this.
24:33
That's kind of like the first level use case.
24:35
The next level use case is actually plugging DAF straight
24:38
into the Elms and having if I asked the LLM hey, give me some
24:42
documents that show me. I don't know that are written in
24:49
the style of I don't know what's what's your favourite author,
24:51
Hunter Thompson. OK Hunter Thompson And so The
24:56
thing is for a lot of these models, we don't really quite
24:59
have things precomputed and we don't necessarily know what the
25:01
other model query looks like. And so we can actually do is
25:05
actually plug and DAP as one of the query sources for LLM.
25:09
So think of it like, you know we've seen all these demos of
25:12
LLM's writing SQL to then send it off to a database and
25:16
retrieve it and then parse the results.
25:18
We can do the exact same thing for complex data, which is right
25:20
have LMS right. These queries search over these
25:23
large corpuses. On S3 or these dead legs, bring
25:27
back the results and then have the alums interpret it.
25:29
Interesting. Wait, say that again.
25:31
I'm not sure if I fully grasp what you were saying there.
25:34
Ohh what I'm trying to say is like just how we can provide a
25:39
SQL engine for Elms today to understand some of your tabular
25:43
data. We could do the exact same thing
25:44
for complex data, which is I can have my data like full of
25:47
multimodal complex data and have the LLM writing daft queries to
25:52
then try to understand them better.
25:55
Ohh. Dude, that is awesome.
25:59
OK, I see what you're saying. So then these are some of the
26:03
large language model use cases I imagine.
26:07
Four out of those five dentists that you talked to earlier or
26:11
whatever they were when you went out to get a new job before
26:15
starting eventual and they said hey, we we want you to work on
26:19
this use case and this problem area.
26:22
They weren't necessarily doing things with LMS back in those
26:26
days when you started eventual and so there's a tonne of other
26:30
use cases. Is it very much like where you
26:32
see it soar, where you're seeing daft really take off is?
26:37
In your. Past life kind of world where
26:42
it's the self driving car world and I know you mentioned also
26:46
drug discovery, is that another use case area that you see it
26:50
taken off? Yeah.
26:52
So the the area that we see it taking off and the areas we're
26:55
focusing in are essentially the domains that don't get much
26:59
love. And so kind of, you know, trying
27:02
to, yeah. And so right now if you're
27:03
trying to process images, there's a bunch of different
27:05
tools that sort of work. And so for that it's kind of
27:10
crowded. But what we actually have talked
27:12
a lot of our users, if I'm trying to build a way to build a
27:17
queries on audio data and read all these different audio
27:19
sources, transcribe them and then do all these different
27:23
various analytics and filtering, there's not really a good tool
27:26
for that. And So what we really lean into
27:28
is making sure we work really well for these domains.
27:31
Don't get quite much love and so that seems like audio 3D assets
27:36
like game assets. Microscopy images from biotech,
27:40
which is like these. These large lossless images that
27:44
are hard for Python libraries typically interpret.
27:47
Yeah, and also just weird wacky things.
27:50
Like A big use case of ours is someone sets up a Kafka stream
27:55
and they dump a bunch of these proto buffs.
27:57
It has three. And then now they're like, hey,
28:00
how do I just philtre over a billion roto buffs for things
28:05
that have this in their field and defrost great for that.
28:09
Ohh, nice. OK so.
28:12
You could potentially work incredible for playing around
28:16
and querying a lot of MLB's community podcasts.
28:21
That's a great use case. Like if there's like a a bunch
28:24
of sources or URL's for the podcast and we can pull them in,
28:28
transcribe them, chunk up the text and then.
28:32
You know, and then find out, interestingly, you said and then
28:34
maybe train a model to fine tune it on your voice and replace
28:37
you. Test.
28:38
Then I don't even have to be here.
28:40
I can just have you interview the fake me.
28:43
Maybe I'm not even real right now, man.
28:45
Who knows? Yeah, so that's, yeah, we'll
28:48
have to do some kind of hackathon on that later because
28:51
I think that would be super fun to pull all that data.
28:54
And also it sounds like you're making it really easy compared
28:58
to what I had thought it would we would have to do.
29:04
Yeah, it's really easy. I mean you could spin it up in
29:06
colop and then get the query working and then it's a one line
29:10
switch to switch it from running locally to running on a
29:13
distributed cluster. Nice.
29:15
Ohh, that's awesome. So now what are some things that
29:18
you took from your self driving days and all this tiny ML stuff
29:23
that you were doing and you're now bringing into daft or
29:27
eventual? Yeah, I think there's a couple
29:31
of concepts that I think are really important.
29:33
I think one thing that I took away early was.
29:37
If you focus on traditional data systems and taller space, the
29:41
individual rows don't really matter.
29:43
It's about the analytics result right?
29:46
But in complex data and self driving, the individual rows
29:50
matter. Hey, Laszlo here.
29:53
If you're serious about Emma Loops, you hit subscribe right
29:56
now. So it's like if I'm trying to
30:03
search for failure cases, the individual results matter, not
30:06
the analytic part of it. And so it's almost like usable
30:10
system, that's like hybrid transactional analytics, but
30:14
you're not really doing transactions.
30:15
So one thing we both adopt early is how do we make sure like
30:19
needle in the haystack queries are really performant and easy
30:22
to do. Yeah, that makes hundred sense
30:27
so. It feels like, and this again
30:31
going back to that whole idea of the self driving cars, it feels
30:34
like. I know we mentioned it
30:36
beforehand. It's one of those high risk or?
30:43
Very high. What?
30:45
There's a special word for it that the EU uses to classify the
30:49
MLS use case? I can't remember.
30:50
It's something high. Damn, I can't remember but.
30:56
It basically it's dangerous because there's lives that are
31:00
potentially affected in very. Bad ways if shit goes off the
31:06
rails and so making these needle and haystack use cases or.
31:13
Cases. Able to you're able to find them
31:17
really quickly in Dave seems like yeah it's a no brainer.
31:20
You came from that world and now you're seeing, hey, how can I
31:24
make that very useful? Even if it isn't with this high
31:30
danger situation. Yeah, yeah, the needle in the
31:34
haystack type queries. We see him a lot now and and
31:36
indeed generate AI space as well, which is find me this
31:39
document that exhibits XYZ and you end up searching.
31:43
Like you're not saying how many documents are there that
31:46
exhibits would say, you're saying give me the documents
31:49
that exhibit this. So we we see this pattern quite
31:53
often now where you want kind of the best about the world and So
31:56
what we're doing that and for the whole safety thing, it's
31:59
quite interesting because I'm seeing a lot of similarities
32:01
between. The self-driving domain and
32:04
thinking about safety and the whole stuff that's happening
32:09
now, like I would say for any AI you develop, it's important to
32:13
essentially have a safety net so we can't do harm.
32:15
And the way you think about how rigorous action has to be is
32:20
what can it do. So when you think about self
32:22
driving, what we would do is, you know, you need a way for
32:26
something to take over. And so we would have the self
32:29
driving stock. You would have a safety layer
32:31
and software to prevent things like.
32:34
You know hitting the curb or hitting a person that was
32:37
outside of the traditional like system that would be running
32:40
for, you know the the end to end automation.
32:42
But then you also have a. Safety driver who would take
32:46
over a minute, start a second service so that they would be,
32:49
you know, hand hovering the wheel and grab the car and
32:52
disengage whenever something, what happens?
32:54
Yeah. And the the way that any of
32:58
these aids are developed, including, you know, the ones
33:00
that we see now is you have feedback loops, you you
33:02
essentially put something out there, you observe how it does,
33:06
you see the failures and then you improve it.
33:08
Where As for self driving that they'd let the business very
33:11
expensive because you need a human to literally constantly
33:14
monitor it so all the feedback, all the validation you're
33:17
getting is supervised by a human.
33:20
But for the whole Geni geni AI space, it's quite interesting
33:24
because I put them in two domains.
33:26
One domain is. Where it's OK to fail.
33:30
And I see that in a lot of creative applications where if
33:34
I'm doing copywriting or image generation, yeah, exactly.
33:39
Any of those, there's not really a right answer and there's no
33:43
right answer to fail. And so it's OK, right?
33:46
There's as a use case where you can put out there and your
33:50
feedback we're getting from users is what images are
33:52
actually clicking and actually downloading based on what they
33:56
generate. But then other domains, like
33:58
let's say. The AI is you still have that
34:01
feedback loop, but then they have to be reviewed by lawyers.
34:04
But each in each time that they're being run it's gonna
34:07
improve over time. So I kind of view it analogous
34:09
but a little bit different. I remember the word, the high,
34:14
high stakes. That's high stakes.
34:16
Yeah, dude, I can't believe I was totally blinking on that
34:19
one. So going back to what you're
34:21
saying though, 100%, I fully agree with you on that.
34:24
Where there are these? Expensive feedback loops, but
34:29
you also want to make sure that there is some kind of safety net
34:32
just in case and it feels like. There's these use cases with
34:38
generative AI where you don't. Need safety Nets per se, because
34:45
the worst that can happen is you get a deformed face on an image
34:48
that you generate, or you get a copyrighted image that you
34:52
generate and you got to figure that out and make sure that.
34:56
You just generate something better and you become the
34:59
curator and whatnot. Or when it's.
35:03
Generating text, then you can add your flavour to it and
35:08
there's not really any high stakes there.
35:10
But when it comes to more of this.
35:15
Legal documentation or if we're talking about I, I mean there's
35:19
there's a few different high stakes potential here that you
35:24
do need a bit of a safety net. Yeah, I agree.
35:28
Like. That's the thing I feel like it
35:31
needs to be understood better. I feel like a lot I I see a lot
35:35
of companies or individuals starting these projects that
35:37
they are great. But you know, for example, legal
35:39
AI is like, I think I see a future where it works with the
35:42
lawyers are using it, but at the same time I pay my lawyers a lot
35:45
of money. Because they don't make
35:47
mistakes. Sure.
35:49
Otherwise I wouldn't do myself. You know what I mean?
35:51
Yeah. Yeah, exactly.
35:53
I I know how to prompt this. Yeah, whatever.
35:56
Harvey came out of that. So I can figure that out on my
36:00
own. I yeah, I understand what you're
36:02
saying on that. And and then coming back to what
36:06
you're doing with Daft and how you're thinking about that.
36:09
How does that tie in? Yeah, so a lot of these use
36:13
cases like for example Harvey might do, is they have these
36:17
corpuses of legal documents and they want to be able to get
36:21
these documents, use image processing on them, extract out
36:25
image, extract out text, and then run them through LMS. Those
36:30
are things that that would be a really, really good tool.
36:32
What we see right now in in the the general AI space is that a
36:36
lot of people are just writing Python scripts that go over each
36:40
file, extract the text manually and then hit the Open AI API and
36:46
they kind of dumped that either in a file or like one of these
36:48
vector databases. And then kind of the process
36:51
completely restart once they switch the model or switch the
36:54
kind of thing that they're doing.
36:55
And so we want to make that really easy to do, which is I
36:58
can build my entire pipeline just using a data frame like I
37:01
would do. Pandas and when you're ready to
37:02
commercialise it or make it production ready, you just
37:05
switch a couple of flags and then you can run this on a
37:07
cluster so. It's it makes their life a lot
37:11
easier. That's what I would say.
37:12
Yeah ohh dude. I see the vision, then I see and
37:16
I understand that. So one thing that I also wanted
37:20
to mention and bring U is. And you kind of hinted at this
37:24
before? Is around.
37:30
How you think about the needs and?
37:35
Tradeoffs when it comes to tabular data versus this
37:40
unstructured or complex data and certain things that you need in
37:44
tabular data and other things that you need in complex data
37:49
and how you look at the two architectures.
37:52
And if you're setting up a system right now and you only
37:55
need to go down 1 route, maybe it's the tabular data route,
37:58
then what are you going to be building for and optimising for?
38:01
And then if you have complex data, what are some things that
38:04
you're keeping in mind and optimising for?
38:07
Yeah, that's a that's a good question.
38:11
I guess like talking about. The.
38:15
Like the the axioms of both, I think is really important.
38:18
So right when we talk about tabular data, you typically
38:21
think like. Integers, floats, you know
38:25
things like strings. And most of the time when you
38:28
have this data, the queries you're running are usually
38:32
aggregations. So what that means is that I
38:34
have like these files of text, they could be large files of
38:37
text and what I'm essentially doing is going through each row
38:42
and then doing things like min, Max, sum.
38:44
These very cheap operations to compute.
38:46
So if you think about it as what is the volume of data I have and
38:50
the volume of compute, it's actually much more data heavy.
38:54
So it's like I might have gigabytes of data, but I only
38:56
have billions of operations, so they're quite balanced.
39:00
Well I've got an image, images and things like text for LM is
39:04
completely different. Whereas if I have like a one KB
39:08
or sorry like 100 KB image. I am processing potentially
39:12
trolling the operations on it. Right.
39:16
Or if I have like let's say a work of Shakespeare which is
39:19
only like a MB, I will be processing, you know, queen
39:22
trillions of operations using MLM, you shouldn't.
39:26
And so the ratios that we think about of compute over the data
39:30
is completely different. So when you're buying callback
39:32
systems, you have to think about it as I will be compute bound
39:35
almost every single time. Where I was like, if you think
39:40
about traditional query engines, you're like OK, all they're
39:42
doing is optimising or you know, reading from S3 and they don't
39:47
really bother about the compute. Which is fair because they'll be
39:51
like that. Ohh.
39:58
So then what are some? OK, I and I love you break it
40:01
down by the axioms and you think about, hey, let's look at the
40:06
fundamentals of one versus the fundamentals of another and what
40:10
the bottlenecks are going to be and what you're going to
40:12
encounter. And so then as you're building
40:14
systems around this, what are some things that you would?
40:17
Say I definitely want to have in my toolkit if I'm dealing with
40:21
one versus the other. Yeah.
40:26
I would say if I was building an allergist engine, I would go for
40:28
something simple and something that failures are handled in,
40:33
you know, in a way where we can recover easily.
40:35
So for example if you're running things on like a spark cluster
40:40
and you, it's fine because the amount of work you have to
40:43
reproduce is not bad, OK. But if you talk about that in
40:46
terms of like complex data cluster, that does not work very
40:49
well. And So what we're building
40:51
around that is. First making sure DAPT
40:54
understands the data types natively so they understand what
40:57
an image is and how how expensive it is it to bring into
41:00
memory, How expensive is it to centred around and also for
41:04
these various other formats as well.
41:06
Also understanding that placement and scheduling is.
41:10
Very important and we require really high utilisation, so an
41:14
example is if you get a lot of these complex data workflows.
41:19
And map it to something like a Spark cluster.
41:21
You only use about 20% of the hardware.
41:25
Tim. And so if you're using GPUs or,
41:27
you know, running LMS or a computer vision model on this
41:30
data, you're pretty much burning five times more money than you
41:34
have to. And so some of things that we're
41:38
doing here is saying, OK, you know what, Our users are
41:40
typically enterprise companies who want to save money and by
41:44
putting things into DAF, they can run things that's Pythonic
41:48
using data frames, but then still get really good
41:50
utilisation and cheap throughput essentially for their whole
41:54
system. And so yeah, the things we're
41:56
designing around there once again are.
41:59
Really understanding the data we're working with having query
42:03
plans that are necessarily not optimised for.
42:06
Table analytics, but for comic data processing and actually
42:10
being really tied into the hardware running on.
42:13
So really understanding the AWS machines we run on and how to
42:17
get the most out of them built into the framework.
42:19
I wanted to go down 1 route but then when you said that last
42:24
part it's it's like ohh I I thought about something else,
42:28
something else got triggered and you have this background in tiny
42:31
ML and how does that play into things and that like with Daft,
42:37
what is it optimising for or how are you thinking about?
42:42
Those kind of. Use cases.
42:46
It's pretty interesting. So.
42:53
So I I guess one thing I'll go into first is what what's the
42:56
eventual goal of Daft, Right. No pun intended there, so.
43:02
The the way that we're building out data is right now, although
43:07
we were. So our user interface right now
43:09
is in Python And all of the engine and stuff is a hybrid
43:13
between Python And REST. As we're building more and more,
43:16
we're getting to a mode where we can run completely serverless,
43:19
where you can run things in Python on your laptop, but then
43:23
when it executes, it can completely route out state in a
43:26
cloud cluster and then spin down as soon as it's done.
43:30
And. One of the difficult things with
43:32
that with you know models is that most of the models right
43:36
now require Python. So you typically use Pytorch or
43:39
Tensorflow, and you embed that into Daft or typical typical
43:44
workloads using Python And do a bunch of this glue code.
43:48
But one observation that we've noticed is that.
43:51
You know, although the beginning we wanted to make things really
43:53
easy for Python to build models and and run them in the app, we
43:58
noticed that a lot of people just run the same models.
44:01
And Daft, like they run the same, you know, Alan model,
44:04
they're in the same computer vision models and they might
44:07
change things like weights, but the models are generally there.
44:09
So one of the things that we're going to in the future, which is
44:12
where my background comes in a tiny ML, is how do we actually
44:15
embed models in DAF where we can run them, compile them down to
44:20
hardware, run them on CPU or GPU natively without any Python or
44:24
any framework. And then execute that very
44:28
efficiently on the cluster. And so that's one of the things
44:31
that we're working on, which is if I have something like a LAMA
44:33
model, we can actually just package that as part of Daft
44:37
running in the cluster and there's no work on your part you
44:39
really have to do. And then you can bring it down
44:45
to the running it on a potato as you said, and having this
44:49
capability. Just the out-of-the-box that
44:52
works. It works and like I think that's
44:56
it's it's getting really interesting because.
44:59
If we think about like the inference per dollar for a lot
45:02
of these models. GPU's are usually the most
45:05
efficient, right? Like in terms of a cost per
45:07
dollar? Embers per dollar.
45:09
But the problem is no one can get GPU's now.
45:12
There's a massive shortage, which is stuff that I dealt with
45:14
that when I was working at Lift. We would use all the GP's in a
45:17
single data centre and we had to learn a lot about how do we do
45:20
things. Differently, so we can actually
45:23
do our work. And so we're seeing that now
45:25
with this whole genre. Airwave is that.
45:28
I I, I'm, I'm in this slacker for a WS support and everyday
45:31
people were like can I get more GPU and the guys are like no.
45:36
Right. Is that we have to think about
45:38
alternatives. And So what do you do if you
45:40
need to get those? If you get your shit done and
45:42
you can't get GPU's, you need to adapt.
45:46
Yeah, yeah, yeah. You have to figure out what's
45:48
the, what's the. Way around this?
45:51
And how can we make it work without the GPUs that we have
45:55
been relying on this whole time? O what's the workaround?
46:01
Yeah, so I think a big one is, I mean have you seen all those
46:04
really cool work on like Llama CPP?
46:09
No, wait, is this the one where it's just super small Llama
46:13
models? Super small.
46:15
So does guy essentially got Facebooks Lama model and then?
46:19
He wrote, he did Whisper and then he did.
46:22
Yeah, yeah, you can run it on your computer, right?
46:24
Like you can run it on a CPU on your whatever laptop.
46:27
Yeah, exactly. And So what he did is made the
46:29
model smaller so if it's better in memory and then wrote a hyper
46:33
optimised version and purely and simply pause using vectorized
46:36
operations and whatnot for this one model and so.
46:41
When you compile with model, you essentially have a single binary
46:44
that does inference the model, and I kind of see that as you
46:48
know the future of these albums, which is.
46:52
You can package these standardised alarms that are
46:55
optimised for your hardware as part of your query and use it to
46:59
extract or do generation or whatever task you want.
47:03
Ohh, incredible. I mean, this is making the
47:05
assumption that I keep telling people.
47:09
That open source models are going to be good enough.
47:14
And right now they're not there. But the big assumption that
47:17
everybody is saying is like open source will be there in six
47:19
months. Don't worry about it.
47:20
It's going no matter what. Yeah.
47:24
I'm a big I'm I'm big on open source like I you know our whole
47:27
a whole company is an open source I I think open source
47:30
will take. I mean I I see I see an all a
47:32
lot of analogues that computer vision loss so.
47:36
In the I started doing computer vision on like 2012 and and
47:41
during that time the big bad computer vision algorithms were
47:44
all in companies. You had companies like Clarify,
47:47
you had Google, you had like Apple, they were doing all this
47:49
crazy computer vision. But then you would have these
47:53
open source research papers come out and then they would be a
47:55
little bit better and then the companies would surpass them.
47:57
And then we got to a point where.
47:59
People just kind of stopped caring because the open source
48:02
was just so good and now what people do is they just use them
48:05
as a black box for their tasks, and that's kind of what we see
48:08
now for computer vision. Yeah, yeah, yeah.
48:12
I mean, I hope it is like that. Don't get me wrong, I am a huge
48:17
proponent of the open source world doing its thing and
48:21
optimising it and making it free and open, of course.
48:26
I just, we haven't seen it yet. And a few of these attempts, I
48:31
try and play around with them and I'm like, dude, this isn't
48:33
six months behind. This is like 2 years behind.
48:38
I I would agree with that, like it's probably gonna be more like
48:41
years, like 2-1 to 2-3 years rather than months.
48:47
Because I'm in the Moat here once again is like a big cluster
48:50
training models on like it. You have a huge cluster where
48:53
you can train these models on, then I think you can do it, but
48:56
they're too good to the base level.
48:58
I think the next part is then making these fine-tuned datasets
49:01
of like Q&A like Open AI is done.
49:05
Yeah, 100%. I'm excited for it though.
49:10
I I'm a big, I'm a big permanent of open source and I I think it
49:14
will take off. Awesome, Sammy, This has been
49:19
fascinating, man. I love talking to you about all
49:21
of this, the history of where you came from, where you're
49:25
going, what you're doing. Last question I have for you is.
49:30
Over the years, you have inevitably succeeded on a few
49:35
things. You've failed on a few things.
49:38
Where do you feel like you succeeded, where others
49:42
typically have failed, and why do you think you succeeded?
49:48
Hmm, that's a good question. I feel like it's because I care
49:56
about things that people told me not to care about.
49:59
Ohh and I think the biggest one is when I started my career.
50:04
Is that? The advice I got from a lot of
50:08
software engineers was. Ohh, don't worry about making
50:12
those run fast or don't worry about going low level.
50:15
It's not worth it and the advice I got for them came from an era
50:20
when. You know computers got things
50:24
getting faster when you do know work.
50:25
So you can write shitty software and computers are just get
50:28
faster and then your code will be no no longer an issue.
50:31
Or the other thing I would say is that Oh yeah, computers are
50:34
much cheaper than software engineers.
50:37
Right, but then I would always feel my optimizer hat on and go
50:41
into the assembly, or go into the level and actually just
50:43
really understand why things were running the way they did.
50:46
And so our deep skill like that ended up paying off and at lift
50:50
and and Toyota ended up paying off.
50:51
And now I feel like that experience is really getting in
50:54
there, which is understanding why things are slow or why
50:58
things are fast because I did not listen to the advice people
51:01
gave me in the beginning of my career.
51:04
Incredible, man. This has been so cool.
51:07
Thank you so much for coming on here.
51:08
I think we'll end it there. Yeah.
51:11
Thanks for having