Latest / Agentic Conversations (formally mlops.community) / Small Data, Big Impact: The Story Behind DuckDB // Hannes Mühleisen & Jordan Tigani // #202

Transcript

Small Data, Big Impact: The Story Behind DuckDB // Hannes Mühleisen & Jordan Tigani // #202 · Agentic Conversations (formally mlops.community)

0:00
My name is Hannah Miller Isen. I'm the Co creator of Doctor B
0:03
and I'm also the co-founder and CEO of Duct B Labs.
0:07
And I take my coffee black. I'm I'm Jordan Tigani.
0:11
I'm the cofounder and chief duck herder at at Mother Duck, and I
0:17
destroyed my taste buds drinking really terrible coffee when I
0:20
worked at Microsoft. So I will basically drink any
0:24
any kind of coffee, even if it's been sitting around for for
0:26
hours. What's up, folks?
0:29
My name is Demetrius and you're listening to the ML OPS
0:32
Community Podcast. And before we jump into the
0:35
podcast today, I have a few musings of my own that I want to
0:38
get into. I love Mike.
0:41
I just absolutely devour mangoes.
0:43
And whenever I get a good, ripe mango, or whenever I'm in a
0:47
place that has mangoes, I am like a kid in a candy store.
0:53
They're delicious. But I have one thing that I'm
0:56
realising as I get older. I do not like about mangoes.
0:59
And maybe you all can relate. Or maybe you're just not old
1:02
enough and you can't Really. What's the deal when you're
1:07
eating the mangoes and the shit gets stuck in your teeth?
1:09
Man, I feel like I'm going to be one of those old men that has a
1:15
toothpick and is constantly picking at my teeth because I
1:19
eat too many damn mangoes and the fibres get stuck in my
1:23
teeth. But I'll be damned if I ever
1:27
stop eating mangoes because of these fibres.
1:30
So today we talked to the founder of Duck DB, the creator
1:36
of Duck DB Harness and the founder of Mother Duck.
1:41
Wow. What a conversation.
1:45
I absolutely enjoyed every second of this because we broke
1:51
down what the relationship is between Mother Duck and Jordan.
1:56
Got real technical on why he feels like this is a potential
2:03
model that others can take and what the advantages of are the
2:08
advantages of having a setup like this are.
2:13
And he also broke down what his main goals are when it comes to
2:18
creating Mother Duck. And then harness went into what
2:21
his focuses are as creating the open source project Duck DB.
2:27
And then we went. Of course we explained what
2:29
Doctor B is, why it's been garnering so much attention
2:32
recently, And if you have not heard about the project, I think
2:35
you're really gonna like this. Because the TLDR of Duck DB is
2:40
it's an open source, very fast, very developer friendly
2:44
databases and that's about all you need to know.
2:47
Go get your hands on it, play with it, honest.
2:50
Talk to us about creating a database that gives a developer
2:57
a magical experience and he talked about how he tries to
3:02
focus on that end developer's experience and what he felt may
3:07
made him be able to do it much more than other projects.
3:13
And I loved his rationale. And really it comes down to
3:16
this. Just care.
3:18
Care about those experiences. Care about your end user.
3:21
Have that empathy and make it not suck paraphrasing his words.
3:27
So I hope you enjoy this conversation with Jordan and
3:30
Jonas. And wow, I know I loved it.
3:43
All right, fellas, so here's the truth.
3:46
I took a mushroom blend of like 7 different kinds of mushrooms
3:51
that are supposed to make me smart, Tur.
3:53
And I did it just so I can keep up with you all in this
3:57
conversation today. It's great to have you both
3:59
here. There's no need for mushrooms,
4:02
but thank you very much, yes. Yeah, that sounds like it sounds
4:06
as somebody have to try. Yeah, I it is definitely not the
4:10
legal, yeah. Well, I live in Holland, so.
4:15
Exactly. Not for honest, huh?
4:17
Yeah, yeah. Everyone that has been listening
4:22
to this show for a while probably has heard me talk about
4:25
the swag that we created for all of our LLM's and production
4:29
conferences, which is just a shirt that says I hallucinate
4:34
more than ChatGPT. And I created that.
4:38
And then I didn't ever wear it, I was just sending it to people
4:42
and saying like yeah, use it, it's great, blah blah blah.
4:45
And then I got myself one and I started wearing it.
4:48
And then I started realising it definitely attracts attention.
4:54
What kind of attention? Yeah, for the people who know,
4:57
they laugh. For other people, it's like, why
4:59
are you like advertising that you you hallucinate and did?
5:02
You get some calls from law enforcement or something like
5:05
that, no? Yeah, I just tell them that I am
5:09
part of the medical studies that are happening these days.
5:12
So it's all sanctioned, right? And.
5:15
That's what's being a Founder is all about, is you hallucinate
5:18
the future and you, you know, you help, you know, usher it
5:20
into existence. That's it.
5:24
That is. That's my that's the story that
5:27
I am going to stick with from now on.
5:29
It's just I'm hallucinating. The better future for us all to
5:34
live and play in and you will thank me in a few years.
5:38
I mean the Greeks had this thing right, where in Delphi they, you
5:40
know, they got, they got this these quite high and then they
5:45
were hallucinating about the future.
5:46
So I mean, it isn't. There's like a lot of sort of
5:48
track record on that. And that's where a famous,
5:50
famous database company got their name.
5:53
Ohh yeah the Oracle. No, the Oracle Delphi, Ohh,
5:59
that's that's so true. I put I'm putting two and two
6:01
together now. Yeah, it makes complete sense.
6:05
So anyway fellas, I'm insanely happy that we are getting to do
6:10
this. I feel honoured that you're both
6:11
on a call at the same time and I know there are a tonne of people
6:16
that are die hard fans of Duck DB and Mother Duck and I have to
6:23
show I can't. So for those that are not
6:26
watching the video, I am lifting up my shirt right now and
6:29
showing I have my duck TB swag on from the launch party and I I
6:35
think I. Told you I've got a Duck TV
6:37
shirt on as well that maybe. Yeah, the original original Duck
6:40
TV logo. Yeah, that's the, yeah, that's
6:43
the collectors item, honest. Feels like you have to strip.
6:46
Yeah, hang on. I have, I have, I have.
6:49
I have some stuff here about it's like.
6:52
A. Yeah.
6:53
I have a I have a Mother Duck Hawaii shirt here so.
6:58
That's another good one that I saw people rocking and I I would
7:02
love those. I do use my Duck TB mug that I
7:05
got from that same party every morning for my coffee because it
7:09
holds the most coffee out of all my mugs and I imagine the people
7:16
that are listening are probably like what's all the fuss about
7:19
this Duck DB, mother duck thing If they are not familiar?
7:25
I would love to go through basically the story of first
7:31
maybe Duck DB, the creation of Duck DB and then and the why
7:35
behind it. And then mother Duck the
7:38
creation and the insane amount of traction that has come off
7:42
the back of that and what the into what the relation looks
7:47
like between the two and how that works before we dig into
7:51
the actual tech of it all. And so honest, I guess this is
7:55
where you get to jump in and tell us.
7:57
Yes, well, yes, thank you. I'd like to compare this to this
8:01
story in The Matrix where somebody tells Neo that he's
8:04
always felt there was something wrong with the world, but he
8:08
couldn't really put his finger on it.
8:09
And that's a bit where, you know, where this whole thing
8:11
started. We were teaching courses for
8:13
people to use like big data systems, and it was, it was a
8:18
very, very bad experience. It didn't really work.
8:20
And at first, you know, you don't really actively question
8:24
this. But if a couple of years it was
8:26
like, maybe there's something like wrong and maybe we can do
8:31
something differently. And that's kind of how we got
8:34
the first ideas to make Doctor B, which was this totally
8:37
ridiculous departure from the common prevailing wisdom at the
8:40
time where everybody was says like, oh, you need to be scale
8:43
out. Otherwise we were doing
8:45
research, people were laughing at us, like if he doesn't scale
8:48
out, it's not, it's worthless. Right.
8:49
And then that became around and you know, we thought, OK, we
8:54
are, this is a pretty wild story, uh, nobody.
8:59
But it was the hill we were kind of willing to die on.
9:01
And so we were kind of building our our system that was like,
9:06
yeah, I could really like proudly, proudly singled out and
9:09
proudly like in process. We can talk about what that
9:11
means later, but but we thought, OK, we're just gonna be happy
9:15
with being the weirdos to the rest of time.
9:18
And then with this crazy thing happened because this very
9:20
respectable gentleman from America, Jordan, turned up and
9:25
said no, I think you're actually right, This that was.
9:28
Locked in there. Yeah, you may be onto something
9:31
with this, with this sort of like going small and that.
9:33
For me, this was a total shock when I when I first heard this.
9:37
So that was really fun. And respectable.
9:39
I think it's probably a good point to mention your
9:43
background, Jordan, and why you are respectable.
9:48
Yeah, respectful is respectable isn't necessarily something I I
9:52
aim for but last I've I've ended up here.
9:56
But no, I I worked on databases for for a while I kind of or I
9:59
snuck into them I guess as a as a as a big data person and we
10:04
could maybe talk about big data a little bit a little bit later
10:06
but held start. Google Bigquery worked on that
10:10
for 10 years that I worked at single store for a couple of
10:14
years. And so it was very into into
10:18
databases, databases, a service, distributed databases.
10:22
And I remember somebody saying like, you know, if you're not
10:27
distributed like people will laugh at you and I'm like, but
10:30
but why is that? Because, you know, there
10:32
actually were, you know, a single store, you know, a lot of
10:36
actually users who were, you know, scaling them up and
10:38
scaling them up and doing really well.
10:41
And then we also had a bunch of customers that wanted to scale
10:44
down and like and scale down with a lot of these big
10:47
distributed systems is really hard because they're just so
10:50
many components and so many, so many moving pieces.
10:53
And then I, you know I encountered encountered duct BI
10:57
think somebody was doing some performance comparisons.
10:59
I'm like wow where did this come from.
11:02
And like I started doing some some some research into it and
11:06
and and realised that like you know these people have something
11:10
really, really interesting and and that you know you can you
11:17
can scale up and they they have a whole you know actual research
11:20
behind behind what they were doing.
11:22
And they could do a lot of things that you know customers
11:26
that I'd seen before wanted to do.
11:28
And that you know there's a lot of databases out there lot of
11:31
database projects. And I think that the the one
11:34
that made me realise that this was real was actually they had a
11:38
blog post on on time zones. And I I remember in Bigquery it
11:43
took us like five years to add you know sort of proper time
11:47
zone support because like it was like it's just so hard.
11:50
Like there's just so many ugly nasty details and like and it
11:55
just breaks your brain. It's like it's like if you have
11:57
a limited size team like why would you why would you spend
11:59
time on this. But it's like the fact that they
12:01
had spent the time on this to do it actually.
12:03
Right. Meant to like, they're not just
12:07
sort of fooling around in this sort of academic research
12:09
prototype. They actually want to build a
12:11
real database. And then, you know, that's sort
12:14
of one of the triggers that let me know that like, hey, this is
12:18
something that I, you know, it would be good to sort of hitch
12:21
my hitch my waggon. To the time zones made you
12:24
realise it was legit. That is awesome.
12:27
That's such a good story. So then honest, like going back
12:30
to the inception was your why of creating it something that
12:37
you're you're teaching classes, you're realising that the
12:41
experience, the developer experience is quite painful and
12:45
overkill I guess is kind of the word.
12:48
And you said we don't necessarily need, was it trying
12:52
to counter the narrative of we don't need this big data all the
12:56
time, we can go smaller or did that come later?
13:00
How did that play out? I think, I think at a time, you
13:04
know, we didn't really have a really let's say straight Laura,
13:08
despite being researchers and supposedly being very sort of
13:10
brainy all the time. As I said, it was more like a
13:13
feeling that that maybe it wasn't really necessary, but I
13:18
think the from just also just talking to, you know,
13:22
practitioners, we we kind of realised oh maybe maybe these
13:26
petabyte sort of datasets are actually quite rare and the
13:32
people's real problems are like some, you know, some some couple
13:35
of 100 megabytes of CSV files and there's absolutely nothing
13:39
out there in terms of software that can support that.
13:43
So yeah, it was, it was really like based on a need, which is
13:48
very uncommon for research, right.
13:50
Like, normally people love to do research into thin air and just
13:53
dream up a challenge and it's making me very angry.
13:57
But but I I I don't, I don't want, I don't want to live like
14:00
that. Especially not if you're like
14:01
funded by taxpayers. I feel like you should somehow
14:04
are sort of indebted to them and need to solve some of their
14:06
problems. At least.
14:08
So we've thought, OK, let's try to fix the data problems for the
14:13
people that we have spoken to. That's as simple as that, right?
14:19
It wasn't. It wasn't trying to, you know,
14:23
make make it a whole, you know, movement as we say in German to
14:27
open a big barrel. You know, we didn't try to open
14:29
the barrel just to try to like fix the problem that we
14:31
perceived in the world. Yeah.
14:33
And I think another thing that it is different was that we also
14:36
from being from the research world, we were kind of sick and
14:39
tired of people making these throwaway research prototypes
14:42
when they were making software. Right.
14:44
So when we started we were like, OK, you know what, Let's try to
14:48
do this properly, of course, as through the best of our
14:52
knowledge. But it it's also a bit of a
14:54
departure. I think it's one of the reasons
14:55
why we don't have to constantly, you know, fix, fix sort of
14:59
horrible issues from the past because we said back then, no,
15:02
we want this to be actually used by people.
15:04
Yeah. And what were some of these
15:07
architecture design decisions that you made early on that you
15:11
felt like were against the grain and you're still happy about
15:16
making them to this day? Yeah.
15:19
So I think the whole, the whole single note thing is still
15:21
something I'm very happy with. That was not obvious in 2018.
15:26
Today I think it's it's clearer than it was back then.
15:30
Can you explain what the single node is?
15:32
Single node means Simmons, that's like you have a a like a
15:36
data system that runs on one computer.
15:40
OK, it can be a big computer, but it's one computer.
15:42
It's not, you know, a collection of many computers.
15:45
Like, you know what? What was the prevailing wisdom
15:48
at the time? So it's single node.
15:49
It's unapologetic single node. And the other thing is the in
15:54
process architecture of the ECB, right, where instead of you
15:58
talking to your data management system through some sort of
16:01
socket protocol which is like everyone else does this.
16:04
We said no, it's going to make our lives even harder and try to
16:07
run inside other people's processes because that has great
16:09
advantages for like data transfer because you can sort of
16:12
super efficient interaction with like application programmes,
16:15
third party libraries, these kind of things.
16:17
So I think those are two things that we sort of we picked and
16:21
I'm still quite glad with other things may have changed down
16:25
like in the last five years, but but nothing I think
16:27
fundamentally the fundamental sort of idea is still the same.
16:30
Yeah. And I guess to put a time frame
16:33
on this too, that would be useful.
16:36
The project started in what year it feels like, what 2018?
16:41
That trends 18 is a farce commit, yeah, obviously that's
16:44
that's. And then Jordan came and rang
16:47
you up in, what, 2020 or 20? Minutes later, no.
16:52
Just last year. It was April of last year.
16:55
Wow. OK, a lot is.
16:56
A lot has happened since then. Yeah, I'll say you guys have
16:59
been busy, man. Yeah.
17:01
At that point, duck TB, you know, I think people were just
17:04
starting to hear duck TB and it was kind of, it was still a
17:06
little bit under the radar kind of if you follow like the DB
17:10
engines ranking, which is sort of like the, I don't know, the
17:14
like Premier League tables, you know, league tables for for
17:19
database nerds. I think they were down at like
17:21
270 and they sort of have been rocketing up the the list since
17:25
since then. And so yeah, it was a little
17:29
bit, a little bit under the radar, but just sort of just
17:32
kind of coming on to its own. People were starting to talk
17:34
about, you know, hey, there's this great, amazing database and
17:37
and talking about it with, you know the kind of effusive praise
17:41
that you know people don't usually talk about data
17:44
management systems. They don't usually talk about
17:47
technical things at all that way.
17:49
So, you know, you could tell that they were, they were on to,
17:52
they were onto something. Yeah, you know, there's
17:55
something special that's true when I I think that's how Ducky
17:58
Bigot came out of my radar. Is that fanatical or yeah, how?
18:06
How can I say this? It's it's not fanatical.
18:09
It's more the absolute love in which people talk about duck.
18:17
TV is incredible because just as you said, Jordan, it's not that
18:22
often you hear people talk about technology that way.
18:25
If anything, technology has to live up to a very high standard
18:28
and as soon as it does one thing wrong, it's like, oh, this is a
18:32
piece of shit. But I started seeing a lot of
18:36
people posting things. And then I think one of the, I
18:43
guess we could call it marketing ploys that I latched on to and I
18:48
was like, whoa, this is a really easy way to understand what
18:53
you're trying to do is the idea of big data is dead.
18:58
So I don't know who, which one of you, which camp did this, if
19:01
it was the duck TB camp or the mother Duck camp, But that made
19:06
everything so clear for me and whatever. 4 words you know.
19:12
So that was, I mean that was, you know, I wouldn't necessarily
19:15
call it a marketing ploy, but it was certainly like you know,
19:18
kind of the culmination of of a lot of experience and databases
19:21
and and maybe it was a little bit over, you know, overly,
19:27
overly done, but you know, basically to make a point
19:30
because I think that we just sort of have been in this this
19:32
big data era for a long time and everyone just sort of assuming
19:36
that everyone has big data. And there's just a lot of people
19:41
that we talked to who were like who you know, they they have you
19:45
know, reasonable sized data that you know is a few gigabytes and
19:48
maybe 10s of gigabytes or even hundreds of gigabytes.
19:51
But that's that's not, that's not really a big data anymore.
19:54
And people kind of felt almost like they weren't real data
19:58
engineers if they weren't, you know, operating on these huge
20:01
systems. And I think that kind of big
20:03
data is dead. At least gave them like a a.
20:07
You know justification of like, yeah, you're, you know, you're
20:11
you're valid you're you're you're experiences are valid
20:15
too. And it sort of comes from, you
20:16
know, working on Big Query for a long time.
20:18
You know, Big Query, you'd think people had big data and big
20:22
workloads. But you know, really looking at
20:25
what people were doing is most people had, you know actually
20:31
very small amounts of data, even really big customers and really
20:34
big names. Most of the work that they would
20:37
be doing is isn't over like their giant logs tables, it's
20:40
over some sort of summarization cleaned up version of the data.
20:43
When you clean up the data you like, you make it, you make it
20:47
smaller and you know virtually all of our, you know, of our
20:51
queries were, you know, almost all of them, probably 99.999%
20:56
were sub TB, you know, 90% were some 100 megabytes.
21:02
And so like you know, kind of trying to get the idea out
21:06
there. I was trying to get the idea out
21:07
there that you know, hey most people don't have big, big data.
21:10
And this is sort of something that I'd seen in the real world.
21:13
And I think since then people have been sort of backing that
21:16
up and validating, validating that the people that do have big
21:20
data, they tend to only use smaller portions of it at a
21:24
time. Like you know, you have sort of
21:26
the hot, the hot data. You might have like 10 years
21:28
worth of logs, but you're really looking at the last seven days
21:32
or the last one day of of of data or you are, you know,
21:38
working on some sort of cleaned up, cleaned up version of that.
21:41
And so, you know, people would always talk about kind of the
21:44
size of their data warehouse. And when they would talk about
21:46
the size, they would generally talk about the, you know, the
21:49
terabytes or the 10 terabytes of logs that they had.
21:52
But like that's actually not all that useful when you have
21:55
separation of storage and compute because like you're, you
21:57
know, basically everything's being stored on S3.
22:00
You know, OK, you can dump a whole tonne of crap into S3.
22:03
But the really important part is like how much, how much of that
22:06
are you're actually using? And so systems have been
22:09
designed for the last decade aided to deal with like that
22:13
whole multi TB size which is sort of no longer no longer
22:17
relevant. And and be there were designed
22:21
in an era where like a machine that had a few gigs of RAM was
22:26
considered really huge And and nowadays like you know if you
22:30
look, you know you look at your EC2 instances, you know very
22:34
often like you know your VM, you know maybe it's only you know
22:36
1616 gigs, but those are running on a physical machine that are
22:40
actually quite large. So you know, many of them are
22:42
have terabytes of RAM and you know and hundreds of and
22:47
hundreds of processors. So there's really not a whole
22:51
lot of workloads that won't fit on you know on those, on those
22:55
machines. And so like kind of just the the
22:57
design that we've been building systems around I think has is
23:03
not as relevant to kind of the sizes and shapes of of data and
23:08
and data workloads. And yeah sort of getting back to
23:11
you know harnesses, you know, you know point about you know
23:15
about single node and scaling up is like yeah single node does,
23:18
you know, single node can, you know can work and and and can be
23:22
used so much simpler and so much faster.
23:25
Because just remembering from the Bigquery days, like a lot of
23:29
the work that we did. You know, a lot of the
23:30
difficulty was on sort of dealing with this complex
23:33
distributed system and its single store was the same way.
23:35
Sort of like how do we do like this distributed distributed 2
23:39
phase commit like that's a really hard problem in a single
23:44
node. Like a lot of these problems
23:46
just get so much, so much easier and that means you can move
23:49
fast. And I think, you know we've seen
23:51
kind of Duck TB, which was awesome in April of last year
23:54
versus Duck DB now is like incredible.
23:57
Like the fact that they keep getting better at all these
23:59
benchmarks and adding more features and and innovating and
24:02
in SQL dialect and all of these things that they wouldn't have
24:05
been able to do if they had, you know, been chasing this sort of
24:09
complex distributed distributed system problem.
24:12
I like this because it it really shows that ease of use is both
24:18
for the developers and the user, so the user developers and the
24:23
developers of the system. And so it has that, basically
24:29
the double whammy. So you can enjoy it from both
24:33
sides, whether you're building the actual database or you're a
24:37
consumer of that database. You get that ease of use.
24:41
And I also want to highlight something else that you said
24:44
Jordan, which is around the big data is dead type thing and how
24:50
many people probably felt that they needed to over engineer or
24:56
at least that was the consensus, right?
24:58
Like you're saying if you're not doing this, if you're not
25:01
setting up for scale then what are you doing?
25:03
Do you even call yourself an actual engineer and then at the
25:08
end of the day you realise you know what?
25:10
9 times out of 10 I don't need it.
25:12
Even if I have the data, I'm not using it and I don't need it and
25:15
nobody else does either. By the way, like my data
25:18
scientist, they don't need that data either.
25:20
They only care about the freshest features that are in
25:22
the last hour, let alone the last like month.
25:26
And so speaking about that, it shows how again it just goes to
25:33
you were able to hit a nerve and it almost feels like there was a
25:37
lot of people that were in the dark about something or they
25:41
felt a certain way and there was a lot of pent up demand there.
25:44
And then you gave a voice to that and everybody's like, yeah,
25:47
this is what I've been feeling for this whole time.
25:50
Of course, like that makes so much sense.
25:53
And so kudos to you for actually like, naming it and letting a
25:59
whole movement rally around it. I do want to talk a little bit,
26:03
like staying on this Jordan, on the creation of Mother Duck and
26:07
then what that story was like. Because I I understand you
26:13
reached out to harness, you were like, hey, this is cool, I wanna
26:15
do stuff with it. How does that work?
26:17
Or did you just start committing and then after a few commits you
26:20
were like wait a minute, I need to like talk to harness and do
26:24
more. So my plan was to like, you
26:28
know, was was to hack on something and like somebody
26:30
really should build a serverless, you know, duck DB, a
26:33
cloud cloud duck TB. And then I'm like, hey, well,
26:36
I've worked on, you know, help build 2, you know, cloud
26:40
databases and service. You know, maybe that should be
26:43
me. And I'm like, so maybe I'll
26:44
just, I was, you know, hack on this for a little while.
26:46
I get about 3 days into it and I'm like, you know, I should
26:51
really kind of reach out to the duck TB folks and see if they'll
26:53
hire me. And and so I I got an
26:58
introduction to to Hannahs and Mark and I remember I talked to
27:01
them on like on Monday, on Monday morning And like it's
27:04
like, you know trying to feel them out like hey, you know, you
27:06
guys you know, thinking about doing something in the cloud
27:09
like I'd love to work on that And like and they were like no,
27:12
we're really not going to not don't want to do that.
27:15
But we'd love to partner with somebody you know who would who
27:18
would do that? And like, you know, seems like
27:20
you have a reasonable background and might be might this might
27:24
you know this might work. You know that same afternoon I
27:27
talked to because the person would introduce to us was Lloyd
27:31
Tab the founder of Looker. And he'd been a bit of a mentor
27:35
to me and I kind of had had known him really well because he
27:38
was he was the one one of the few people that really got Big
27:41
Query and what was possible with Big Query from an early from an
27:44
early early stage And and so he introduced me to Hannas and he
27:48
said you should also talk to my friend Tom who invested in
27:52
looker and you know Tom was Tom Tongas from he was then he was
27:57
at Red Point. Now he has his own his hung out
27:59
his own shingle as as a VC. And so I talked to him in the
28:03
afternoon and I really had nothing written down and just
28:06
sort of this vague ideas and vague plans that I talked about
28:08
for about 15 minutes and he's like, yeah, that's a good idea.
28:12
I wanna fund it, come to my partner meeting next week.
28:14
And so I was like went from like, hey, maybe I should hack
28:17
on this thing to you know. You know got.
28:20
Funding, just wanting to partner, how does Mark wanting
28:22
to partner and like if you see you wanting to wanting to back
28:25
this and it just sort of all of a sudden accelerated
28:29
extraordinarily rapidly from from there and a bunch of people
28:34
kind of came out of the woodwork and like say hey I'd love to
28:37
work on this with you. It's a really incredible people
28:40
and that I think we you know we came up with like sort of an
28:42
interesting way of partnering with the with the Duck B team,
28:45
which I think has also been a superpower for us in getting
28:51
started. So before we talk about the
28:54
partnership and how that looks, because I I know for me it was
28:57
very confusing in the beginning. I thought it was just the
28:59
managed duck TB, honest. Do you remember that story?
29:04
Yeah, what was it? Like for you?
29:06
I mean, I mean, I know the Americans are crazy, but I
29:09
didn't think it would be that crazy.
29:11
So like we talked as Jordan said, we know we talk on the
29:16
phone and we are like, yeah, no, this sounds great.
29:18
Like we totally can talk about that more.
29:20
And then he gets like in a play in the next day or something
29:23
like that. It comes out here to to
29:25
Amsterdam where we are at and just to talk things and and that
29:31
was just at the time we were like what's going on here And
29:36
and we had already been a bit burned by by like VC's
29:40
essentially like a lot of interest from VC's that wanted
29:42
to fund this and they were like yeah you need to make a service.
29:46
But we didn't really want to because we know some people that
29:48
did database as a service really well.
29:50
And I said look this is really a separate thing and you really
29:52
need to know what you're getting into.
29:54
So we thought maybe we can find some, you know, competent person
29:59
to respectable person and not at all crazy person that that that
30:04
once that wants to do this and then you know Jordan just shows.
30:07
Up, you know, and. And it was, it was, it was
30:09
really wild, I have to say. But we, you know, we got along.
30:13
And so it was, it was, it was, it was a great success.
30:17
I'm still very grateful for that.
30:20
Match perfectly, man. Yeah, and what a, what a story,
30:24
Jordan. Going from trying to get a job
30:27
to having funding secured all in the same day and then feel like,
30:31
alright, tomorrow I'm going to Amsterdam.
30:33
It was pretty much like that. It was actually crazier than
30:35
that. But like, I I didn't need to
30:38
probably write, write a book or something at some point.
30:41
Yeah, like, I think it this works out, it'll be an
30:43
interesting book. If it doesn't work out, it'll be
30:45
sort of like you know what can go wrong.
30:49
You know the the memoirs. Man, this is all, yeah, make
30:52
sure you're taking notes and keeping it for later because
30:54
that is an awesome book that I'm sure many would love to read.
30:58
Now let's get into the partnership and how it looks
31:01
because I was under the impression basically I heard a
31:04
Duck DB and then next thing I know it's like ohh yeah, mother
31:07
duck. They came out of stealth and
31:09
they've got this cool product and then they've got funding.
31:12
And so that was what I heard and I immediately assumed, OK, Duck
31:19
DB is the open source and the mother truck is the managed
31:22
service on top of that like the managed database.
31:24
But it I think there's a lot of nuances and differences and also
31:29
inspiration in the ways that you both work together in the
31:32
partnership works. Yeah, I mean it's what it's
31:36
true, the impression is true that activists, the open source
31:39
part and Mother Duck is a managed service.
31:41
That is fair. That's correct.
31:42
But I think what is different is that it's two different
31:46
companies that are basically doing these two things and
31:49
they're very clear, with very clear sort of goals and
31:53
different staff, different leadership, all that stuff, so.
31:57
Yeah. And it does.
31:58
So there, there's so many questions that come to mind
32:01
because it is, for you, honest, right?
32:03
Like you're probably just trying to figure out how to make a
32:07
better database and then you don't care about shareholder
32:11
value or the VC money that got dumped into Jordan's startup.
32:15
None of that really matters to you.
32:17
So it feels like at some points there could potentially be
32:21
misaligned incentives or you could want to go two different
32:24
ways. Have you hit that yet, or is it
32:26
still too early and you haven't had that problem?
32:30
I don't think we have hit this problem yet.
32:33
I I think, I think just to be called making database something
32:36
like it's just making a database.
32:37
I think it's an it's a big insult to me, like making
32:40
databases the end all to everything but.
32:43
Especially I knew you were German when you were like, I
32:46
just want to optimise the shit out of this database.
32:49
Well well, we haven't optimised the shit out of yet.
32:51
In a big departure from being German, we have actually looked
32:54
at making the user experience good.
32:55
So this is a very very very non German solar activity.
32:58
But and I think, I think the the relationship is absolutely
33:03
special. I think that's that's I haven't
33:06
heard it a lot. We we didn't want to go for
33:09
these sees just to keep because we want to really focus on you
33:13
know making something that as many sort of people can use as
33:16
possible right. And then sometimes there's a bit
33:19
of a conflict there with the product strategy of a company,
33:24
let's say, right. I mean we've seen this time and
33:26
again there's an open source sort of project, they add a
33:28
feature and they later have to drop it again because you know
33:31
their investors say you can't and you have to increase revenue
33:34
and then you better take this feature out then everybody
33:36
screams at them. I mean that's this is very
33:38
unfortunate Will and and you know by having two different
33:43
companies we kind of I think elegantly sized up this problem.
33:47
Hey, this is Mike Delbalso, Cofounder and CEO of Techkon.
33:51
ML OPS Community is the best way to stay in the loop on the
33:54
latest ML OPS news and best practises.
33:57
It's also a great way to connect with experts and get support
34:00
from an amazingly helpful community.
34:02
Subscribe and stay in the loop. Yeah, there seems to be like
34:14
this pattern of people that are building companies based on open
34:17
source products that they really like.
34:20
The way they get funding is adoption of the open source
34:22
product. And so they focus only on on the
34:25
development of the of whatever that product is and you know
34:29
getting it out as many as many people's, people's hands as
34:31
possible. And then people ask well how are
34:33
you gonna monetize and they say we're gonna monetize via a
34:36
service, we'll have SAS and then they like have like two people
34:39
working on that and kind of in the background.
34:41
But it's not really like their goal is to build something,
34:44
something amazing because that's really how they get, how they
34:47
get the funding, that's how they get the, you know the the, the
34:50
product LED growth adoption etcetera.
34:52
But the problem about that is that that really under feeds the
34:56
the databases or the the service part, you know whether it's a
35:00
database or something or something else.
35:01
And so they really don't focus on innovating in terms of like
35:05
delivery of the service, how that service works, making
35:08
something amazing and different and differentiated.
35:11
And then also that that ends up leading to something that like
35:15
AWS can just clone because like you have this great open source
35:18
thing and then OK you have this sort of like half assed service
35:23
and and Amazon could be like, OK well I can run this service
35:25
better than you can and meanwhile they're eating eating
35:28
your lunch and and so we wanted to sort of take it from the
35:32
other direction is be like OK the doctor B team, Doctor B Labs
35:35
team is going to build an extraordinary database and
35:39
that's going to be their focus. That's what they want to do and
35:41
that's that's that's wonderful. We're gonna build an awesome
35:46
differentiated innovative database as a service.
35:50
And we think we're doing a bunch of really cool things with like
35:52
the hybrid execution and kind of how we're doing auto scaling and
35:56
individual you know talk to you back ends.
35:59
I'm happy to happy to talk through like some of some of
36:02
those things. And then you know then the key
36:05
thing is like you've got to have a way to kind of tie yourselves
36:08
together at least loosely. And so when we started, one of
36:13
the things that we really wanted to make sure that we did was was
36:16
not just say, hey, you have awesome open source technology.
36:20
Thank you. We're gonna take that and make a
36:21
bunch of try to make a bunch of money out of it.
36:24
You know we did, you know want to include them.
36:26
We gave them a cofounder share. We have like a a development
36:30
relationship with them that that does make it you know make it so
36:34
that if things work out for us, it works out if they it works
36:37
out for them. But they are also free to go and
36:41
and build things how they how they they believe is the right
36:45
is the right way to to build a database.
36:47
And I think this, this, this relationship that, you know, I
36:50
think if this works out, it'll be seen as like, hey, this is a
36:54
good way to do kind of open source, open source development.
36:58
And if it doesn't work out, it'll be like a cautionary tale.
37:03
And you know, because people have warned you know as you as
37:05
you mentioned that you know the the relationship is going to be
37:08
is going to be hard as you know incentives, incentives diverge.
37:13
You know, if you know people who are building competitive things,
37:19
you know, pick up on duck TB, you know the duck B team is
37:24
gonna be super excited about that.
37:25
And meanwhile we're gonna be like, hey, they're eating our
37:27
lunch and not as and not nearly as excited about it.
37:30
And I think that's something we just have to you know to bridge,
37:33
we'll have to cross when we when we get to it.
37:35
And we want to just focus on building a high trust, high
37:39
trust relationship that you know we hope we'll take take us as as
37:43
as as far as possible. So I do want to get into all the
37:47
features and like the differentiators and when you see
37:51
people using Duck DB versus when you see people using Mother Duck
37:55
and what the use cases and benefits are for each of those.
37:59
But before we get into that, I'd love to just talk about building
38:06
an incredible developer experience.
38:08
Because it's going back to this idea of there's been a bit of a
38:13
movement and there's people don't usually talk about
38:16
technology the way they talk about Duck DB.
38:19
And Honest, you're like, we haven't even started with the
38:22
optimization. We've just been focused on the
38:24
developer experience. Both of you, I imagine have that
38:29
in the like. I don't know if you have it
38:32
written on your wall or if it's framed.
38:35
What is this look like and how do you optimise for that?
38:40
How do you go about building that?
38:44
Is it that you're in 90% user meetings versus actually 10%
38:51
developing the product meetings or like break it down so that
38:56
the rest of us when we are building out, maybe it's we're
38:59
building out our internal machine learning platform, we
39:02
can have these experiences that people come to us and say, whoa,
39:07
this was a cool developer experience.
39:09
So I I do not sit in a lot of meetings with you didn't.
39:14
Users as a type but. I, no, no, I think, I think it's
39:19
maybe there's an aspect that we actually care about.
39:22
I mean we actually care about data management systems and we
39:25
want to we want to not you know make it difficult to use these
39:31
things because we feel like uh this would this would really
39:36
really hinder their adoption and that that would mean which means
39:39
that our impact is is not as big as it could be with the work
39:42
that we do. Like it's I think it harks a bit
39:44
back from sort of first principles about what you want
39:47
to do as a scientist that we kind of started out as is like
39:52
you want to you want to make you know we have you have your ideas
39:54
sort of be adopted in the world and how do you do that.
39:56
Well you you make make it make it not, you know make it not
40:01
suck. I don't know.
40:02
It's it's very, it's very, it's very.
40:05
Easy to grasp, yeah. It's a simple concept.
40:07
You, you know, you you ask yourself is this really the best
40:10
it can be? And is it?
40:12
Is it like, can we change this to be better?
40:14
Can we Like for example, due to be has no external dependencies,
40:18
right? We just you can just install it
40:20
as If you want, you can just compile it with just a compiler.
40:23
OK, And what that is annoying, you know, for us because we have
40:29
to you know we can't just like pull in like some library when
40:31
you do something. We have to be like now OK, can
40:34
we maybe inline some part of this code?
40:36
Can we maybe do it ourselves? Maybe we can trick the operating
40:40
system into doing something. But the upside is that it it
40:45
sucks less to to work with Doctor B because it means that
40:49
you can just install it and you don't have to ask your admin and
40:51
you just run down on the binary from the Internet.
40:54
By the way ask your parents downloading binaries from the
40:56
Internet. But it's it's it's it's just
40:59
simple and and I think I think we really love like data
41:03
management systems as as as a piece of technology and it's
41:06
really hurting my feelings to see things like Oracle being or
41:11
other big database companies. I don't want to single them out.
41:13
But being being these things being incredibly hard to use
41:16
being incredibly clunky, being like this like the you know the
41:18
mummies of the 70s are looking at you when you stare into their
41:21
command line interface or SQL dialects.
41:23
It's just like this is if you care about this.
41:27
It's really hard to see and it's not like I think it's also like
41:31
there's this, this is much this step where data management
41:34
systems have like been around for a while.
41:37
They're incredible piece of technology, but they they have
41:39
aged a bit and nobody has really been able to reimagine them
41:43
without throwing away a bunch of other stuff in the process which
41:46
you know like may have gotten a bit over the edge and we kind of
41:50
we in some sense activity is an incredible incredibly
41:53
traditional system. Like it has a sequel interface.
41:55
Wow. It deals with tables like wow.
41:57
It doesn't do like graph or stream or whatever.
42:00
It's bulk processing, no, you run the query on the data,
42:02
that's it. But at the same time it's also
42:07
the case that this is still the biggest use case for data
42:09
management systems and analytics.
42:11
So, so targeting that was just a really great idea but then you
42:14
just take, just take the pain away.
42:16
It's hard to describe for me, but but it's it's as it's
42:20
really. I care deeply about how people
42:22
perceive the things I love. You know what I mean?
42:26
I also an interesting thing about the perfectionism like
42:29
that I I I find pretty amazing is is you know duck TV is is
42:34
super fast and on all these benchmarks this duck TB is like
42:38
is has been kicking butt but they're like the the the
42:42
performance is almost the exhaust you know of like the
42:45
engine that is just sort of trying to do the right thing and
42:48
it's like hey we build this right we do do it the right way.
42:51
Like I love the story of the the the new, the new changes to like
42:55
to aggregation that you know they're trying to make it scale
42:58
better to to be able to have to to not run out of memory when
43:01
you have we're aggregating a lot of a lot of things and basically
43:03
in the mechanism that they were using to do that, Ohh, it has a
43:07
side effect of like making you know these benchmarks much
43:11
faster and making a lot of these types of queries much faster.
43:13
But like but that was just sort of a side effect.
43:16
It wasn't like, hey, we're trying to go crush these
43:18
competitors on these, you know, TPCH, you know, Query 9 is you
43:24
know, needs to be faster. And so we need to put in some
43:27
special optimization rule for for this.
43:29
Like it was really just, hey, this is the right thing to do.
43:32
They do it and like it has incredible outcome.
43:38
Yeah, and I I like this principle of doing the right
43:43
thing, not the easy thing. It's like, ah man, no external
43:48
dependencies is not going to make our life easier, but it is
43:52
going to make the experience better.
43:56
So we might as well just get in there and do it.
43:59
How much are you having students dog feed or eat the product per
44:06
se and then you are learning from them.
44:08
It's just a basically a research lab and you got a bunch of
44:11
Guinea pigs going on in there. I don't think we've we took us.
44:16
We waited a very long time till we use stuck Debian courses
44:18
actually because it's nothing is more embarrassing than the
44:21
professors half baked thing that he forces on you.
44:25
I've been in telling you I've been that student and it's not
44:28
it's not it's not it's not it doesn't look make this make
44:31
anyone look better right let's so so we have actually waited
44:35
quite a long time now we're like OK I think the in the beginning
44:41
we didn't have a tonne of feedback but we did have some
44:44
interactions with people in sort of the practitioners to to see
44:48
whether we're going in the right direction and and as you
44:50
mentioned there is all these all these excitement there was a
44:54
tonne of excitement about due to be like version 0.1 that that we
44:58
couldn't comprehend because it was quite half baked right but
45:02
but people got excited and and I think they got excited because.
45:07
Yeah because we cared. I don't know it's it's it's very
45:09
mean to say we cared because everybody else also cares but
45:13
they have very very very competing goals like if you're
45:16
if you're in a company you're you're you care about paying
45:18
your people that's a that's an important sort of thing.
45:21
We were at a Research Institute when we came up with a thing
45:24
where you're where you're as like your incentives are aligned
45:28
quite differently where you you know you're there to do do the
45:32
right thing and I don't know but you're there to like you have
45:36
simply you have notified like abstracted away a lot of
45:39
concerns that other people usually have.
45:40
So we could say OK this is going to take us a year longer to do
45:43
like this but it's the right thing to do.
45:45
So there we go which you could never do in a startup right.
45:50
Your seed funding round will not allow these kind of things.
45:53
It's. Yeah, I think also like you know
45:57
most database people of databases, they focus on like
46:00
the like the just the database part, like it's OK.
46:04
So once you give me the query and I compute the results and
46:09
then my job is done and I think that but the the, the places
46:13
where people have problems ends up being like, OK, what happens
46:17
before then, you know how do you, how do you install it, how
46:19
do you get the query there, how do you get the data there?
46:21
How do you like make it make it easy and then how do you get the
46:25
results out and like and then you know duck TB focused on on
46:30
the whole the whole end to end experience.
46:33
Whereas most other database companies like that's somebody
46:37
else's problem And so and so people were kind of you know not
46:42
being receptive or a lot of database companies and people
46:45
like tend to not be receptive to like the like.
46:48
Ohh well that must be you. You weren't able to install
46:51
that. That must be a user error or ohh
46:52
like you know the you couldn't you couldn't import the CSV
46:57
file. That must be a user error.
47:00
That's not my problem like and I think duck TB just you know
47:05
focused on those problems as well which are hard problems.
47:08
I need to make those to make those easy and I think that that
47:11
also is one of the reasons why people, you know, we're so
47:15
excited about it. And now I think the thing about
47:19
it is that after we have that experience, it's like it's
47:23
opened up our mind to that it's possible and so now hold
47:27
everything else to such a higher standard.
47:29
Ohh, I'm so. Sorry, this is like yeah, but I
47:33
but you know the CSV things is actually an excellent example
47:37
because it's we observe this right.
47:40
So you're you're you're you're you wanna do something with
47:42
data. You have a CSV file because of
47:44
course you have a CSV file and then so it's actually after
47:47
installation. It's the second thing you kind
47:49
of do with your shiny new data system right.
47:52
And if that is a horrible experience then you are sort of
47:55
scarred for life and you would rather go home and I don't know
47:58
punch the wall or something. So.
48:01
So we actually wrote a research paper about CSV reading that the
48:07
results of which are now part of effectively.
48:09
And This is why we have, I think the planet's best CSV parser at
48:12
this point. Which is something that you
48:15
would that we have like 1 like PhD of computer science who does
48:18
nothing else, then work on the CSV parser because we realise
48:23
it's such an incredible crit. It's it's such a critical piece.
48:26
It's like if people can't get their data into your system, it
48:29
doesn't matter how good your join operator is, right?
48:31
It's absolutely irrelevant. So yeah, it's, it's, it's, it's
48:37
sometimes it's amazing and Jordan has said like this entire
48:40
package, but it's also just sometimes amazing.
48:43
Have actually these people ever try to use these things
48:46
themselves and it's actually something you do.
48:47
We encourage our team and we do it ourselves.
48:50
It's just use this thing, just try just process some data and
48:54
see what happens. And then if you run into
48:56
something like. Hey.
48:57
Maybe other people run into this as well, right?
48:59
It's it's pretty obvious but. Then you throw a PhD on it.
49:04
You say, well, that in turn, with that that research.
49:08
I insist that's your next PhD if good luck.
49:13
I didn't know. I mean we were fortunate that
49:16
let's say that the the degree density at the company is quite
49:19
insane. So it's like.
49:21
Well, I love this idea too, because it in events, I've heard
49:27
it explained as the moments in between the moments.
49:32
So like when you have an event, you have the big moment of the
49:36
singer going out onto stage, But there's a lot of moments that
49:40
happened before. You get to where the singer is
49:43
on stage and that incredible song where your heart is jumping
49:47
out of your body. There's the parking, there's
49:51
getting through the ticket line, there's waiting, watching the
49:54
opening act. All of that are the moments
49:57
before the moment, right? And so if you can optimise
50:00
those, the whole experience as a memory in your mind is going to
50:05
be incredible. And it feels like that's kind of
50:09
how you are looking at it. You're saying we're not gonna
50:11
just focus on how incredible these joins are because that's
50:16
great and everything, but what we're really going to focus on
50:19
are these moments before the moments or in between the
50:22
moments I guess is what you could call it.
50:24
And so it's cool that you also have the ability to think long
50:29
term and say, you know, what's the biggest pain here?
50:33
And it's probably going to take us a year, but we're going to
50:35
figure out this CSV headache and so we go after it.
50:40
Yeah, absolutely. I I think it's a very good way
50:42
of explaining it. I'm actually going to steal that
50:44
metaphor with the concert because I like it.
50:47
Uh, That or it's like it's, I mean, you can also, I think
50:50
airports are also a great example for this.
50:52
Like, you know, the flying is exciting, but like the process
50:55
of getting there, like does impact the overall experience
50:58
quite drastically let's say so. But he's trying to be the Taylor
51:02
Swift of of databases. No I don't think I'm attractive
51:07
enough for that but no but I I think I think it's also like
51:10
it's it's we also talk to people and and we we are not we are not
51:14
the you know the reference if, if, if, if I struggled with
51:17
something it's probably really bad because if you like an
51:20
expert in data management systems struggles with something
51:23
it's not a great look. I have struggled immensely with
51:27
all the database systems out there, which is not a great
51:29
experience. But I mean, we also really love
51:31
to hear from people. So if you're listening and
51:35
something really makes you angry, like to be, we really
51:37
want to hear from you, right? Because because we are not the,
51:40
you know, the absolute reference and what is good or bad and what
51:42
is difficult is not difficult. In fact, we are terrible
51:44
reference. Uh, but at least we try and we
51:47
really need to hear from people where they're struggling with.
51:50
And yesterday, two days ago, I was in Berlin at like a mother
51:54
duck meet up and somebody just walked up to me and said, like,
51:57
listen, here's this thing that really bothers me about that to
52:01
be, can you do something about it?
52:02
And actually this afternoon I was, you know, working on this
52:06
because it matters. It matters that this yet this
52:10
person had a valid point and there's something we can do
52:13
about it. So let's do it right.
52:15
It's. Incredible.
52:17
Yeah. And that is the love for the
52:20
experience. And it's very clear that you
52:23
understand the value in that and that it's almost like now that
52:28
you have it, you can't let it go to waste, right?
52:31
You gotta keep it up. And so that is, that is one of
52:35
the other questions I wanted to ask around.
52:38
How do you go out there and gather feedback and talk to the
52:42
community? It feels like the community is
52:44
gigantic now and they can give you this feedback.
52:49
It's almost like the opposite problem of a company or a open
52:54
source project that is starting where you're just trying to get
52:57
anybody to give you feedback. Now, I imagine you're trying to
53:02
figure out what the biggest signals are, or what the best
53:06
feedback is, or all of that. It's it.
53:09
It seems like it's a a bit of a difficult position to be in.
53:14
It's a big data problem now. But no, in all seriousness we
53:19
are we we have like people in charge of this like this
53:23
deferral people that talk to be lapses Alex and Gabor and and
53:26
they part of their job is to kind of collect these signals
53:30
from the community. And obviously sometimes it
53:32
reaches me, I can't say I read every single issue report or
53:36
every single sort of post on Discord anymore.
53:38
But if things come up often enough it it does leave sort of
53:41
a mark sort of like, Oh yeah, no, this is a common problem.
53:45
These all these like these twenty people have all run into
53:47
the same problem. OK, interesting, right?
53:49
Awesome. Yeah.
53:50
So I think I think we're still at this point because we're
53:54
still this point where people add on report have no issues to
53:58
report back. Like sometimes it's difficult to
54:01
kind of deal with you know yet another sort of issue report
54:04
because our worldview is deeply skewed.
54:06
We only see the problem. So I've never really we don't
54:08
hear about the success story so often, but it is a good sort of
54:12
channel into the into into the users and see where it goes.
54:16
And yes of course you have to understand the same time
54:17
everybody every time that somebody says it didn't work,
54:19
like there have been 10 or 100 people that write on the exact
54:23
same issue and haven't said anything.
54:24
So it's it's an interesting sort of them dynamic I would say.
54:29
And so Jordan getting back to Mother Duck cause thank you for
54:33
your patience while I went on that gigantic tangent from
54:37
Atlantis. But I I do want to know talk to
54:41
us about mother dog and what basically you saw as alright,
54:46
this is mother duck. We feel like this managed
54:50
service can have extra abilities and how you go about building
54:54
those features and why you think those are are useful to have.
55:00
Is it also going out there and talking to people?
55:02
I imagine? Is it what the users are seeing?
55:06
And you had to start from somewhere, right?
55:08
And you came out of stealth not too long ago?
55:11
I was at the party. I can't remember.
55:13
When it was. And yeah, last June.
55:16
Well, so first thing you know Dot TV has set a very high bar
55:19
of you know like so making, making the the end ends you know
55:24
process work and that's why people love Duck TV.
55:27
And if we don't kind of continue that to the the service oriented
55:32
parts of of the product then we're going to be sort of
55:34
letting letting people down and letting down their their
55:37
expectations and things aren't going to work.
55:39
So we have to also make sure that we have very very great
55:43
attention to detail on how do you connect how do you you know
55:47
how do you visualise what are the what are the ways that that
55:51
that mother duck can can work and and so I think we're we're
55:56
spending a lot more time energy and you know and just focus on
56:02
on those parts then then a lot of other companies might.
56:06
I think the the nice part is like we can sort of take the the
56:09
awesome database for granted which is which is hard and like
56:13
you know you know making sure we don't get wrong answers and all
56:15
those kinds of things like we we don't have to focus on those the
56:19
way other you know databases of service companies companies
56:22
might. But I think we're also trying to
56:24
innovate in sort of the delivery of of the of the database and
56:29
and and of the service. Like I think in you know when
56:33
Snowflake and Bigquery you know came out they had a lot of
56:37
things that they were doing differently in terms of
56:39
separation of storage and compute and sort of you know
56:42
simplicity of of you know very few knobs and and how you
56:47
interact with how you interact with them that have started to
56:52
become like standard. You know separation of stories
56:55
and computer is sort of now like now table stakes.
56:58
But I feel like it's been a long time since people have actually
57:01
innovated in in like what are the things you can only do in
57:04
the cloud? And so those are kind of the
57:08
kinds of questions that we're asking ourselves.
57:10
And you know for example, one of those the things that we're
57:13
doing is and it's sort of something that's also special
57:16
with doubts, this duck TB is so lightweight, you know, can scale
57:20
down, has no dependencies the like the JDBC driver which is,
57:24
you know the connector that most code uses to actually talk to
57:28
the database, has the database code inside of it.
57:31
And so you know in the web UI and can actually run Duck B can
57:36
run the database inside the web UI in as as web assembly.
57:40
It can do so like very, very fast.
57:42
And So what are the decisions we made is that our client is
57:45
always going to be Duck B. So every time you're talking to
57:48
Mother Duck, there's a Duck B, there's a Duck to be locally.
57:52
And so all the people that are building connectors to Duck DB
57:55
and all the the ways that Duck TB is being being used, those
57:59
will just work with, those will just work with Mother Duck and
58:02
the other kind of interesting thing that does.
58:04
So if you have a full blown analytics database on your
58:08
client as well as in the cloud, what are the things that become
58:12
enabled by it? Because I think also when I was
58:14
working on Bigquery, there was a bunch of times where we wanted
58:17
to do sort of complex manipulations or interesting
58:20
things client side and we're like we basically just said no,
58:23
no, no, don't do that because you can't like basically clients
58:26
can't be trusted to do hard things and but now we have a
58:29
real database there and these same database that we're running
58:33
in the cloud. And so things like hybrid
58:37
execution where we basically you can join local data against
58:41
remote data. So if you have, like if you're
58:43
running in a Jupyter notebook and you and you're running
58:49
you're running Python And Pandas code, you have a data frame.
58:51
You can join that data frame against you know data that lives
58:54
in the cloud. And we'll make that that join
58:57
optimal. And we will, you know, there's a
59:00
bunch of kind of interesting things we can do on the security
59:02
side. And and then and also just sort
59:07
of building, building incredibly reactive like user user
59:12
interfaces and visualisations where you know if you're running
59:15
the database in your client you can basically do you know 60
59:20
frame per second data visualisations like video games
59:23
style against datasets that actually are quite large, you
59:27
know that won't fit in your browser.
59:29
And like you can do some really clever things.
59:30
And those are just some of the things that we're working on to
59:33
just sort of take Dot B and and kind of supercharge it.
59:37
Because I think Duct B is not a data warehouse.
59:39
It's an amazing you know query engine.
59:41
It's amazing data management system.
59:43
And I think also we are kind of adding the pieces around it from
59:46
like user management and you know you know working on a team
59:51
and and you know larger, larger data sets and and durability and
59:56
time travel and some of those kinds of things that people
59:59
expect now out of out of a data warehouse.
1:00:03
And you know and and sort of layering that on top of the
1:00:06
this, this core Core Data engine while at the same time giving
1:00:08
people the same feel because they have a local Duck B that
1:00:12
you know, it's just like using Duct B but now boom, I get
1:00:16
access to the cloud. And this idea that you're
1:00:21
saying, you started from saying how can we rethink what is
1:00:26
possible now that we have the cloud.
1:00:28
IT again goes back to basically what you were saying honest on
1:00:33
man. The databases have collected
1:00:38
cobwebs over the years and nobody's rethink how we're going
1:00:43
to make this experience better. And so Jordan, like when you're
1:00:48
thinking ahead in the future for Mother Duck and what is possible
1:00:52
because we have the cloud, what are some things that get you
1:00:55
excited? I mean I think some of the the,
1:00:59
you know innovating in the in the delivery, innovating in, in
1:01:02
visualisation innovating in sort of this, this sort of tight
1:01:07
tight coupling where people don't have to know where data
1:01:10
is, where people don't have to know where something gets run.
1:01:14
You know you can basically move, you can move data around and you
1:01:17
can move like to the extent that like sometimes people care where
1:01:20
the data is like you know you know there are laws that say
1:01:23
like you have to process German data in Germany and like
1:01:26
Australian data in Australia. But for people that don't care
1:01:29
like they just want it to be fast.
1:01:31
And so we can actually make sure that things are close to where
1:01:33
the users are running. You know is one is one example
1:01:38
and you know I think there's there's a bunch there's a bunch
1:01:40
of other things that I think we'll we'll hopefully start to
1:01:43
see in in Mother Duck in the in the next upcoming months.
1:01:47
Yeah and but just talking about that, not caring where the data
1:01:50
is, you could still put constraints on it for saying,
1:01:53
hey, I don't care where my data is as long as it's in Germany,
1:01:56
right? Or as long as it's in Australia
1:01:59
and I don't, I don't need to think about that.
1:02:03
But I put my constraints on it in the beginning.
1:02:05
I said it once and then forget it.
1:02:07
Exactly. Rather than saying OK, this
1:02:09
needs to run an EU S4 which happens to be a data centre in
1:02:12
Berlin. Because that just doesn't give
1:02:15
you the flexibility of OK, maybe there's there's other data
1:02:19
centres in Germany and like and some some of your other data is
1:02:22
and that other data set in Germany.
1:02:24
And all of a sudden you have to start paying egress fees because
1:02:26
somebody else is using that other data centre and like
1:02:28
etcetera. Like we we want to just make it
1:02:31
so you know, you set the you set the constraints that you care
1:02:34
about, but it otherwise you shouldn't have to.
1:02:37
You know you shouldn't have to take up mental energy with
1:02:39
having to to think about those things.
1:02:41
So being that the majority of people here that are listening
1:02:46
are probably dealing with machine learning and machine
1:02:50
learning problems, Duck DB and Mother Duck obviously have
1:02:56
thought about the way that machine learning engineers play
1:03:00
with the data and the database. What are some things, and I'll
1:03:03
just loft this up to either of you, What are some things that
1:03:07
you see when it comes to machine learning and the goose buzzword
1:03:12
of them? All AI and all that fun stuff
1:03:16
even dare I say large language models or vectors or any of
1:03:20
those big old buzzwords. But what are the places that
1:03:25
you're seeing machine learning engineers interact with Duct BB
1:03:32
or mother? Yeah, I'm gonna.
1:03:35
I'm gonna skip my usual rant and go straight for we we do.
1:03:40
We do actually, you know, we we, we interact with people that
1:03:44
used to be in sort of machine learning pipelines and there are
1:03:46
several things we do to make their lives better.
1:03:49
Part of it comes from the architecture of the activity
1:03:51
itself because it's in process. So it can be in a Python process
1:03:54
and then you're going to run your model in the same Python
1:03:57
process and hey, your data is already in that process.
1:03:59
Which means we can actually very cheaply ship data back and forth
1:04:02
from any kind of models and then, you know, do things that
1:04:08
the machine learning sort of frameworks are usually terrible
1:04:11
at, like reading or, I don't know, handling updates,
1:04:14
persistence, you know, consistency, that whole stuff.
1:04:17
So that's something I think where we have clear strengths
1:04:19
that can be great benefit to to classical machine learning
1:04:24
workflows. You mentioned vectors, that's
1:04:27
also something you know people are trying to actually people
1:04:30
build vector databases on top of the ECB which is which we love.
1:04:34
And we've recently added a new type product BA data type called
1:04:37
a fixed size list data type because it turns out these
1:04:40
vectors always have the same length and if they have the same
1:04:44
length, which means we don't have to store the length for
1:04:46
every individual vector, which means we can't store it more
1:04:49
efficiently. But that's the kind of the
1:04:51
level, think about it, like what can we do from a basic
1:04:54
infrastructure perspective to integrate to sort of be more
1:04:57
useful to that community. That's kind of our take on it.
1:05:01
We don't, you know, we don't have to integrate a large
1:05:04
language model ourselves. I don't think we want to, but we
1:05:08
can. We can be a building block for
1:05:12
for machine learning sort of communities.
1:05:15
So that's my take on it. I think people in in machine
1:05:18
learning want to integrate interact more closely with their
1:05:24
data. And you know I think we've seen
1:05:27
things like you know B, QML and like and and other other
1:05:31
attempts to sort of like operate over you know do do machine
1:05:35
learning or data that's stored in your in your data management
1:05:38
system. And I think a lot of people
1:05:39
who's like the data lakes and sort of lake, lake houses to to
1:05:44
be able to do that. But I think there's an advantage
1:05:47
to to something like Duck TB or to duck DB which you know in the
1:05:52
fact that it is, it is scale up since a lot of these machine
1:05:54
learning algorithms scale up better than they scale out, you
1:05:58
know they can be very hard to sort of split across you know
1:06:01
multiple multiple machines. And so having a, having a,
1:06:05
having a local scale up database is is very is very useful.
1:06:11
And I guess the other thing is kind of the the lingua franca of
1:06:14
of AI and ML is is is Python And you know duck TB has extremely
1:06:19
good Python integration and some you know special things that you
1:06:23
know you can do in Python with the duck TB relational
1:06:26
relational API that that can sort of turn into either pandas
1:06:31
operations or Spark operations. They kind of make it easier to
1:06:37
work with to work with your data that's stored the stored in a
1:06:39
database. And they finally, I'll add that
1:06:42
you know mother dog we are adding some, you know, LM
1:06:45
features text to SQL, text to SQL stuff.
1:06:48
People are changing, you know data data analysts are changing
1:06:52
how they're, how they're interacting with data with you
1:06:54
know the advent of LMS. And so I think we would be remiss if we
1:06:58
weren't at least sort of investing, investing in that
1:07:00
area. We're also working on a a joint
1:07:05
duck TB specific LLM, you know, tuned fine-tuned LM with a
1:07:11
company called Number Station AI Ohh I.
1:07:13
Was. This we're gonna open source at
1:07:17
least at least one one version of that we've been we've been
1:07:20
hard, hard working on and so we do want to also make sure that
1:07:22
we're giving back to to the community.
1:07:25
Incredible, yeah. Diego, who is I think the head
1:07:27
of product there, has been in the community for ages since the
1:07:32
beginning. And also Inez has come on here a
1:07:34
bunch and I love number Station, love what they're doing and I'm
1:07:39
very happy to hear that you are creating your own text to sequel
1:07:43
model. That was one of the questions I
1:07:45
was gonna ask. I was like, ohh, are you using
1:07:46
the open source number station or are you gonna be what is it
1:07:50
the I can't Remember Q sequel or something.
1:07:53
I can't remember what their open source foundational model is
1:07:57
called, but it's very nice to see that.
1:08:01
And fellas, this has been awesome and I'm so happy that
1:08:08
you did this. What do you think?
1:08:09
Did the mushrooms work? Was I asking good questions?
1:08:12
Was a great questions that you had great questions.
1:08:15
Thanks for having us. Excellent.
1:08:17
Yeah. Thanks so much.
1:08:22
Yeah. Hey everyone.
1:08:23
My name is Aparna, founder of Verizon, and the best way to
1:08:26
stay up to date with Emma Lops is by subscribing to this
1:08:29
podcast.