Latest / Agentic Conversations (formally mlops.community) / The DuckLake Lakehouse Format // Hannes Mühleisen // #339

Transcript

The DuckLake Lakehouse Format // Hannes Mühleisen // #339 · Agentic Conversations (formally mlops.community)

• 0:00
  If you're used to, oh, I have to set up these credentials and
• 0:04
  those credentials and this Docker container and and this, I
• 0:09
  don't know, firewall rule and all these things, right?
• 0:14
  If you're used to that and it's something just goes, yeah, I've
• 0:18
  done it. It's spooky.
• 0:21
  It can't. It feels to me.
• 0:23
  It feels very spooky. I heard some crap where like
• 0:32
  people if they find treasure it, they have to like give it to
• 0:36
  Spain because it was insured back in like the 1600s and then
• 0:40
  the boat went under when they were taking it across the ocean.
• 0:46
  And so like, it's rightfully Spain's.
• 0:48
  Yeah, there there is some interesting things.
• 0:50
  There is a court in London, the Arbitration Court for maritime
• 0:53
  sort of events and, and obviously it precedes
• 0:57
  international law, but they had a way of dealing with these
• 1:00
  things, like when it's still around, it's very fascinating.
• 1:03
  Yeah, that is so wild just to think like, yeah, this hasn't
• 1:07
  been anybody's for hundreds of years.
• 1:09
  And I did all the work or I got lucky and I found it.
• 1:13
  But now I got to forfeit that because, you know, 10
• 1:17
  generations ago somebody's. Yeah.
• 1:21
  Took it all the way across the ocean that is too.
• 1:23
  Wide I think I think some some treasure hunters will just keep
• 1:26
  it quiet right and be like hey you know nothing to see here
• 1:30
  it's star boats and we. Found a bracelet.
• 1:34
  That's it. No gold bars.
• 1:36
  Maybe now, yeah. Exactly.
• 1:38
  I think, I think this has happened plenty of times that
• 1:40
  later on people realized that they had been finding stuff and
• 1:43
  not oh. Yeah.
• 1:45
  And then too late. So man, I, I wanted to mention
• 1:50
  this to you because you have just been on fire since we last
• 1:54
  talked. We chatted with Jordan and you
• 1:56
  probably two years ago now and Jordan was doing mother duck,
• 2:00
  you're doing Duck DB obviously. But recently, I think in the
• 2:05
  last three months, it's just gone parabolic because of Duck
• 2:09
  Lake, right? That is great.
• 2:12
  Excellent to hear that's this is our our evil plan finally coming
• 2:15
  together, right? All that quacking, yeah.
• 2:19
  Yeah, yeah, yeah, yeah. You know, they laughed at us
• 2:21
  now, you know, and now we quack at them.
• 2:23
  But no, it's, it's, it's interesting.
• 2:26
  I, I mean, in a, in a way it, is it, it, the duck leg is
• 2:31
  something that does fix a architectural sort of pattern
• 2:38
  that was problematic with duck to be in the past.
• 2:40
  And now it's possible, which is like, if you what if you have
• 2:43
  many, many computers trying to access data at the same time.
• 2:46
  So that's, that's really for us. This is the exciting part,
• 2:50
  right? It's not so much, oh, we made
• 2:51
  another format. It's like, OK, great.
• 2:55
  But more is like, no, you can actually now do something with,
• 2:58
  with duck TB. That was very difficult in the
• 3:00
  past. And so this headache, how were
• 3:05
  you seeing it? Like how did you get the
• 3:06
  inspiration to fix the headache? Yeah, that's super interesting.
• 3:10
  I was actually in being interviewed about something
• 3:14
  else. No.
• 3:16
  By Joe Joe The. Light bulb went off.
• 3:18
  Yeah, yeah. And it was, it was interesting.
• 3:20
  It was at A at a conference in Paris and I was sitting there
• 3:23
  and we were talking about something unrelated.
• 3:25
  And. And then, you know, you talk
• 3:26
  about it and it goes like grumble, grumble, grumble.
• 3:29
  And. And maybe if you watch the
• 3:31
  recording, you can see the moment where I'm like quiet for
• 3:34
  a second. But that was that was the
• 3:39
  original idea. And then I came back and started
• 3:43
  talking to Mark about it, and I wasn't actually so excited about
• 3:46
  it in the beginning. But what was this?
• 3:48
  Just like they can't reach duck DB or multiple computers can't
• 3:55
  reach it or. No, it, it started from another
• 3:57
  place. It started from us looking at
• 4:00
  basically the way the industry was heading with the lake house
• 4:03
  formats with iceberg and Delta. And we and then especially this
• 4:08
  development to slap a catalog server on top of these things in
• 4:13
  order to, you know, be able to deal with multiple tables, deal
• 4:16
  with, you know, transactional guarantees that that
• 4:19
  development. And we've been watching this and
• 4:21
  we've been a bit like really and, and, and then.
• 4:28
  It seemed like an anti pattern for you.
• 4:30
  That isn't it was not an anti pattern.
• 4:32
  It was not. Well, there's parts of it that
• 4:35
  looked like an anti pattern, but the what was interesting for us
• 4:39
  was that these, these lake house formats, they like they were
• 4:42
  started life as something that lived on object storage only,
• 4:47
  right? And they had, and the proponents
• 4:49
  of the people that designed these formats had always said
• 4:52
  no, like there should not be any services involved here.
• 4:55
  This should not be a database specifically involved in this.
• 4:59
  This needs to be based on the files.
• 5:01
  And then later on it turned out that that isn't really possible
• 5:05
  for many reasons. And then they put these catalog
• 5:11
  servers on top. But what people forget is that
• 5:13
  at the heart of this catalog server is a database, usually
• 5:17
  just postcards. And then what occurred to me is
• 5:21
  that if it's already in the syntax in, in, in the sort of
• 5:25
  the architecture diagram, we could, we might as well use it
• 5:28
  for more things, right? And that, that was the original
• 5:31
  idea, right? And so that with that idea, I
• 5:33
  said, hey, let's and there's a drawing somewhere in the office
• 5:35
  where I just. You scribble the duck.
• 5:38
  Yeah, it's a scribble, Scribble. The duck.
• 5:41
  Yeah, it's I. Think, I think one of I think
• 5:45
  Pedro has kind of collected it for immortalized, yeah, for
• 5:48
  future use or something. But.
• 5:52
  And I came to this idea and I went to Mark and said, hey,
• 5:54
  look, how about this? And and then, you know, a week
• 5:58
  later or so, he comes back and says we need to do this.
• 6:00
  And I was like, whoa, this, you know why?
• 6:03
  And then it was actually he had realized that that having a set
• 6:10
  up like that with with, you know, like your data files being
• 6:14
  an object store, your metadata server being a database that
• 6:19
  lives separately from maybe your Duck DB instance.
• 6:22
  With that set up, you can actually solve them.
• 6:25
  So if the multiplayer use case of Duck DB, so maybe to explain
• 6:29
  in in Duck DB it works a lot like Sequel Light usually where
• 6:34
  you have a file, you point it to the file, it does things if you
• 6:39
  want to change this file. So normal Duck DB before duck
• 6:42
  lag would actually lock that file and it's still multi
• 6:46
  threaded and you can throw multiple parallel queries at it.
• 6:49
  But one process only could have that file.
• 6:52
  And that created some issues with people that wanted to, I
• 6:55
  don't know, have multiple computers working on queries at
• 6:58
  the same time, these kind of things.
• 7:01
  But with with duck leg, that's actually fixed because now
• 7:04
  there's no real limitation on on the amount of computers that can
• 7:08
  participate in this anymore because there's not a single
• 7:10
  file. There's a metadata service and
• 7:13
  there's files on object store, which is a very different set
• 7:16
  up, but it does fix this problem.
• 7:17
  And so that was the, that was the realization that Mark had
• 7:21
  was that that fixes this use case.
• 7:25
  And so that I think for us was the motivation to really push
• 7:28
  through with it. It wasn't like, because in, in
• 7:30
  general, we don't, we don't really, we're not really setting
• 7:33
  out to reinvent things that exist, right?
• 7:36
  Like that's, that's not really our.
• 7:40
  Especially something surprisingly that has so much
• 7:42
  adoption already, like an iceberg or.
• 7:45
  Delta or hood? All that stuff is like, oh, I
• 7:47
  thought that was solved. And then you come out with duck
• 7:50
  leg and I'm just like, whoa. But it instantly did you see it
• 7:56
  catch fire right now? Yeah, yeah, it's, it's it's
• 7:59
  quite amazing, I have to say from, from this being something
• 8:04
  like we published this, I don't know, a couple of months ago, I
• 8:06
  don't know, three months ago. And from day one, I started
• 8:13
  getting these enthusiastic emails from people, like people
• 8:15
  normally to explain, normally I don't get a lot of emails from
• 8:19
  the random people on the Internet, right.
• 8:21
  Like that is not a thing, but it started, it started this like
• 8:26
  these emails started coming like this is amazing, this is awesome
• 8:29
  and so on and so forth. And I thought, oh, that's
• 8:31
  interesting. And, and what is what I think
• 8:34
  what, what surprised me more was that I mean, at this point we're
• 8:39
  at duck like 0.2, right? It's a fairly low version
• 8:42
  number. It should be clear to everyone
• 8:43
  that this is fairly early stage stuff, right?
• 8:47
  But we are already talking to several, several organizations
• 8:51
  that are running this in production are perfectly happy,
• 8:53
  right? And I'm like, I'm like, this is
• 8:56
  the IT took Duck DB years to get to that point of adoption where
• 8:59
  people would actually bet the farm on it, right if.
• 9:02
  You want to. But but then that for this to
• 9:05
  happen within a month or two for for duck leg, that was pretty
• 9:08
  crazy. That like signals to me that
• 9:11
  either they already had the trust because of duck DB and so
• 9:16
  they were more willing to do that, or the pain was so great
• 9:21
  that this solved they instantly jumped on it.
• 9:25
  Do you have an intuition of which one it is?
• 9:29
  I think there were definitely some doc DB people that saw this
• 9:32
  as, as I said, as something that fixes something in the in the
• 9:36
  setup for them. But I think what I see, what I,
• 9:39
  what is this in my impression, what I see more is that people
• 9:43
  look at the architecture and they get it.
• 9:47
  It's, it's, it's you can intuitively understand the
• 9:50
  architecture of Duck Leg. It's fairly straightforward.
• 9:53
  You have your objects store, you have your metadata store.
• 9:56
  And then, you know, you talk to these things and it's, it's,
• 10:00
  it's not, it's not like 15 hops of XMLRPC or I don't know, you
• 10:06
  know, like there's, there's not, there's not, there's no, there's
• 10:09
  no, there's no Jason involved. There is, there is, there's,
• 10:15
  there's just these two things. And people know how to talk to
• 10:18
  databases. People have been talking to
• 10:20
  databases for 50 years. People know how to talk to
• 10:22
  object storage. It's I think the first service
• 10:25
  that Amazon pushed out ever right it?
• 10:28
  Wasn't called S3, but yeah I I think I heard that too.
• 10:31
  I. Feel like storage was was BLOB
• 10:32
  store it was easy 2 or S3 one of the two I don't remember and and
• 10:38
  so people know how to talk to BLOB storage right.
• 10:41
  So it is, it is in a way the, it's I think also something that
• 10:45
  we really like here at Adaptively Labs is we, we, we
• 10:50
  strive for simplicity in design, right?
• 10:52
  Like anything, anything is anything.
• 10:54
  We look, we look at everything and say, OK, this, this is like
• 10:57
  this. Does it, you know, does it have
• 10:59
  to be like this is, can it be simplified?
• 11:02
  Can we RIP it in half? And maybe, you know, it's like
• 11:07
  we, we have, we have a very, very strong sort of sense of
• 11:10
  aesthetics in, in, in, in design of software.
• 11:15
  And, and at least that's my impression.
• 11:17
  And I think it's shared within our team.
• 11:19
  And, and so, so that's why when, when we started thinking about
• 11:23
  what Duck Lake should look like, it was clear it needs to be
• 11:26
  absolutely minimal. And, and it's like, you know,
• 11:29
  like a, like an Apple product if you want, right in a very, in a
• 11:33
  much uglier space. I I fully admit, but it's it's
• 11:37
  like it has not something that really always gets to me.
• 11:40
  But there's one, there's one thing where where at Apple where
• 11:43
  they show like ATV remote and it has 100 buttons and it's ugly as
• 11:47
  hell. And and then they show that.
• 11:51
  I think at the time it was the, the I book remote.
• 11:55
  There was a thing at some point, remember?
• 11:57
  They definitely have had a few remotes.
• 12:00
  And it has like 11 button and the circle around it.
• 12:03
  And they showed this next to each other.
• 12:05
  It was maybe Apple TV, maybe it was Apple TV, doesn't matter.
• 12:08
  And you look at that and you think, yes, obviously this is
• 12:11
  the one I want. And that's how we look at
• 12:14
  technology, right? So it's a absolute, it needs to
• 12:17
  be, it needs to be, you need to be able to look at it and say,
• 12:20
  yes, this is, this is the way it has to be.
• 12:22
  And it's also just to protect ourselves.
• 12:24
  Like we are not smarter people or anything like that.
• 12:27
  Like we need to understand the stuff we're building.
• 12:29
  So if it's simple, it's, it's easier, right?
• 12:31
  It's, it's, it's, I think that's how this design came about.
• 12:36
  And it's, yeah, it's, it's worked out.
• 12:38
  Like also in the beginning, who knows?
• 12:39
  You think, oh, this could work. And then, you know, you, you,
• 12:43
  you go half a year into developing it and you end up
• 12:46
  with something that you can maybe run a test on and you go.
• 12:50
  Yeah. This works, not.
• 12:52
  Quite how I had it in my head, but I guess it works so.
• 12:55
  No, no, no, it it, it worked the way we had it in mind.
• 12:57
  There was no that that it it did work it we were amazed when it
• 13:01
  it's it's actually, I think using duck like it.
• 13:03
  I don't know if you had a chance of, of playing with it yet, but
• 13:06
  it's I, I, I feel it's always spooky because it's like it's
• 13:12
  very easy to get up and running. You install duct DB.
• 13:14
  OK, it's generally easy to do. You have to basically type
• 13:19
  install duck like in your duck DB shell.
• 13:21
  And then that's and then you can start.
• 13:23
  Using it. And that's, it's spooky, right?
• 13:26
  If you used to, oh, I have to set up these credentials and
• 13:30
  those credentials and this Docker container and and this, I
• 13:35
  don't know, firewall rule and all these things, right.
• 13:41
  If you're used to that and it's something just goes, yeah, I've
• 13:44
  done it. It's, it's spooky.
• 13:47
  It can't. It feels to me.
• 13:49
  It feels very spooky. Yeah, you don't trust it because
• 13:51
  it's almost too easy. It's like it's too easy.
• 13:55
  When we, when we try our, our stuff, right, like we tried and
• 13:57
  we go like, Oh, there's no way this just did something right.
• 14:00
  This was, this is like, I expected this to, to, to, to
• 14:03
  scream at me or something, but it, it, no, it, it really is.
• 14:07
  It really is interesting to I, I find it really fun to, to
• 14:10
  interact with myself. It's also something we care
• 14:12
  about right at database developers usually don't get to
• 14:18
  use that stuff themselves. It's there's a bit of a
• 14:21
  disconnect between users and and developers and we really think
• 14:26
  that we really try to use our own stuff, let's just say.
• 14:31
  Overtime companies tend to or projects in general tend to get
• 14:38
  more requests from users and then they bloat.
• 14:43
  And I think that you have done a really good job of focusing on
• 14:47
  the really important stuff to create that minimal aesthetic in
• 14:51
  a way. How do you?
• 14:54
  Prioritize that. Yeah, it's a, it's an excellent
• 14:57
  question. And I do observe this as well.
• 15:00
  And there's, I think there's two things about it.
• 15:03
  One is the governance model of duck to be, isn't, isn't, I'm
• 15:08
  sorry to say it's not a democracy, right.
• 15:11
  So we've seen, we've seen lots of specifically Apache projects
• 15:17
  get to this feature bloat because there's a committee,
• 15:21
  there's maybe 15 committers or something like that.
• 15:24
  There is, you know, there, there has to be some level of
• 15:27
  consensus between them. But if somebody says, I strongly
• 15:30
  feel about this, this should not go in.
• 15:32
  That's not really the way you do it in these committees.
• 15:34
  You try to make everyone happy. And that's also what I would do.
• 15:37
  I mean, it's totally human nature to try to accommodate
• 15:41
  everyone's request, but the result and then it's like it's
• 15:45
  just one additional dependency. You know, it's not so bad, you
• 15:51
  know Yeah, exactly. Famous last words.
• 15:54
  So we have we don't have that. We have we have mark and and
• 15:58
  myself. We we decide what what happens
• 16:00
  in duck TB. We we have a we have a vision, a
• 16:03
  product vision, if you want right, the make of road map.
• 16:07
  Obviously we do think about what people want.
• 16:10
  We do look at, you know, what's goes on in our issue tracker and
• 16:14
  things like that. What what are the things people
• 16:16
  want? We but, but in principle it's,
• 16:20
  it's not, it's not like a it's not, it's not a, it's not a
• 16:25
  committee thing. It's not, it's not a design by
• 16:27
  committee. Like this is the anti pattern,
• 16:28
  right? Like it's not designed by
• 16:30
  committee. It's designed by people that
• 16:32
  agree on, on, on a sort of vision for the thing.
• 16:34
  And I think another thing that we have done early on in duct DB
• 16:37
  is make this extension system where we say, OK, there's duck
• 16:41
  DB, it runs sequel if you want Duck DB to talk to Kafka or
• 16:47
  something like that. We don't bake that into the
• 16:50
  system. We can have an extension that we
• 16:53
  have this whole elaborate expansion system where you can
• 16:56
  basically say install something. And I said this example, I
• 17:00
  mentioned this example about Duck Lake.
• 17:01
  You can say so Duck Lake and it will automatically magically,
• 17:05
  you know, fetch an extension, install it, load it, make it, if
• 17:11
  you know, rig it into the system.
• 17:13
  And then and so that's also a way of keeping the system sort
• 17:16
  of clean, right, where you say everything that's like really
• 17:19
  not strictly core has to live in an extension.
• 17:22
  And and that's that's also how we have the no dependencies in
• 17:27
  the in the in the in the standard install, right?
• 17:30
  Like it's all. Everything that does have
• 17:32
  extension dependencies is an extension.
• 17:35
  It's almost like you took this approach of like the App Store
• 17:39
  and you said, OK, if you want to run different things, then go
• 17:43
  out and get yourself an app and download that app and you can
• 17:46
  run it on the phone of your choosing.
• 17:50
  But I do still see, especially with companies that have quote
• 17:55
  UN quote, but emulated dictators and not necessarily open source
• 17:58
  projects, the companies will start with this really slick
• 18:03
  fine thing. And I'm thinking specifically of
• 18:06
  like Versailles, how they came out and they were, which in
• 18:10
  this, you know, almost like we're going to make it so easy
• 18:13
  for you to do what you want to do.
• 18:16
  And then over time, I've heard people start complaining now
• 18:20
  that Versaille is like this gigantic piece of software that
• 18:25
  is nowhere near identical to what it originally set out to
• 18:29
  be. And it's like there's this
• 18:32
  tension that potentially because they are AVC backed company,
• 18:38
  they have to. They need.
• 18:41
  That market dominance. And so they're going to continue
• 18:46
  to add features and you end up looking like Microsoft Word at
• 18:50
  the end of the day, where it's just like, Oh my God, I don't
• 18:53
  know how to do anything because of this.
• 18:55
  Top bar has so many damn icons and I don't know what any of
• 18:59
  them mean. Nah, I think you're absolutely
• 19:02
  right. And by the way, who says
• 19:03
  benevolent the rather bear this Nobody, nobody.
• 19:06
  This is not. Anyways, but I think I think
• 19:09
  something I always I always use as a word.
• 19:11
  I it comes to my mind as was always GitHub, right?
• 19:14
  Like GitHub is always gaining a button a month or something and
• 19:17
  and then it's always like clearly some enterprise customer
• 19:21
  needed this, but I don't need this.
• 19:22
  And it's like, I mean, I. I am hard pressed to think of a
• 19:28
  company or a project that doesn't do this.
• 19:31
  Yeah. I think we are, yeah.
• 19:33
  I think, I think you have a point there with the VC funding.
• 19:36
  And I think what's also what's also happens to companies is
• 19:39
  that at some point they have a sales team.
• 19:42
  Sales team wants to sell, sales team goes to customer and says,
• 19:45
  Hey, you want to buy our thing? They say no, because you don't
• 19:47
  have feature XYZ. They go back to dev team and say
• 19:51
  we need XYZ tomorrow. And then it like who in the
• 19:55
  company will say no, right? Like the CEO wants to make more
• 19:58
  money, right? The sales team wants to sell.
• 20:01
  Are the developers going to say no?
• 20:03
  That's really not their position to be in, right?
• 20:07
  So, so it's I think for for us in inductive labs, I mean, the
• 20:10
  company is not we see funded as you know, do we say we say no a
• 20:16
  lot? It's it's.
• 20:19
  It's almost like a. Luxury that you have, we are in
• 20:22
  a very. Luxurious position and we are
• 20:23
  very grateful to the people that we work with to basically listen
• 20:30
  to us when we say, look, we maybe we can do it another way
• 20:34
  that works for everyone. And it does not, you know,
• 20:38
  create a whole lot of sort of technical implications.
• 20:41
  And very often we will say, OK, this is not, we don't want to do
• 20:45
  that. However, we're going to add a
• 20:47
  hook. So if you want to make an
• 20:49
  extension that does that and changes the behaviour, fine,
• 20:54
  right, that that's up to you then.
• 20:57
  And so that's, that's the way we've been able to get around
• 21:00
  this. And we've actually been pushing
• 21:01
  this pretty pretty far. We've we've currently, if you
• 21:04
  have a somebody working here, Daniel, that's working on a, on
• 21:08
  a pluggable sequel parser where extensions can register new
• 21:13
  syntax at runtime. And it's pretty wild.
• 21:15
  And we're doing this so we don't have to Add all that stuff to
• 21:19
  the SQL syntax because we don't want to.
• 21:21
  But other people then can do that, right?
• 21:23
  Like if somebody wants to do prequel, it's one of these new
• 21:25
  dialects people seem to like, great, you can make an extension
• 21:29
  that adds the syntax and everyone's happy.
• 21:31
  Go wild. So that's, that's, that's kind
• 21:34
  of how we do this. And then also like in, in Duck
• 21:36
  Lake, right? Like we haven't changed, we have
• 21:40
  changed minor things here and there in Ducktb itself to make
• 21:43
  Duck Lake possible. But the Duck Lake
• 21:45
  implementation, so this may be to explain duck like has, of
• 21:49
  course, a specification which is on the Internet.
• 21:52
  And there's also an implementation of this
• 21:53
  specification, which is the duck to be duck like extension,
• 21:56
  right? That is an extension, right?
• 21:58
  We haven't just committed a bunch of code into into duck to
• 22:02
  be and say now it can do duck like.
• 22:03
  No, it is also an extension like everything else, right?
• 22:06
  It's an extension actually uses a bunch of other extensions as
• 22:09
  well. So we have we can have
• 22:10
  extensions using other extensions.
• 22:11
  So the Duck like extension will, for example use the Parquet
• 22:15
  extension to read and write Parquet faults, OK.
• 22:18
  And it will also use the database integrations with other
• 22:22
  systems like Postgres, MySQL, Silverlight and so on that we
• 22:26
  have to talk to the metadata server.
• 22:28
  So in the end, the Duckling extension itself isn't actually
• 22:32
  a whole. It's actually not huge.
• 22:33
  It's just saying it's talking to everything and telling things
• 22:37
  what to do and it's followed. Your own rules, yeah.
• 22:44
  Everybody else has to. Do it.
• 22:45
  We should probably have to do it.
• 22:47
  It's, it's very tempting to, to work around these rules, but so
• 22:50
  far we have managed not to do this.
• 22:52
  And I think, I think it's really interesting to I mean, Doctor B
• 22:56
  has been around for for seven years, I think at this point.
• 23:01
  And yeah, we haven't, we haven't had to fundamentally change
• 23:06
  this, this approach. I think the, the one of the,
• 23:09
  there's two sort of milestones in the development that I would
• 23:12
  say 1 is the, the extensions like that, that everybody, we
• 23:15
  have community extensions, everybody can make extensions.
• 23:18
  And the other one is, would be I think Duck Lake because it does
• 23:20
  open up a whole, a whole new sort of set of use cases.
• 23:24
  But what are the? Values that you specifically
• 23:27
  adhere by when you are building or the.
• 23:30
  Values not. Maybe it's.
• 23:32
  Values is a very strong word, but I know that there tends to
• 23:35
  be like the principles that you build from and I think just from
• 23:41
  talking to you these few times, 1 is that developer experience
• 23:46
  and that some simplicity is probably one.
• 23:49
  What are other ones? I think, I think this is really,
• 23:53
  this is really simplicity is a is a big one and it does have a
• 23:57
  huge amount of implications, right.
• 24:00
  So it so simplicity has implications on your delivery.
• 24:05
  It has implications on your installation process.
• 24:07
  It has implications on your SQL dialect if your SQL system
• 24:12
  congestion. CSV parser.
• 24:15
  Right, like we have this crazy CSV parser that tries to infer
• 24:20
  things. That's it's actually there's
• 24:22
  somebody did a paper recently and found that ours is ours is
• 24:25
  both the most flexible and the fastest, which is like, so
• 24:29
  that's that's great. So the simplicity is really is
• 24:32
  really, really a base thing. What I think the simplicity
• 24:36
  alone isn't isn't enough to give you a good experience, right.
• 24:40
  So another thing that comes in there is, is for example, things
• 24:43
  like reliability, right when we say databases are, and I think
• 24:49
  I've used to used this in comparison before.
• 24:51
  They're not like Eck timers. There are databases are systems
• 24:54
  where people build systems on top of.
• 24:57
  So ideally you never, it's like ideally you never hear of them,
• 25:00
  right? Like if you hear from the
• 25:01
  database up top is like the something went wrong, right?
• 25:05
  Something something that shouldn't have happened.
• 25:07
  And it's usually very hard to recover from database errors.
• 25:12
  And so I think that's also something that we've been
• 25:14
  spending a lot of time on behind the scenes.
• 25:16
  So for example, one of our guys allowance he's worked the past
• 25:21
  five years or something on is larger than memory computation
• 25:25
  inductive B. So where it's like if your query
• 25:28
  creates an intermediate result, for example, a join or an
• 25:34
  aggregation that is bigger than URM, what happens then, right.
• 25:38
  So most systems actually just say now, yeah, that's correct,
• 25:43
  let's not do that. But we realize that that isn't
• 25:47
  that isn't what you need to do. You need to be you need to be
• 25:49
  basically reliable and and and people need to be able to rely
• 25:53
  on you that you can finish this query.
• 25:55
  So, so that's a bit has been a big focus in the past couple of
• 25:59
  years. And we're, you know, we're we're
• 26:01
  doing lots of things with memory management also generally code,
• 26:05
  right. We don't want surprising errors
• 26:09
  when you've used that to be. So we're we're actually, we are
• 26:12
  very grateful that there is I think there's like at least
• 26:15
  three universities at the moment that are running experimental
• 26:18
  sequel fuzzers on top of that TV, which is really cool, right?
• 26:24
  Shout out to Manuel at this point, which is basically again,
• 26:30
  helping us to to increase robustness of the system by
• 26:33
  catching weird sequel corner cases.
• 26:35
  And I can assure you there are many of them.
• 26:39
  But then we can handle them in a sane way.
• 26:41
  And yeah, so that's that's a big, that's a big aspect as
• 26:45
  well. But when you say.
• 26:47
  Reliability, you're talking about the reliability of the
• 26:51
  package, of this open source package, right?
• 26:53
  It's not the reliability of me and my system that I've set up,
• 26:58
  because usually when I think of reliability, I think of somebody
• 27:01
  who's got their company or their SAS tool and there's one piece
• 27:07
  of it is this one piece of it is that.
• 27:10
  And that whole system needs to be reliable.
• 27:12
  You're talking about actually like the product itself, the
• 27:16
  open source database needs to be reliable.
• 27:18
  And so you're doing a lot of it's, it's not necessarily
• 27:22
  reliability in the way that I think of it.
• 27:24
  That's. Right.
• 27:25
  It's not the five nines or whatever.
• 27:28
  Yeah, it's the, it's the, I don't know how many, how many
• 27:32
  nines you have to have. I'm, I'm not, I'm not, not in
• 27:34
  that world. Many, but it's, it's more like,
• 27:40
  it's more like you're using a piece of software.
• 27:42
  It needs to work and it's, it's, it's actually we, we, we have a
• 27:45
  quite hard job there because many people use duck TB and then
• 27:48
  they let their users throw queries at us.
• 27:51
  It's like, OK, so we so, so we don't know these queries before
• 27:56
  and they don't know these queries before and either, but
• 27:58
  they have to sort of be able to trust that we will do the right
• 28:03
  thing TM with those queries will not, you know, will not crash or
• 28:09
  will not, will not run out of memory, will not, you know, just
• 28:13
  do something sane. We'll somehow be able to balance
• 28:16
  the memory requirements between multiple queries running at the
• 28:19
  same time. There's all sorts of wild things
• 28:21
  that we didn't think would be a thing back when we started, but
• 28:26
  they are absolutely important. So we are spending a lot of time
• 28:29
  on that also on our CI set up, right.
• 28:32
  Like I think we are very grateful to Microsoft for
• 28:35
  sponsoring our, our, our CI hours, which is like, I think
• 28:38
  every commit is, is, I don't know, 10s or I think we're
• 28:44
  getting towards 100 CI hours for every commit.
• 28:47
  Yeah, I mean in. In total, right, Like it's, it's
• 28:51
  end to end. It's like a cup.
• 28:52
  It's like maybe an hour and a half, but it's, it's a lot at
• 28:57
  this is this like this because we run millions of queries in
• 29:00
  that in that time on different platforms with different
• 29:02
  compilers and different operating systems with different
• 29:04
  configurations with different. And that's, that's something
• 29:08
  that that that I think it has to that we realized very, very
• 29:12
  early on in the before we even started programming.
• 29:14
  That was the way that we would have to do this because that to
• 29:18
  be came out of a research lab and and software that comes out
• 29:21
  of research labs usually not really ready for prime time, But
• 29:27
  we from the beginning on, we we decided that that was that was
• 29:30
  going to be a big, big factor. So that reliability guess speed
• 29:36
  is also one. I think it's fun.
• 29:38
  It's really fun to make something fast, right?
• 29:40
  Like I feel like I've I've recently I recently done a demo
• 29:45
  on this. It's like, OK, so here's a
• 29:48
  computation Dr. speed and we'll count the we'll count the the
• 29:52
  the time, we'll measure the time it takes.
• 29:55
  Then we'll look at how many CPU cycles does the CPU have and how
• 29:59
  does this match up? And we got, you know, within 5%
• 30:02
  of the theoretical maximum, which is really cool, right?
• 30:05
  If you can show that, look, there's no way how you can do
• 30:10
  this faster. We are at the absolute limit of
• 30:13
  this, of this machine. I mean, that's also something
• 30:15
  that I really don't care about speed a whole lot.
• 30:18
  Like I, I, I drive a bike and I have very slow boat, but but on,
• 30:23
  on, on, on computers, I feel like it's such a, it's such a
• 30:26
  cool thing because it's an enabler, right?
• 30:29
  It's an enabler that allows you to do more with, with, with less
• 30:33
  resources. And I think it's not something
• 30:35
  that we've been banging on a whole lot about, but if you can
• 30:39
  use a smaller computer, it's less power.
• 30:41
  It's like everyone's happy. There's no bad things about
• 30:44
  this. It's faster, it's cheaper, it's,
• 30:46
  you know, greener, it's everything.
• 30:49
  And so that's, that's a really worthwhile of pursuit of speed,
• 30:52
  right? Like if you can make a Formula
• 30:54
  One car faster, the planet or there's side effects, there's
• 31:00
  side effects and there's also the, the, the goal is really not
• 31:03
  changing a whole lot. It's not like we can drive to
• 31:05
  the the work faster because somebody made a car faster.
• 31:11
  Right. No, it's you can't.
• 31:13
  There's a speed limit, but but or traffic, but you can
• 31:17
  absolutely improve somebody's experience and actually empower
• 31:21
  them to do more. When when you make the database
• 31:23
  twice as fast, it's it's, it's, it's pretty impressive, right?
• 31:26
  Like suddenly you can do it on a laptop, something that didn't
• 31:29
  fit on a laptop, you can do it on a phone, you can.
• 31:32
  So that's, I think really something that at that level of
• 31:36
  speed, I, I really like it's because it translates directly
• 31:39
  to, to enabling, enabling people to do things.
• 31:43
  And, and so that's, yeah, that's really, that's really fun.
• 31:46
  Do you think? That you, by the way that you
• 31:50
  set up these extensions and almost like the raw materials
• 31:54
  that you give people, whether it's reliability or speed.
• 31:58
  You're. Almost tempting them to be.
• 32:02
  Creative. With how they use it, because I
• 32:05
  think one of the talks that you gave last year at one of our
• 32:10
  virtual conferences was all on different ways that you've been
• 32:13
  seeing really cool uses of Duck DB.
• 32:16
  And when I hear you break this down, it makes me feel like, Oh
• 32:22
  yeah, well, of course, you know what?
• 32:24
  You're just giving them raw materials and saying like, go
• 32:27
  and make your Lego house. Yeah, for sure, yeah, I think I
• 32:32
  think it's absolutely amazing what people are doing and it's
• 32:36
  it's really fun to see and it's interesting because I feel I
• 32:41
  also like the to know what to in order for something to be part
• 32:45
  of somebody's Lego house, it needs to behave like a Lego
• 32:47
  brick. And what does that mean?
• 32:49
  Right. It means it needs to be kind of
• 32:50
  self-contained, not not something that not too
• 32:54
  expensive. And then I think for Duck to be
• 32:56
  what this translates very well into our into how we set up,
• 33:00
  right? If you're Duckdb is something
• 33:02
  that you can integrate into something.
• 33:04
  You don't have to run a separate cluster to run Duckdb.
• 33:06
  It's just you throw it in your application.
• 33:08
  We actually have tons of people that ship Duckdb as part of like
• 33:11
  a electron app, right? Not a problem.
• 33:15
  It's a node package. You drop the node package into
• 33:17
  your app. Done right, you have tons of
• 33:20
  people that ship Duckdb as part of a Java app because we have a
• 33:23
  JDBC driver that runs on all the platforms, you know, you just
• 33:26
  pull it in and and so I think Doc DB's design itself also
• 33:31
  lends itself to being a Lego brick because it is small and
• 33:35
  contained and, you know, doesn't.
• 33:37
  Yeah, plugs in well. With other plugs in.
• 33:42
  Right, does have all the interop.
• 33:43
  If you can talk to the files, can talk to Arrow.
• 33:46
  If you want, can talk to. So as I think, I think because
• 33:49
  of that, it's also it's also a a reason why people have been
• 33:53
  doing funky things with it, because you can plug it in,
• 33:57
  because you can decide how to go and configure it, you know, with
• 34:00
  your extensions. You can say I want to have a the
• 34:02
  big version with everything in it or I cannot.
• 34:05
  There was also very minimal ones like the web browser in the
• 34:08
  Vasan version of the activity is very minimal, right?
• 34:10
  Like it has it has it's like 2 megabytes or so in a download,
• 34:14
  which is which is pretty wild. I think 3 maybe at this point.
• 34:17
  Yeah, it's like so that's yeah, that's really the Lego.
• 34:22
  It's it is, it is, it is really the design lends itself for that
• 34:26
  use case for for being put into other people's stuff and now.
• 34:30
  The funky use cases, have you seen any new ones since that
• 34:34
  talk a year ago? Are there any that come to mind
• 34:36
  right away? Well, I'm right away.
• 34:41
  I don't think so It is. I mean, I see a lot of, see a
• 34:43
  lot, a lot of use cases. I think the maybe was maybe the
• 34:49
  the DeepMind thing, The DeepSeek one was that was, did I mention
• 34:52
  that maybe last? No, I think that happened
• 34:54
  afterwards. That was pretty wild.
• 34:56
  So there's this you're you know more about this than I do, but
• 34:59
  there's this company called DeepMind, DeepSeek, DeepSeek
• 35:03
  DeepSeek. Yeah.
• 35:05
  And. They.
• 35:06
  They wrote a blog post basically saying, hey, yeah, by the way,
• 35:09
  we do all our data wrangling with, with Duckdb and here's our
• 35:13
  orchestration script that we've, that we've written to make this
• 35:17
  work. That was a huge, that was a huge
• 35:20
  moment for not necessary for us because we're like, cool, but
• 35:26
  but people really were like, Oh my God, this is huge for, for,
• 35:31
  for, for Doctor B because it's, yeah, they, they obviously
• 35:35
  picked Doctor B because it, it had all the, and it was
• 35:38
  interesting because they picked, the reason they picked it is
• 35:40
  because it had these primitives that they needed like a sword,
• 35:43
  like a joint, like an aggregate. But, and then they orchestrated
• 35:47
  the, the scale out of that because they have a very
• 35:50
  specific use case. And so they could make that
• 35:52
  happen. I think it's called Deep Pond or
• 35:55
  something, something, I forget the name.
• 35:57
  Of course it's got deep. In there, I mean, sorry, but
• 35:59
  sorry, but sorry, yeah. Sorry to the developers, I don't
• 36:03
  exactly remember the name, but that is that is that is
• 36:07
  something that I was really, I was really surprised by.
• 36:10
  And I always give you happy about this kind of thing,
• 36:13
  actually, Deep Pond. Makes a lot of sense.
• 36:15
  If it's where your duck BB instances are going to hang out,
• 36:20
  then it's going to be in a pond and it's called small pond I'm.
• 36:24
  Sorry, small pond, all right. There we go.
• 36:29
  I look too tough, but but yeah, so a small pond, but I think
• 36:34
  this is really this really interesting to see that.
• 36:36
  And it's it's, yeah, you can't, you can't Like not many database
• 36:41
  systems are built for this. I think the only other one that
• 36:43
  comes to mind is Sequel Lite. And obviously it's been a huge
• 36:47
  success, but it just can't deal with a billion rows, right?
• 36:52
  Like it's not, it's not what it's built for, Yeah.
• 36:55
  Well, I know that my friends at Tecton created a new product
• 37:01
  within Tecton that was built on Duck DB and they were telling me
• 37:05
  when they came on here and talked about it how they
• 37:08
  wouldn't have been able to build that.
• 37:12
  Two years ago. Or three years ago, before the
• 37:16
  rise of like the whole small data movement that Jordan kind
• 37:19
  of made popular. And what they mentioned was how
• 37:25
  vastly different teams are inside of the same company and
• 37:29
  the team needs are. And so for feature engineering
• 37:35
  that Tecton does, you have some teams that are very advanced and
• 37:38
  they need spark and they want all the bells and whistles and
• 37:42
  they want to be able to just like, do some really heavy
• 37:45
  things. But then you have other teams
• 37:47
  that are like, no, I just kind of need the basics.
• 37:51
  And. So that's why they were able to
• 37:53
  build this super lightweight product on top of Duck DB for
• 37:59
  that. Yeah, I, I think that's fairly
• 38:03
  common. We see that a lot.
• 38:05
  It's, it's also I think one of the reasons that we built Duck
• 38:09
  DB in the 1st place because we saw that there, there was
• 38:14
  basically you, you had Hondas and then the next step was Spark
• 38:19
  right there. Was no there.
• 38:22
  Was nothing in between right? I mean, or there was maybe there
• 38:25
  was Postgres and then there was Spark, like if you want to stay
• 38:28
  in that world. And, and and then that is also,
• 38:32
  I think what what also drove us is to be like, OK, let's put
• 38:37
  something in the middle and let's see how far we can push
• 38:39
  this. And and then there was all these
• 38:42
  all these all these. There was these things coming
• 38:45
  out from, from from Amazon Redshift, for example, where
• 38:48
  they showed like, Oh yeah, actually our, our P 99.9 of data
• 38:54
  of, of, of of data set sizes that we use during queries is
• 38:57
  like 300 gigabytes or something like that, which showed that the
• 39:02
  boundary could be pushed actually quite far to to, to to
• 39:07
  all the way over there right where I'm.
• 39:10
  There will always be use cases where you go to 1 Spark, right?
• 39:13
  Like if you run, if you need something where it's like or
• 39:15
  similar systems where you need like 1000 nodes to get through
• 39:19
  like 15 petabytes of data, like by all means.
• 39:23
  But our, yeah, our, our, our suspicion is that this will be
• 39:28
  not so many people. And the vast majority of people
• 39:32
  are quite capable of using duck to be for this stuff.
• 39:36
  Yeah. And.
• 39:37
  Much happier to use something that's way more simple than
• 39:40
  Spark and yeah. Yeah, I know for sure.
• 39:43
  I mean it, it, it is as complex as it as it is mainly because
• 39:47
  it's running a distributed system.
• 39:48
  And if you if, if you want to build a distributed system,
• 39:51
  there's some, some, some ugliness cannot be avoided,
• 39:54
  right? Like you need to do retransmit
• 39:55
  and it's going to be absolute hell.
• 39:59
  And you need to distribute shuffle.
• 40:00
  It's going to be absolutely hell.
• 40:02
  If you don't have to do that, it's, it's, it's a very
• 40:05
  different sort of position and which is why they, the
• 40:09
  comparisons are also not always entirely fair.
• 40:12
  We don't have to do any of that, right?
• 40:14
  They're, they're building their thing.
• 40:16
  You know, we're building our thing.
• 40:17
  It's, it's, it's all good. We are, we're thankfully not in
• 40:20
  this world where there's a limited amount of oxygen for
• 40:23
  data system. I feel like there is always more
• 40:25
  data problems to be solved and we don't have to pretend like we
• 40:29
  need to sort of. I don't know encroach on one in
• 40:33
  order to do better with the other.
• 40:35
  That is so true, yeah. No, there's this is maybe like
• 40:40
  academics have sometimes have this have this thing.
• 40:42
  It's also with duck leg, right? Like we are not, we are not here
• 40:45
  to say this has to be the, the thing that that everyone uses.
• 40:51
  And actually lots of people have asked us like, what if, what if
• 40:55
  the incumbents just take up your idea and then use that like the
• 40:58
  idea of the centralized catalog. And I, I've said this a couple
• 41:02
  of times. I said great, you know, this is
• 41:05
  excellent. This is we are, we're not here
• 41:07
  to, we're not here to enshrine ourselves.
• 41:09
  We're here because we care about technology, we care about how
• 41:12
  data is being processed. If the if the you know, the
• 41:16
  iceberg people want to wants to basically not copy, but get
• 41:23
  inspired by this and and and you know, by all means, right, Like
• 41:28
  it's not they're they're very welcome to.
• 41:30
  I would consider this a huge win, right to say yeah, they
• 41:35
  they agree with us on and this is now changing how this is
• 41:38
  being done. And and again, or if the other
• 41:42
  the other insertive is people just using duck TB and and Duck
• 41:46
  lake. And that's also that's also
• 41:47
  perfectly fine with me. I think that's that's that's not
• 41:51
  it's not a US or them kind of situation.
• 41:53
  It's more like it's a different approach and at the end of the.
• 41:57
  Day Anybody who's ever been on the data team knows it's
• 42:00
  probably both. Yeah, and you got to be the 1:00
• 42:04
  to. Support them, yeah.
• 42:05
  I mean we do support Duck to be does support Iceberg and and
• 42:09
  Delta Lake of course like we are, we are working with a bunch
• 42:12
  of people on on making that work.
• 42:14
  We are very grateful to the, the companies we work with that
• 42:19
  they, you know, they basically help us that we can support also
• 42:22
  iceberg and, and Delta Lake and, and so Europe you can use them
• 42:26
  both. It's not a, it's not a, it's not
• 42:28
  really an issue. It's just that.
• 42:32
  What I think what's Duck Lake, what's special about Duck Lake
• 42:34
  is that it is it's really changing how you can use Duck DB
• 42:38
  right in a, in a, in a way that that in a way that is that is
• 42:42
  like fast and scalable and still simple, still relatively simple,
• 42:47
  right? That's I think that's the that's
• 42:50
  the that's the really interesting thing.
• 42:52
  And in a way it's it's kind of turning the architecture also
• 42:55
  around where we're saying we'll put compute first and the
• 42:59
  computers happening wherever you need it.
• 43:02
  And all the all the management is actually a sort of sub
• 43:04
  function of that or the storage and the metadata is a kind of a
• 43:07
  sub function of compute. Before you always had this like
• 43:09
  top down where now we had control compute storage and then
• 43:14
  maybe user on the very bottom. Now with with that click you and
• 43:17
  and actively you can actually build something looks the other
• 43:19
  way around where you say no, we'll just put compute
• 43:22
  everywhere on every client's machine.
• 43:25
  Why not? And then the other functions are
• 43:28
  actually something that serves this purpose and not some.
• 43:32
  I think it's also something that I want to also change is
• 43:35
  actually the people have been seeing database as this kind of
• 43:37
  mighty thing that it's kind of arcane.
• 43:42
  It needs to be feared. It's like there's like
• 43:45
  specialized priests like the DBA that that know something about
• 43:49
  it and the actor. I'm sorry the.
• 43:51
  Druids of databases, Yes, they need to be weird.
• 43:55
  They need. To be respected they are.
• 43:58
  Yeah, exactly. Job.
• 43:59
  Security. It's absolute.
• 44:01
  I mean, that's, this has it has it has gotten down a little bit,
• 44:04
  I think, but but the, but I think something that, well, I
• 44:09
  think this, this, this is not necessary that this is not a
• 44:12
  technology that is special in any way.
• 44:13
  I mean, it is very complex technology that I admit after,
• 44:17
  after spending the last, I don't know, 15 years on it, but it is,
• 44:22
  it is, it is complex technology, but there is no need of, of, of,
• 44:25
  of, of reverence or anything like that.
• 44:28
  It's, it's just technology and we can actually put in
• 44:30
  everyone's hands and that takes a lot of the of its, of its
• 44:35
  specialness away, right? Like if every if every phone
• 44:39
  runs duck DB, then how is your how is your centralized data
• 44:44
  warehouse any special right? It does the same.
• 44:47
  We cannot do anything different and and when you have that.
• 44:52
  Mentality, it's much more ephemeral and you don't
• 44:56
  necessarily take it as like, Oh my God, it's this sacred thing.
• 45:01
  It's something that can be spun up and destroyed in a way that
• 45:07
  you probably weren't thinking about it before, yeah.
• 45:11
  Absolutely. I I think, I think having having
• 45:16
  doctor you for example, what a juice case.
• 45:18
  I really love when people started putting back to being
• 45:19
  lambdas or serverless compute, right where the database, the
• 45:25
  database that in our mind was this thing that you run and then
• 45:29
  you know, it runs. No, no, no, no.
• 45:31
  It runs like for half a second at best.
• 45:35
  Maybe it runs for 10 milliseconds.
• 45:37
  Who knows, right? This is unheard of, right?
• 45:40
  Nobody's ever built a system for that, like a database system.
• 45:45
  It takes like a Postgres takes, I don't know, or half, half a
• 45:49
  minute to spin up from scratch, right?
• 45:51
  Like it's like, oh, let me do this and let me do that and,
• 45:56
  and, and, and, and this would not be something that's feasible
• 46:00
  to run in a Lambda. But Doctor, we, why not, right?
• 46:03
  Like we actually optimized for this.
• 46:05
  Mark spent some time on this where he was like, no, no, no,
• 46:09
  no, we will, we will go on down to below 10 millisecond startup
• 46:13
  time or something like that. Because it was, we were in this,
• 46:16
  we were like eagerly initializing some stuff.
• 46:18
  And now it's all lazy because we, we, we needed to optimize
• 46:21
  the, the startup time. Yeah, because it's not everybody
• 46:25
  needs all the functions. Hey, you know, like you can load
• 46:28
  them when you need them. And so and so that's, again,
• 46:31
  this is something that we do, right?
• 46:33
  We go, we go in and said, hey, this start up time, you know,
• 46:37
  it's 30 milliseconds. It's not horrible, but maybe it
• 46:40
  can be 5 milliseconds, right? Let's make it fast.
• 46:43
  Let's make it fast. Let's make it, let's make it,
• 46:46
  yeah, let's make it fast. And again, right, like it's
• 46:48
  directly translates to to how many requests per second you can
• 46:53
  run on this thing because yeah, start up time, it's kind of
• 46:57
  wasted. Yeah.
• 46:59
  And. It's your imagination again.
• 47:01
  That's what I I think is so brilliant.
• 47:03
  And I know we had Robert Caulk on here talking about Ask News
• 47:08
  and how he would when people would run queries.
• 47:12
  He was of this mind of like he had these knowledge graphs going
• 47:17
  on, but he didn't have an ontology.
• 47:21
  And he almost would like spin up databases, win queries would
• 47:25
  happen, and then they would dissolve after that person
• 47:29
  logged off. And so that type of thinking
• 47:34
  about how you're going to create your infrastructure is just
• 47:36
  unheard of in my mind on a normal way of doing it, quote UN
• 47:41
  quote normal because you wouldn't really think like, OK,
• 47:45
  let's spin up and dissolve databases at the speed of a user
• 47:49
  session. Exactly.
• 47:51
  Yeah, no. And I think it's also one of
• 47:53
  these things where it takes time.
• 47:56
  It's one of the things that I've, I've realized at some
• 47:58
  point is that these kind of sort of fundamental shifts in
• 48:02
  paradigms or in how people think about things, they do take time.
• 48:06
  And I'm, I'm perfectly happy to give people time.
• 48:09
  We are thankfully not in, you know, in a rush to raise our
• 48:12
  Series B or or whatever. Yeah.
• 48:16
  So it's it's something it's something that if it if it takes
• 48:20
  another five years for that to sort of arrive in the in the
• 48:24
  commerce sort of consciousness, that's that's perfectly fine.
• 48:27
  And I think I think we are we've not even started scratching the
• 48:30
  surface of what what kind of interesting architectures people
• 48:32
  can build with Doc to be. And it's it's it's it's really
• 48:36
  interesting. Like I'm I got the real team is
• 48:39
  kind of keeping, keeping track of all the things that people
• 48:42
  are doing. And then sometimes I get some
• 48:43
  update and it's, it's, it's always, it's always super
• 48:48
  fascinating. And, and also the kind of the
• 48:51
  kind of companies that are adopting Ductive.
• 48:54
  Like I think it was this week that Alibaba came out and say,
• 48:58
  yeah, we now have a service based on Ductive.
• 49:01
  It's here is the announcement. It was mostly in Chinese.
• 49:04
  So I think it will, it will take some time until this arrives
• 49:08
  maybe in San Francisco, but it's, it's, it is, it is there.
• 49:12
  And they are just, they just run with it and, and yeah, and
• 49:16
  that's, that's excellent. And that's, I think something
• 49:18
  we're going to see much more of. And I think what gives me what,
• 49:21
  what I find impressive is the, or what I see is that the, let's
• 49:26
  say the, the market cap of the sort of the average new adopter
• 49:30
  goes up and up and up. Like that's, I think the that's
• 49:32
  the interesting thing, right? Like we when we started, when we
• 49:36
  started the company back in 2121.
• 49:39
  Sorry, I was around in 2001, but it was it was 2021.
• 49:46
  It was. It was.
• 49:47
  We were very grateful for the early adopters, but I, I respect
• 49:51
  really their, their, their, their, you know, their, their
• 49:55
  bravery. Yeah.
• 49:57
  And dedication, yes. And now it's a couple of views
• 50:02
  down the road. What we see is we see enterprise
• 50:06
  basically or gigantic, gigantic corporation to say, oh, I will
• 50:10
  build this product on top, why not, right.
• 50:12
  And that is something that that is quite interesting to see.
• 50:15
  And it is also changing how we, that's also changing our company
• 50:20
  a little bit because you know, we suddenly talking to bigger
• 50:23
  companies, which is an interesting change.
• 50:27
  Like I can tell you, like maybe with a beer at some point, I can
• 50:31
  tell you about purchasing departments, but that's a
• 50:33
  different discussion. It's a It's a fun place, Yeah.
• 50:39
  I bet procurement is one of my favorite words in the English
• 50:43
  language. It is.
• 50:44
  It is a. Fun word.
• 50:44
  Yeah, yeah, they're they're special.
• 50:47
  But but yeah, so, so that's that's really interesting to
• 50:51
  see. And I think with with Duck Lake,
• 50:52
  what we also see is that this is something that works better for
• 50:56
  bigger organizations, right. Duck to be used to work really
• 50:59
  well for the set of. I mean, we've mentioned it in
• 51:01
  the announcement. I think we said we've, I think
• 51:03
  we've kind of beaten the single player campaign, right.
• 51:06
  Let's, let's now, let's now think about multiplayer.
• 51:10
  And and then you ask yourself, when do you need multiplayer?
• 51:12
  Right, Like you need multiplayer when you are when you have 1000
• 51:15
  people and you somehow need to work on, on, on data together at
• 51:19
  a centralized place with all the corporate, you know apply to it.
• 51:27
  It's not that we are. Not that we are like, you know,
• 51:31
  that we are focusing on enterprise feature not at all,
• 51:33
  right, because you can you can usually put these around the
• 51:36
  whole thing, right, But but we see definitely that duck lake as
• 51:42
  a as a concept is, is seeing like adoption in places that are
• 51:46
  generally more people, right, more people at the same time.
• 51:50
  And that's that's also very interesting to see.
• 51:53
  It's, it's like the multiplayer, it's a whole new game, right?
• 51:56
  It's like, yeah, it's the nature of the.
• 52:00
  Beast that you created again, you gave the foundational
• 52:02
  building blocks and now you're it's, if you build it, they will
• 52:06
  come. So you built it and now they're
• 52:08
  coming. And so you're getting to see
• 52:10
  this new wave. Yeah.
• 52:12
  Yeah, I mean, it's not, it's not necessarily true that when they,
• 52:15
  when you build it, they will come, right.
• 52:16
  Like I, I think, I think it's, it's, it's also a bit strange to
• 52:23
  see that people are coming to talk to be in the 1st place,
• 52:26
  right? Like we didn't expect this,
• 52:28
  right? Like we thought, oh, OK, you
• 52:30
  know, it's going to be, I know me and my 400 Twitter followers
• 52:35
  at a time, maybe give it a like, you know, and, and yeah, now,
• 52:42
  now it's now it's much more than that.
• 52:44
  And that's, that's also something that we're like it's.
• 52:47
  We still can't really. Grasp what what has happened
• 52:50
  right, We see a fraction of the activity.
• 52:53
  So we are, we are currently like, this is one of these funny
• 52:56
  things that happened with Duck to be, we're currently close to
• 52:58
  a petabyte of traffic each month for duckdb.org.
• 53:01
  And this is mostly from people downloading Duckdb extensions,
• 53:05
  right? Or computers.
• 53:09
  But, but that's one of these things that, that, that is just,
• 53:13
  it's just vastness of, of, of something that that is beyond my
• 53:17
  comprehension. I don't, I don't know what
• 53:18
  petabyte is. And then probably when you talk
• 53:20
  about to me in 2 years, it will be, Oh yeah, you know, we're now
• 53:23
  like a 2. 100 petabytes. Of traffic or something and and
• 53:27
  then and that that will be the new normal but that that is not
• 53:30
  normal. I think we still remember brain
• 53:32
  is perennially like 5 years behind where where Ductibi is.
• 53:36
  It's it's kind of, it's kind of funny to to see that.
• 53:38
  But yeah, thankfully the thankfully the amount of issues
• 53:42
  hasn't, hasn't been going up at the same pace.
• 53:44
  So we'll keep track. Luckily, yeah, you you have been
• 53:50
  almost had these two big shifts that you're talking about with
• 53:54
  inductive, one being extensions and then the other being Duck
• 53:57
  Lake. You could, if you want, go and
• 54:03
  build anything that you feel is interesting.
• 54:08
  Is there? Something that right now you
• 54:10
  wish Duck to be had that you want to go and make happen.
• 54:14
  Is it as Seminole as like these two that you're talking about or
• 54:18
  are you kind of sitting around waiting for that next shower?
• 54:21
  Thought I. I think it's the second I at the
• 54:24
  moment there is no, no huge thing in my brain where I'm
• 54:28
  where I'm saying this is, this is what we need to do right now.
• 54:32
  There is, but the world is changing, right?
• 54:35
  This is also something that that I've maybe not understood
• 54:39
  properly in the beginning is that the requirements will will
• 54:46
  be changing constantly and and there's, there's new
• 54:49
  developments in how people use data systems, there's new
• 54:52
  formats, there's new compute paradigms, there's no new
• 54:56
  hardware. So that's, I think it's a, it's
• 55:01
  not that I'm I'm concerned of it.
• 55:03
  I was never going to be another thing.
• 55:04
  It's just that I don't, I don't have AI, don't have a huge
• 55:09
  project in my, in my mind right now, I think.
• 55:12
  And that's, and that's OK because there's always, there's
• 55:15
  so many things to do in inductive itself, right.
• 55:18
  So we're, we're doing a big push on, on memory allocation at the
• 55:20
  moment, for example, right? Like that's a huge thing.
• 55:24
  We we also just read. Yeah.
• 55:25
  So, so it's, it's currently being, being, being worked on.
• 55:29
  There's currently is constantly projects going on.
• 55:32
  There might just not be as as huge as as as something like
• 55:38
  something like Jack Lake or the one dot O was also really really
• 55:41
  a big thing for us. Maybe I did a better one dot O
• 55:44
  We are we are releasing the the next version 1.4 in a in a
• 55:48
  month. It's.
• 55:50
  Going to be you know, I'm always amazed when we write the release
• 55:53
  notes and you just scroll through these thousands of pull
• 55:56
  requests to see what, what the team has has has sort of, you
• 56:00
  know, built in the in the in the in the months since the last
• 56:04
  feature version was released. That's that's very it's very
• 56:07
  it's very impressive like we are we are up to about 20-5 people
• 56:10
  now so that that really shows right the the amount of activity
• 56:13
  that goes on on on every you know, every single every single
• 56:17
  day. It's it's quite impressive and
• 56:19
  I'm yeah, I'm really grateful for for everybody that's that's,
• 56:22
  that's working with us. And of course everybody who
• 56:24
  sends pull requests because it is, it is in the end making
• 56:27
  making doctor you better. And yeah, no, but I don't, I
• 56:32
  don't have AI, don't have AI, don't have a sort of a, you
• 56:36
  know, a big, big thing in my mind.
• 56:37
  I'm sorry. It's all right.
• 56:40
  Give us some time. Yeah.
• 56:41
  You're still fresh off of Duck Lake.
• 56:43
  I'm sure something will come up. Give us, Give us.
• 56:45
  Give us a couple of months for Duck, like, OK, we'll we'll,
• 56:48
  we'll, we'll need some time, but you can.
• 56:51
  Still, we'll do A1. .0 at some point, right?
• 56:53
  The Duck Lake 1.0 is probably going to be actually fairly
• 56:56
  soon. We're thinking a baby D towards
• 56:58
  the end of the year. Wow.
• 57:01
  Because you know. It's, it's, we haven't found any
• 57:03
  very catastrophic problems with it yet.
• 57:06
  And, and people are using it. So it's, yeah, yeah, it's, it's
• 57:10
  it's pretty great.