Latest / The Pragmatic Engineer / Distributed databases with Peter Mattis

Transcript

Distributed databases with Peter Mattis · The Pragmatic Engineer

0:00
SPEAKER_01: He built a GIMP image editor while at college,
0:02
SPEAKER_01: designed the original storage system behind Gmail,
0:04
SPEAKER_01: and built many more large, complex, and widely used systems.
0:07
SPEAKER_01: This is Peter Mattis, co-founder and CTO of Cockroach Labs.
0:10
SPEAKER_01: Before we sat down to talk,
0:11
SPEAKER_01: Peter told me,
0:12
SPEAKER_01: for the last 30 years,
0:13
SPEAKER_01: I've always been a prolific coder, but my current output is a bit insane.
0:17
SPEAKER_01: And this isn't Vibe-coded junk,
0:19
SPEAKER_01: but database-worthy,
0:20
SPEAKER_01: high-quality, high-performance code, thanks to working with strong coding models.
0:24
SPEAKER_01: Today we cover why B-trees are so important when building databases,
0:28
SPEAKER_01: and why Peter kept reaching for the data structure over and
0:30
SPEAKER_01: over throughout his career, how he wrote This episode is presented by TurboBuffer.
0:58
SPEAKER_01: A ridiculously scalable fast and cheap hybrid search engine built on top
1:01
SPEAKER_01: of object storage by an engineering team I've really gone to like
1:04
SPEAKER_01: after spending time with them.
1:05
SPEAKER_01: The Turbo Buffer engineering team is doing something really really cool.
1:08
SPEAKER_01: They're completely redesigning their storage architecture from first principles
1:11
SPEAKER_01: to make storage faster, cheaper and more reliable at scale.
1:15
SPEAKER_01: If you follow Turbo Buffer you know that their storage architecture was
1:18
SPEAKER_01: a massive part of their early success.
1:20
SPEAKER_01: Redesigning a winning architecture is a big deal.
1:22
SPEAKER_01: It's one thing for your query plans to pass unit tests.
1:25
SPEAKER_01: It's another to bring each query plan to performance parity,
1:29
SPEAKER_01: or better, while maintaining correctness and reliability in production.
1:33
SPEAKER_01: Here's the cool part.
1:34
SPEAKER_01: TurboPuffer is documenting the whole thing.
1:36
SPEAKER_01: Their new storage architecture,
1:38
SPEAKER_01: which they're calling Teepuff V3,
1:40
SPEAKER_01: demands some hardcore systems engineering and they're building it in public,
1:43
SPEAKER_01: sharing design decisions and benchmark results as they ship.
1:47
SPEAKER_01: They're keeping a work log of their journey, and the first post just dropped today.
1:51
SPEAKER_01: Follow along at turbopuffer.com slash v3.
1:54
SPEAKER_01: That is turbopuffer.com slash v3.
1:57
SPEAKER_01: Peter, welcome to the podcast.
1:59
SPEAKER_01: Oh, I'm
1:59
SPEAKER_00: happy
1:59
SPEAKER_00: to be here.
2:00
SPEAKER_00: This is awesome.
2:01
SPEAKER_01: I wanted to get into, how did you get into tech originally?
2:04
SPEAKER_01: When did you figure out computers are interesting?
2:07
SPEAKER_00: I figured that out in kind of elementary school, high school.
2:10
SPEAKER_00: Gaming was a little bit of a gateway drug for me as a software engineer, as many other people.
2:14
SPEAKER_00: I remember like early on, my mom did programming at IBM at some point.
2:18
SPEAKER_00: I'm never even quite sure what she did,
2:20
SPEAKER_00: but we had computers always around our house, like Apple 2 Plus, Apple 2GS.
2:23
SPEAKER_00: I'm from that era on up and, you know, go to the bookstore.
2:27
SPEAKER_00: I'd find a book on BASIC or magazine,
2:30
SPEAKER_00: type in programs,
2:31
SPEAKER_00: no idea what it was doing, but, you know, just kind of like I was addicted to it.
2:34
SPEAKER_00: You put stuff into these computers and you get interesting stuff out.
2:37
SPEAKER_00: And then I got to college and I was like, I didn't think there was any money in computers.
2:41
SPEAKER_00: I didn't know anything about it.
2:42
SPEAKER_00: I started as a mechanical engineer, following the footsteps of my dad.
2:45
SPEAKER_01: Oh, you started mechanical engineering as your socialization?
2:48
SPEAKER_01: Yeah, as my major, yeah.
2:49
SPEAKER_00: And I got in there and I was like doing, like I'd done some computer stuff before.
2:53
SPEAKER_00: It was a real foolish move to do this, but...
2:55
SPEAKER_00: First semester of doing these homework assignments in mechanical engineering,
2:58
SPEAKER_00: they were god-awful, like six pages for a single problem.
3:00
SPEAKER_00: And I happened to take a CS course at the same time.
3:04
SPEAKER_00: And it was so easy.
3:06
SPEAKER_00: And then everybody almost was failing it.
3:07
SPEAKER_00: I'm like, I'm in the wrong field.
3:09
SPEAKER_00: Let me switch.
3:10
SPEAKER_00: And then you switched.
3:11
SPEAKER_00: And then I switched, yeah.
3:12
SPEAKER_01: What was the first software that you built,
3:15
SPEAKER_01: either at college,
3:15
SPEAKER_01: it must have been at college,
3:17
SPEAKER_01: that you were like,
3:18
SPEAKER_01: all right, this is a piece of software that I'm kind of proud of.
3:20
SPEAKER_01: That's a complete piece of software.
3:21
SPEAKER_01: Yeah.
3:21
SPEAKER_00: Well, I mean,
3:21
SPEAKER_00: the big thing that I did along with my roommate in college,
3:25
SPEAKER_00: we had this course, what was it, a compiler's course?
3:28
SPEAKER_00: I'm not quite sure anymore.
3:28
SPEAKER_00: This is like 30 years ago.
3:30
SPEAKER_00: And we were kind of bored with it.
3:31
SPEAKER_00: So we wanted to do something like kind of fun on the side.
3:34
SPEAKER_00: And I'd done journalism in high school in my junior and senior year.
3:38
SPEAKER_00: I knew stuff about kind of computer graphics and wanted to do something like Adobe Photoshop.
3:43
SPEAKER_00: Started just kicking the tires and we built up this program that a lot
3:45
SPEAKER_00: of people know of called the GIMP.
3:47
SPEAKER_00: And along with the GIMP, I did a lot of the graphics library, GTK.
3:51
SPEAKER_00: This has since evolved just massively since then.
3:53
SPEAKER_00: It's kind of interesting because after college, I kind of stepped away from it.
3:57
SPEAKER_00: Didn't really stay involved much past my first year out of college,
4:00
SPEAKER_00: but definitely people still know me and it led to other some interesting events
4:04
SPEAKER_00: in my career.
4:05
SPEAKER_01: And it was like starting GIMP,
4:07
SPEAKER_01: it was literally just you saying, all right, I want to do something like Photoshop.
4:10
SPEAKER_01: How hard could it be?
4:11
SPEAKER_01: And then you just, this wasn't through what you learned in college, right?
4:14
SPEAKER_01: This was like you figuring out how to build,
4:18
SPEAKER_01: you know,
4:18
SPEAKER_01: like a graphical,
4:20
SPEAKER_01: I guess, engine, rendering, drawing, data structures, all of that stuff, right?
4:24
SPEAKER_00: All that stuff, figured it all out.
4:25
SPEAKER_00: I remember trying to look at some papers back then.
4:28
SPEAKER_00: My roommate was looking at papers.
4:29
SPEAKER_00: We're just fingering it all out.
4:31
SPEAKER_00: And it's one of these things,
4:32
SPEAKER_00: you almost kind of need to be a little bit naive to start anything like that.
4:35
SPEAKER_00: Because if you know how hard it's going to be, you would never start it.
4:38
SPEAKER_00: So I think there's this, like,
4:39
SPEAKER_00: you have to have this level of, like,
4:40
SPEAKER_00: if you're really truly wise about the effort, you, like, you never get into it.
4:43
SPEAKER_00: But then as you get going, it just keeps on snowballing.
4:45
SPEAKER_00: It gets farther.
4:46
SPEAKER_00: And then at some point you're like, wow, this is really awesome.
4:50
SPEAKER_00: But there's an interesting little tidbit associated with our work on the GIMP,
4:53
SPEAKER_00: which is kind of a little bit known.
4:55
SPEAKER_00: I think we've talked about this before,
4:56
SPEAKER_00: but it's just kind of fascinating where we were getting to the point
5:00
SPEAKER_00: where we're like,
5:00
SPEAKER_00: we should release this And back then,
5:02
SPEAKER_00: there was Usenet newsgroups where people would post stuff,
5:05
SPEAKER_00: and there was one on graphics.
5:06
SPEAKER_00: And remember, like,
5:07
SPEAKER_00: just a couple weeks before we were going to release our first version of the GIMP,
5:11
SPEAKER_00: someone else came on, they were like, I've been working on this graphics program.
5:14
SPEAKER_00: And it did everything the GIMP did, every last thing, and then some.
5:18
SPEAKER_00: And we're just like, oh, that sucks.
5:20
SPEAKER_00: I guess we'll just keep on working on it.
5:21
SPEAKER_00: It's been fun.
5:22
SPEAKER_00: And then we release the GIMP.
5:24
SPEAKER_00: Never heard from this other guy again.
5:26
SPEAKER_00: And I just took a little lesson with that.
5:28
SPEAKER_00: There's always going to be someone else working on your idea.
5:30
SPEAKER_00: You can't get dissuaded if they pre-announce it.
5:33
SPEAKER_00: Nothing ever comes of it.
5:34
SPEAKER_00: And a lot of the marketing behind some of that stuff that you might hear is like,
5:37
SPEAKER_00: I mean, I don't know if we like, we stole his thunder.
5:40
SPEAKER_00: I'm not even sure what happened.
5:41
SPEAKER_00: Never traced down what happened, but there's a real lesson there.
5:45
SPEAKER_01: So there's a possible future where you read this announcement
5:49
SPEAKER_01: where someone's saying,
5:50
SPEAKER_01: I'm going to build all of this thing and you go like, ah,
5:52
SPEAKER_01: you know,
5:52
SPEAKER_01: someone did it and you kind of go back and just do something else and
5:55
SPEAKER_01: the game never happens.
5:56
SPEAKER_01: That's right.
5:57
SPEAKER_01: Wow.
5:57
SPEAKER_00: Yeah.
5:58
SPEAKER_01: I guess especially today with startups.
6:00
SPEAKER_01: Sounds like just do your thing, put it out there, at the very least, right?
6:04
SPEAKER_00: The advice I give people is like,
6:06
SPEAKER_00: you might have a unique idea,
6:08
SPEAKER_00: but most likely there's like a dozen people out in the world who've had
6:10
SPEAKER_00: the same idea.
6:11
SPEAKER_00: And there might be a couple of them working on it, but a lot of people don't even work on it.
6:14
SPEAKER_00: They're just like whatever they can't get going on their idea.
6:16
SPEAKER_00: So...
6:17
SPEAKER_00: And what were some fun things that the game pled to?
6:20
SPEAKER_00: Well, you know...
6:36
SPEAKER_00: I got kind of tired with graphics, and that's why I kind of moved away from it after college.
6:39
SPEAKER_00: I got into storage systems, and I bounced around.
6:43
SPEAKER_00: You know,
6:43
SPEAKER_00: I worked at one of the early search engines,
6:45
SPEAKER_00: Inked Me, and then I got over to another startup.
6:48
SPEAKER_00: And I met through that time period.
6:50
SPEAKER_00: In that first startup I was doing, I met Larry and Sergey from Google.
6:54
SPEAKER_00: Because it turns out that Larry, I think it was maybe Sergey.
6:57
SPEAKER_00: I'm not quite sure.
6:58
SPEAKER_00: The very first version of the Google logo was done in the GIMP.
7:01
SPEAKER_00: So they knew of us.
7:02
SPEAKER_00: Somehow one of them, you know, looked me up.
7:04
SPEAKER_00: Asked me to come interview at Google.
7:06
SPEAKER_00: I did back in 2001.
7:07
SPEAKER_00: This is when
7:09
SPEAKER_01: Google was three years old.
7:10
SPEAKER_00: Yeah, three years old.
7:11
SPEAKER_00: And I said no.
7:13
SPEAKER_00: You did not.
7:14
SPEAKER_00: I said no.
7:14
SPEAKER_00: Yeah.
7:15
SPEAKER_00: No, here's the calculation in my mind.
7:17
SPEAKER_00: I was like, Google's down in Mountain View.
7:18
SPEAKER_00: I was living up in San Francisco and I didn't want to do the commute.
7:21
SPEAKER_00: So I went and worked at another startup for a year.
7:24
SPEAKER_00: And that went after a year.
7:25
SPEAKER_00: I like I saw it wasn't going anywhere.
7:27
SPEAKER_00: And they called me back up and they said, hey, do you want to interview again?
7:29
SPEAKER_00: I was like, sure.
7:30
SPEAKER_00: Like, we're not going to be able to offer you the same stock options you got before.
7:33
SPEAKER_00: I'm like, OK, I can't remember what it was.
7:35
SPEAKER_00: And I still can't remember what they offered me the first time.
7:37
SPEAKER_00: But I probably would have made a lot more money if I had taken that first.
7:40
SPEAKER_00: But I did all right.
7:41
SPEAKER_00: I'm not like complaining, but it's one of these things like, yeah.
7:43
SPEAKER_00: Now, I think back about that.
7:44
SPEAKER_00: I'm like, oh, a lot of good stuff came out of it.
7:46
SPEAKER_00: Some point along the line, Red Hat was going public.
7:48
SPEAKER_00: They actually offered friends and family stock during their IPO to a lot of people.
7:52
SPEAKER_00: We got offered friends.
7:53
SPEAKER_00: I made a little bit of money off that.
7:54
SPEAKER_00: Not a lot, but it was like just after college.
7:56
SPEAKER_00: It was actually quite significant at the time, you know?
7:59
SPEAKER_00: So, I mean,
7:59
SPEAKER_00: there's a number of good things that kind of came out of that that
8:01
SPEAKER_01: And I also used
8:02
SPEAKER_01: the GIMP as a,
8:03
SPEAKER_01: you know, I was looking for like, oh, I can't afford Photoshop and here's the GIMP.
8:07
SPEAKER_01: And it did so many things.
8:08
SPEAKER_01: So like,
8:09
SPEAKER_01: I'm sure there's so many people like you had a positive impact of being able
8:13
SPEAKER_01: to use this thing.
8:14
SPEAKER_01: And the nice thing that I,
8:16
SPEAKER_01: what I really liked about it is it was free,
8:18
SPEAKER_01: but I wasn't like stealing any software from anyone.
8:21
SPEAKER_01: You see what I mean?
8:22
SPEAKER_01: This was at a time where free software was not as common as today.
8:26
SPEAKER_01: Free and open source was not as mainstream as it is today.
8:29
SPEAKER_01: Yeah.
8:30
SPEAKER_00: And then the other cool thing I've heard from a lot of people,
8:32
SPEAKER_00: because people still, like I mentioned this, like, oh, I've used it.
8:35
SPEAKER_00: And then some people,
8:36
SPEAKER_00: you know,
8:36
SPEAKER_00: software engineers are just like, I learned how to program by looking at your code.
8:39
SPEAKER_00: And I'm just like, wow, that was a code I wrote 30 years ago.
8:41
SPEAKER_00: And I wasn't nearly as good as a software engineer as I am now.
8:44
SPEAKER_01: And then you got into Google second time around.
8:46
SPEAKER_01: You said yes.
8:47
SPEAKER_01: What did you start to work on?
8:49
SPEAKER_00: Yeah, yeah.
8:50
SPEAKER_00: Well, I got in there and they said like,
8:51
SPEAKER_00: hey, you know, this was early days, 2002, it was April 1st, auspicious date.
8:57
SPEAKER_00: Your start date?
8:58
SPEAKER_00: Yeah, April 1st, 2002.
9:00
SPEAKER_00: And I got in there and they were like,
9:02
SPEAKER_00: well, you know, we're going to actually build email, Google email.
9:06
SPEAKER_00: And it wasn't called Gmail at the time.
9:08
SPEAKER_00: There's a codename internally.
9:09
SPEAKER_00: It's called Caribou.
9:11
SPEAKER_00: Yeah.
9:12
SPEAKER_00: And got in there, started working on it.
9:14
SPEAKER_00: And I was kind of tasked with working on the backend threading and message storage
9:18
SPEAKER_00: and indexing system and worked on that.
9:21
SPEAKER_00: You know, pretty hardcore for the first year and a half.
9:24
SPEAKER_00: Actually,
9:24
SPEAKER_00: I was on to,
9:25
SPEAKER_00: maybe it was three years on it through the launch,
9:27
SPEAKER_00: which actually happened to be April 1st, 2004.
9:29
SPEAKER_00: Do you remember this?
9:30
SPEAKER_00: There was a, it was like considering April's full joke.
9:34
SPEAKER_01: Yeah, I remember,
9:35
SPEAKER_01: so correct me if I'm wrong,
9:36
SPEAKER_01: but the launch said Gmail with one gigabyte of storage or something
9:41
SPEAKER_01: or infinite storage and like star one gigabyte,
9:43
SPEAKER_01: I'm not sure what it was,
9:44
SPEAKER_01: but back then most email providers would give you about 10 megabyte We were talking
9:49
SPEAKER_01: about this internally,
9:50
SPEAKER_01: it was like kind of a shock and awe campaign for the industry.
9:52
SPEAKER_01: I think it was actually four megabytes for like Hotmail or Yahoo Mail.
10:13
SPEAKER_00: And then you have this.
10:13
SPEAKER_00: And not only that we had so much more storage, but it was indexed really fast.
10:18
SPEAKER_00: So you could do a search and come back almost instantaneously.
10:22
SPEAKER_01: But can you tell me internally,
10:24
SPEAKER_01: like when the project started, okay,
10:25
SPEAKER_01: we're going to do email for Google,
10:27
SPEAKER_01: how did you and the team arrive to the point of like,
10:30
SPEAKER_01: okay, we will offer all this storage and we'll do it fast?
10:33
SPEAKER_01: Because this was at a time, if you can take us back, but what I remember is...
10:37
SPEAKER_01: Hard drives were still expensive.
10:39
SPEAKER_01: They were relatively slow.
10:40
SPEAKER_01: We're talking HDD.
10:41
SPEAKER_01: We're not talking necessarily about SSD, if I remember.
10:44
SPEAKER_01: Can you take us back of what it was like,
10:47
SPEAKER_01: what the constraints were like,
10:48
SPEAKER_01: and then how you innovated to actually do something that's never been done before?
10:53
SPEAKER_00: Yeah, yeah.
10:54
SPEAKER_00: So at the time, Google internally had this large distributed file system called GFS.
11:00
SPEAKER_00: And we were basically looking at numbers and being like,
11:03
SPEAKER_00: yeah, we think we can build on top of this.
11:04
SPEAKER_00: They have a lot of knowledge about how to do search and retrieval.
11:08
SPEAKER_00: It ended up being that Like some of the existing systems they had for search
11:11
SPEAKER_00: and retrieval,
11:11
SPEAKER_00: they started building the prototype on there and then that was completely rewritten.
11:14
SPEAKER_00: That was actually what I got involved in because I got in there
11:16
SPEAKER_00: and there was already a prototype and then it was completely rewritten.
11:19
SPEAKER_00: So I was on that, you know, threading side.
11:21
SPEAKER_00: We decided to have message threading right from the get go, which is also kind of innovative.
11:25
SPEAKER_00: That wasn't common in email systems.
11:27
SPEAKER_01: And threading to have the message thread,
11:29
SPEAKER_01: which I assume needed data structures on the back and it needed like,
11:32
SPEAKER_01: you know, storage and figuring out the How do B-trees relate to email threads?
11:48
SPEAKER_00: It's just like you have to have some storage there where you have a thread ID,
11:51
SPEAKER_00: a message comes in, you have to look up.
11:53
SPEAKER_00: It was using the search index to match actually the subject or the message ID into the thread.
11:57
SPEAKER_00: And there are some other things taking place in the B-tree as well
12:00
SPEAKER_00: because we were keeping track of the unread counts of threads and whatnot.
12:04
SPEAKER_00: So, you know, I can't even remember all the details now.
12:06
SPEAKER_00: This is ancient history.
12:07
SPEAKER_00: This is 2004.
12:07
SPEAKER_00: What are we in?
12:08
SPEAKER_00: 2026?
12:09
SPEAKER_00: 22 years ago.
12:09
SPEAKER_00: So it's like left my memory.
12:11
SPEAKER_00: But there was definitely B-trees.
12:12
SPEAKER_00: There was also this, you know, inverted index taking place there.
12:15
SPEAKER_00: All this code has since been completely ruined.
12:17
SPEAKER_01: What were the economics?
12:19
SPEAKER_01: In the team,
12:19
SPEAKER_01: you must have done the economics of being able to offer free email, which,
12:23
SPEAKER_01: of course,
12:24
SPEAKER_01: I'm sure you did some maths of how it could be subsidized,
12:27
SPEAKER_01: but to not make a terrible, terrible loss.
12:30
SPEAKER_00: Yeah, yeah.
12:31
SPEAKER_00: Yeah.
12:31
SPEAKER_00: No, I mean,
12:31
SPEAKER_00: we were kind of, uh,
12:32
SPEAKER_00: kicking around ideas and, um,
12:34
SPEAKER_00: you know,
12:35
SPEAKER_00: one of the ideas that had come up at some point was,
12:37
SPEAKER_00: Hey, maybe we can put ads on this.
12:39
SPEAKER_00: And it was one of these crazy things.
12:40
SPEAKER_00: So I didn't, wasn't involved in this.
12:42
SPEAKER_00: This is Paul Buchheit went on to do some cool stuff at, uh,
12:46
SPEAKER_00: Friend feed and Facebook and ended up,
12:48
SPEAKER_00: I think, a partner at Y Combinator at some point.
12:50
SPEAKER_00: But he just like one night,
12:51
SPEAKER_00: he's like, no,
12:52
SPEAKER_00: I think I can just take out some of our existing ad functionality and incorporate
12:56
SPEAKER_00: in there.
12:57
SPEAKER_00: And it just went like gangbusters.
12:58
SPEAKER_00: And there's a huge business that,
13:00
SPEAKER_00: you know, kind of grew up from that, which is kind of incredible.
13:02
SPEAKER_00: So, I mean,
13:03
SPEAKER_00: I think there's a lot of things that it's just like,
13:04
SPEAKER_00: You kind of have a sense of what can be done, like,
13:06
SPEAKER_00: oh, we think this is going to cost, you know, per user a couple of dollars per year.
13:10
SPEAKER_00: How do we monetize that?
13:11
SPEAKER_00: And, you know, we didn't want to charge for it.
13:13
SPEAKER_00: Eventually they did charge for it because you have the whole Google Workspace stuff.
13:16
SPEAKER_00: But at the time it was more like, ah, can we do this for free?
13:19
SPEAKER_00: Oh, it looks like it can be economical.
13:21
SPEAKER_00: And it took a little bit of a leap of faith.
13:24
SPEAKER_01: Do you remember the launch?
13:26
SPEAKER_01: The reason I'm asking is because I remember that it was an invite-based system.
13:30
SPEAKER_01: Not everyone could get in,
13:32
SPEAKER_01: and I assume that must have been to control the expected demand because, again,
13:36
SPEAKER_01: you were offering something for free that was paid before,
13:38
SPEAKER_01: and it was pretty obvious that there would be massive demand.
13:42
SPEAKER_01: How did you...
13:44
SPEAKER_01: Think about it, kind of monitor demand, decide how many people to onboard.
13:48
SPEAKER_00: Honestly, this is a team effort, and I wasn't involved in that.
13:51
SPEAKER_00: I mean, I do remember being worried about, like,
13:53
SPEAKER_00: the load that would happen,
13:54
SPEAKER_00: and then someone came up with the idea of,
13:56
SPEAKER_00: like, hey, we should do an invite-based system.
13:58
SPEAKER_00: And this also had, like, it played dual roles, so it kind of constrained the growth.
14:02
SPEAKER_00: But it also kind of drummed up this excitement, oh, you can get me a Gmail invite.
14:06
SPEAKER_00: I remember people asking me at the time,
14:07
SPEAKER_00: I was like, yeah, I can get you as many Gmail
14:09
SPEAKER_01: invites as you want.
14:09
SPEAKER_01: And then after you built the system, where did you move on to?
14:14
SPEAKER_01: What was your next project?
14:15
SPEAKER_01: Was it the build system?
14:17
SPEAKER_00: Well, there was the build system.
14:18
SPEAKER_00: That was kind of all muddled in my mind because I was kind of doing that part-time
14:21
SPEAKER_00: when I was doing Gmail as well.
14:23
SPEAKER_00: At some point, Google actually had this large monorepo.
14:26
SPEAKER_00: I think they actually still have the monorepo.
14:27
SPEAKER_01: They still have the monorepo.
14:28
SPEAKER_00: And they started out with the Google one.
14:30
SPEAKER_00: Repo, that was before my time.
14:31
SPEAKER_00: Then they moved on to Google 2.
14:33
SPEAKER_00: That was what was there when I got to Google.
14:35
SPEAKER_00: And at some point, we saw the strains of Google 2.
14:37
SPEAKER_00: And Google 2 was just one single monolithic makefile.
14:40
SPEAKER_00: I think it actually had some submakefiles, but it was this really large unwieldy makefile.
14:45
SPEAKER_00: Someone came to me and was like, I think we could do something.
14:47
SPEAKER_00: I'd had some interest in the build systems.
14:49
SPEAKER_00: And I kind of put the foundation in for Google 3.
14:52
SPEAKER_00: There was a bunch of people involved.
14:54
SPEAKER_00: I was kind of doing the initial work on Google 3.
14:56
SPEAKER_00: And Google 3 The initial insight was like, hey, makefiles kind of sucked to write.
15:01
SPEAKER_00: We introduced this thing called build files, and I decided to do it.
15:05
SPEAKER_00: It's just this stripped down kind of Python language, but it was still Python at that point.
15:09
SPEAKER_00: And what gconfig spit out at the end was a monolithic link file,
15:12
SPEAKER_00: but one you didn't have to write.
15:13
SPEAKER_00: Over time, this evolved.
15:15
SPEAKER_00: It became Blaze internally.
15:18
SPEAKER_00: There's some other systems that are associated with now.
15:20
SPEAKER_00: I don't even know how complicated it's gone.
15:21
SPEAKER_00: I haven't seen it for a long, long time.
15:23
SPEAKER_00: Then that became Bazel externally.
15:25
SPEAKER_00: It became Buck.
15:25
SPEAKER_00: There's some other,
15:26
SPEAKER_00: you know, people went to Facebook and they were like, that was awesome at Google.
15:29
SPEAKER_01: What was the reason that make files or make didn't really work?
15:33
SPEAKER_01: Was it, are we talking about build performance?
15:36
SPEAKER_01: Are we talking about maintainability or readability?
15:40
SPEAKER_01: Yeah.
15:40
SPEAKER_00: So like,
15:41
SPEAKER_00: I think make itself is like,
15:43
SPEAKER_00: it's okay in declaring dependencies,
15:45
SPEAKER_00: but it's a little bit like kind of assembly language.
15:48
SPEAKER_00: And you didn't want to write, you know, all your dependencies in assembly language.
15:51
SPEAKER_00: And if you didn't know what you were doing,
15:52
SPEAKER_00: it was easy to make a mistake and miss dependencies and whatnot.
15:55
SPEAKER_00: And so the kind of my thought behind the build files is like, hey,
15:58
SPEAKER_00: you need to express these dependencies,
16:00
SPEAKER_00: but at a higher level and just like kind of cleaner semantics associated with it.
16:03
SPEAKER_00: And from that, you can compile down into the assembly language.
16:06
SPEAKER_00: And then eventually folks were like, oh, you don't have to compile down to that.
16:09
SPEAKER_00: We can just kind of implement, you know, the dependency kind of update engine directly.
16:13
SPEAKER_01: And then that's where the performance improvements were able to come in.
16:16
SPEAKER_01: Because at Google scale,
16:18
SPEAKER_01: when you have a large repo,
16:19
SPEAKER_01: in general,
16:20
SPEAKER_01: my understanding is that the reason Bazel and Buck are so popular
16:24
SPEAKER_01: for large codebases is it can help you improve your build performance.
16:28
SPEAKER_01: It gives you a lot more levers to play around with,
16:31
SPEAKER_01: from caching,
16:31
SPEAKER_01: from being smart about cache generation, to obviously just raw performance.
16:34
SPEAKER_00: Yeah, that's exactly it.
16:36
SPEAKER_00: Yeah.
16:36
SPEAKER_01: So you kind of just dabbled and like, okay, I'll make this build file.
16:40
SPEAKER_01: Did that, people took that over.
16:42
SPEAKER_01: But what was your next main focus?
16:44
SPEAKER_00: Well, so I mentioned GFS earlier, Google File System.
16:47
SPEAKER_00: And at some point we realized that there were some limitations in GFS,
16:51
SPEAKER_00: scalability bottlenecks, because I've been working on the storage system for Gmail.
16:55
SPEAKER_00: I actually dabbled in another storage system,
16:57
SPEAKER_00: you know, kind of a research thing that never went anywhere.
16:59
SPEAKER_00: But because we're working on that,
17:01
SPEAKER_00: I got invited to participate in like the founding team of Colossus,
17:04
SPEAKER_00: which was the successor to GFS.
17:06
SPEAKER_00: And as far as I know, Colossus still exists.
17:08
SPEAKER_00: It's gone through multiple iterations at Google,
17:10
SPEAKER_00: but it's like the second generation, you know, distributed file system.
17:13
SPEAKER_01: So what is Colossus?
17:15
SPEAKER_01: Yeah.
17:16
SPEAKER_00: So when I say distributed file system,
17:18
SPEAKER_00: externally, you might think of something like S3, kind of blob storage.
17:21
SPEAKER_00: It had a flat namespace,
17:23
SPEAKER_00: kind of like S3,
17:23
SPEAKER_00: you give names and there's like a minor hierarchy there,
17:26
SPEAKER_00: but it's like very limited,
17:27
SPEAKER_00: but it's not like a So you don't have the full directory hierarchy.
17:31
SPEAKER_00: You didn't even have the full kind of permission system.
17:33
SPEAKER_00: A lot of that stuff got it added later, but the files are not stored in your local machine.
17:36
SPEAKER_00: There is a fleet, you know, kind of a service out there that has all the files.
17:40
SPEAKER_00: They're writing it down to their hard drives and their SSDs.
17:43
SPEAKER_00: And your client can access that.
17:45
SPEAKER_00: And it's all replicated.
17:46
SPEAKER_00: So if there's any crashes and whatnot, you're not losing your data.
17:49
SPEAKER_00: I don't know when, at some point, S3 got erasure coding.
17:52
SPEAKER_00: We did erasure coding in Colossus.
17:54
SPEAKER_00: That was kind of a big breakthrough.
17:55
SPEAKER_01: Which coding?
17:57
SPEAKER_00: We used Reece Solomon.
17:58
SPEAKER_00: So for the audience who's not familiar with erasure coding,
18:01
SPEAKER_00: You might think of like, I want to have replicas.
18:04
SPEAKER_00: And there's this thing in hard disks called RAID,
18:07
SPEAKER_00: where it's like, I don't actually have to have full replicas.
18:09
SPEAKER_00: I can actually,
18:10
SPEAKER_00: you know,
18:10
SPEAKER_00: if you take A and B,
18:12
SPEAKER_00: you can XORM together, and then you can have this kind of third kind of version.
18:16
SPEAKER_00: And there's more and more complicated versions of that.
18:17
SPEAKER_00: Reed Solomon is kind of,
18:19
SPEAKER_00: you know,
18:19
SPEAKER_00: I think there's actually better codes now,
18:21
SPEAKER_00: but it's like one of the known ways to do this.
18:22
SPEAKER_00: One of the known ways, yeah.
18:23
SPEAKER_01: Yeah.
18:23
SPEAKER_00: And we had to,
18:24
SPEAKER_00: you know,
18:24
SPEAKER_00: kind of pioneer internally,
18:26
SPEAKER_00: like, oh, how are we going to actually make this work in distributed file systems?
18:29
SPEAKER_01: Where you focus on latency, on being able to store data more efficiently.
18:33
SPEAKER_00: It's storing it more efficiently.
18:35
SPEAKER_00: So in GFS,
18:36
SPEAKER_00: and there's varying costs where it's like, well,
18:39
SPEAKER_00: you're not storing two copies or one copy of your data or two copies,
18:42
SPEAKER_00: you're storing three.
18:43
SPEAKER_00: Triplication.
18:43
SPEAKER_00: Three times as much storage you're having to use.
18:45
SPEAKER_00: And with Reed Solomon, you can get that down quite a bit lower.
18:48
SPEAKER_00: I can't remember offhand exactly what our...
18:50
SPEAKER_00: What we used for read Solomon,
18:52
SPEAKER_00: I think it was like essentially 2x, but you get that with also the redundancy too.
18:56
SPEAKER_00: So it's like it's smaller and the redundancy is higher.
18:58
SPEAKER_01: So because I guess the naive thing,
19:01
SPEAKER_01: if you're saying,
19:02
SPEAKER_01: all right,
19:02
SPEAKER_01: I want my data to be replicated at three places,
19:04
SPEAKER_01: you take three notes,
19:05
SPEAKER_01: three And then you're saying that the algorithm here is you could take not 3x
19:08
SPEAKER_01: the data, but 2x the data, split it smartly,
19:20
SPEAKER_01: That's exactly right.
19:23
SPEAKER_01: And like kind of the mental model,
19:26
SPEAKER_00: you know,
19:28
SPEAKER_00: if you just want to understand at a high level,
19:33
SPEAKER_00: which is essentially you might want to say like,
19:35
SPEAKER_00: I want to have eight replicas of this data.
19:37
SPEAKER_00: I think I can't remember offhand.
19:39
SPEAKER_00: I think S3 might use nine.
19:40
SPEAKER_00: They've actually talked about this publicly.
19:42
SPEAKER_00: You have kind of nine chunks of data,
19:45
SPEAKER_00: but any five of those chunks can be used to reconstruct it.
19:48
SPEAKER_00: And what this means is you can lose any four copies and you can still reconstruct your data.
19:51
SPEAKER_00: And oftentimes it's more like,
19:53
SPEAKER_00: you know,
19:53
SPEAKER_00: the first five chunks are exact replicas and the other four are kind of parody ones.
19:58
SPEAKER_00: I've kind of forgot some of the details that escaped my mind, but
20:00
SPEAKER_01: it's along this line.
20:01
SPEAKER_01: But when you come up with an algorithm, you can then prove that this algorithm will work, right?
20:05
SPEAKER_01: Like this is a little bit like I know in...
20:08
SPEAKER_01: Software engineering, like maths and algorithms is a bit out of fashion.
20:11
SPEAKER_01: But in this case,
20:12
SPEAKER_01: this is really important because once you once you can prove it
20:14
SPEAKER_01: that this algorithm works, it will work.
20:16
SPEAKER_00: It will work.
20:17
SPEAKER_00: And, you know, it's like the math behind here is like Gawah fields over like GF2 somewhere.
20:23
SPEAKER_00: I never actually understood the full math behind it.
20:27
SPEAKER_00: I always regretted not doing more math in college, but you didn't have to.
20:30
SPEAKER_00: Like Reed and Solomon proved how this works.
20:32
SPEAKER_00: I think it was like back in the 1970s, something associated with communication networks.
20:36
SPEAKER_00: So you just take that and kind of use that expertise,
20:41
SPEAKER_00: but leverage that and have to do all the engineering behind it to make it work in
20:43
SPEAKER_00: a storage system.
20:45
SPEAKER_01: And then when building a distributed storage system like Colossus,
20:48
SPEAKER_01: what were other things beyond,
20:50
SPEAKER_01: okay, you want to store data in a resilient, but efficient way.
20:53
SPEAKER_01: What were other things that problems that you needed to solve?
20:56
SPEAKER_01: I'm thinking things potentially like sharding and resharding or metadata.
21:02
SPEAKER_01: And then Google was like, no, we need
21:04
SPEAKER_00: to have this scale up to 10,000 machines.
21:23
SPEAKER_00: And there's some bottlenecks.
21:24
SPEAKER_00: There was this GFS master as a single node, and it was a bit of a bottleneck.
21:27
SPEAKER_00: So we're like,
21:28
SPEAKER_00: ah, we need to have a distributed master to store the metadata for all the objects.
21:32
SPEAKER_00: And Google at the time happened to have this system called Bigtable.
21:35
SPEAKER_00: And so Colossus stored its metadata inside Bigtable.
21:39
SPEAKER_00: And one of the things that I'm kind of proud and kind of like also,
21:42
SPEAKER_00: you know,
21:43
SPEAKER_00: a little bit embarrassed by,
21:44
SPEAKER_00: one of the design choices I went down,
21:46
SPEAKER_00: but actually worked out, is we wanted to use Bigtable for the metadata for Colossus.
21:51
SPEAKER_01: And the metadata,
21:53
SPEAKER_01: just for those of us not as into distributed systems,
21:55
SPEAKER_01: what is the metadata in a distributed file system?
21:57
SPEAKER_00: Yeah, it's like the names of the files,
21:59
SPEAKER_00: and for each of the files, the files are broken into chunks.
22:02
SPEAKER_00: What were they?
22:03
SPEAKER_00: 64 megabyte chunks.
22:04
SPEAKER_00: And then you have to have the list of chunks for each file.
22:06
SPEAKER_01: Yeah.
22:06
SPEAKER_00: And then you have to periodically,
22:08
SPEAKER_00: you know, the master has to be scanning over this and doing repair work.
22:12
SPEAKER_00: But there's more metadata than that, but that's it in a nutshell.
22:15
SPEAKER_00: Yeah.
22:16
SPEAKER_00: We wanted Colossus to have this,
22:18
SPEAKER_00: you know, kind of scalable, you know, service, Bigtable, to store its metadata.
22:22
SPEAKER_00: The biggest user of GFS is Bigtable.
22:24
SPEAKER_00: So we want Bigtable to work on top of Colossus.
22:26
SPEAKER_00: So wrap your head around that.
22:27
SPEAKER_01: So you have the circ look, the pen then?
22:29
SPEAKER_00: Yeah, well, there's a bootstrapping thing, right?
22:31
SPEAKER_01: Bootstrapping, yeah.
22:33
SPEAKER_00: Oh, yes.
22:33
SPEAKER_00: Yeah.
22:34
SPEAKER_01: Which one starts up?
22:36
SPEAKER_01: You need to mock something somewhere, right?
22:38
SPEAKER_00: Yeah, no.
22:38
SPEAKER_00: I mean, the way it actually worked at the time,
22:40
SPEAKER_00: and they've since replaced this,
22:41
SPEAKER_00: you know,
22:41
SPEAKER_00: because this is like the way to get started and leverage what you have,
22:44
SPEAKER_00: and eventually you kind of get rid of it.
22:46
SPEAKER_00: But there is a kind of a foundational Bigtable.
22:49
SPEAKER_00: That Bigtable didn't use Colossus.
22:50
SPEAKER_00: There's the Colossus using the Bigtable,
22:52
SPEAKER_00: and then there's the normal Bigtable sitting on top of Colossus.
22:55
SPEAKER_00: And it all worked for years, so I don't even know when they got rid of that.
22:58
SPEAKER_00: They got rid of it at some point, but that was...
23:01
SPEAKER_01: But I guess it sounds like you can make hacks that go really long knowing
23:06
SPEAKER_01: that they're hacks and they get you off the ground, right?
23:08
SPEAKER_00: They get you off the ground, right?
23:09
SPEAKER_00: Because if we had to implement that kind of big table layer from the get-go,
23:13
SPEAKER_00: it would have just delayed how long it took to get Colossus built.
23:16
SPEAKER_01: One of the things that strike me about a system like Colossus is it promises,
23:23
SPEAKER_01: or this was internal to Google,
23:24
SPEAKER_01: but even distributed file systems that are external,
23:27
SPEAKER_01: they will promise high throughput, high availability, and low latency.
23:33
SPEAKER_01: And to me, it's always a bit conflicting of like, well, it's pretty easy to...
23:37
SPEAKER_01: How did you reconcile?
23:38
SPEAKER_01: How did you get
23:52
SPEAKER_01: I mean, these distributed file systems, I mean, the latency isn't super low.
23:55
SPEAKER_01: I mean, particularly when they're running on hard disks,
23:57
SPEAKER_01: which Colossus was doing at the
23:58
SPEAKER_00: time, which S3 does.
24:08
SPEAKER_00: So like S3 is a high performance system.
24:10
SPEAKER_00: Google GCS, the competitor from Google, which is built on top of Colossus.
24:14
SPEAKER_00: It's high performance,
24:15
SPEAKER_00: has incredible throughput,
24:17
SPEAKER_00: but the latency is like 20 to 30 milliseconds for first read.
24:20
SPEAKER_00: And that is bounded by your hard disk latency.
24:23
SPEAKER_00: If you put it on SSD,
24:24
SPEAKER_00: it gets down to closer to SSD latencies,
24:26
SPEAKER_00: but not actually kind of the state of art SSD latencies,
24:29
SPEAKER_00: which is kind of this crazy thing that's been happening in our industry.
24:31
SPEAKER_00: It's just like how much faster the hardware has been getting.
24:34
SPEAKER_00: Reading from a hard drive, maybe five to 10 milliseconds nowadays.
24:38
SPEAKER_00: I never touch hard drives.
24:39
SPEAKER_00: Reading from an SSD over NVMe, 30 microseconds, 50 microseconds.
24:44
SPEAKER_00: So that's a microseconds.
24:46
SPEAKER_00: There's a thousand microseconds in one millisecond.
24:48
SPEAKER_00: So we're talking like a huge, huge difference.
24:50
SPEAKER_01: Well, there are now startups or infrastructure companies that are starting
24:53
SPEAKER_01: to take advantage of the fact that they can they can have an NVMe layer and
24:57
SPEAKER_01: the end date they They pull things up either predictively or not.
25:00
SPEAKER_01: But as you say,
25:01
SPEAKER_01: when the physical reality changes,
25:03
SPEAKER_01: you can build systems on top of it that should take advantage of it.
25:06
SPEAKER_00: Yeah, absolutely.
25:08
SPEAKER_00: The disks are so much faster with SSDs.
25:10
SPEAKER_00: The networks are so much faster.
25:12
SPEAKER_00: I mean, just kind of crazy how fast the IntraZone leases are to Google or Amazon Center.
25:18
SPEAKER_00: It wasn't like that when we were building Colossus.
25:21
SPEAKER_00: I can't even remember what the numbers were, but milliseconds to do a network roundtrip?
25:24
SPEAKER_00: And now it's down in, you know, 100 microseconds within a zone.
25:28
SPEAKER_00: I mean, I just look at these things and I'm like,
25:30
SPEAKER_00: holy crap, you know, the hardware guys have really done a good job.
25:32
SPEAKER_01: Yeah, sometimes I feel that our software should feel way more snappy.
25:37
SPEAKER_01: And there are some snappy software,
25:38
SPEAKER_01: but sometimes I almost wonder if we're getting too complacent with all
25:41
SPEAKER_01: the abstractions or not even doing this like napkin maths.
25:45
SPEAKER_01: Simon Erickson at TurboBuffer talks about this napkin math word like you.
25:48
SPEAKER_01: It was like, all right, here's the theoretical limitation of the hardware.
25:52
SPEAKER_01: Yeah, yeah.
26:08
SPEAKER_00: And sometimes like when you're architecting something,
26:10
SPEAKER_00: you need to think about the human kind of perceptible latencies.
26:13
SPEAKER_00: But oftentimes when you're dealing with the storage system layer,
26:15
SPEAKER_00: you know,
26:16
SPEAKER_00: I know Simon working on Turbo Puffer, they're doing great stuff over there.
26:19
SPEAKER_00: You have to think about the machine scale and the machine speed,
26:21
SPEAKER_00: which is a lot faster than human perceptible.
26:24
SPEAKER_00: Like a human can tolerate maybe 100 milliseconds of delay,
26:27
SPEAKER_00: or if you're playing a game,
26:28
SPEAKER_00: maybe you need to have like, you know, frame rates of like every four milliseconds.
26:31
SPEAKER_00: But the machine wants much, much faster than that.
26:34
SPEAKER_00: Simon says, napkin math, I call this speed of light numbers.
26:37
SPEAKER_00: And like sometimes it's literally the speed of light bottlenecking you.
26:41
SPEAKER_00: The cross zone latencies between zones, cross region latencies is speed of light and fiber.
26:45
SPEAKER_00: You want to hear a kind of crazy fact?
26:47
SPEAKER_01: I love hearing crazy facts.
26:48
SPEAKER_00: The fastest way to send a pack of data across the globe is to send it into space.
26:53
SPEAKER_01: Is it because the speed of light is faster in vacuum?
26:57
SPEAKER_01: Quite a bit faster.
26:58
SPEAKER_01: Well, it's not full vacuum.
27:00
SPEAKER_01: No way.
27:01
SPEAKER_01: So you cover a higher distance?
27:03
SPEAKER_00: Yeah, well, you actually, I believe the way to do this, you send it straight up.
27:06
SPEAKER_00: And then with Starlink,
27:08
SPEAKER_00: you send it straight up,
27:08
SPEAKER_00: you bounce around between Starlink, you send it down to the other side.
27:11
SPEAKER_00: So you want to get it out of the atmosphere as quickly as possible.
27:13
SPEAKER_01: But now, if it's on my speed up, The speed of light is only the speed of light in vacuum.
27:17
SPEAKER_01: In every other medium, it's slower.
27:35
SPEAKER_01: I was talking, I did a deep dive on the hedge fund industry and they didn't tell me exactly.
27:40
SPEAKER_01: They said that they do use satellites and microwaves and some of these things
27:43
SPEAKER_01: that they will not tell you because, you know, this is their thing.
27:45
SPEAKER_01: But I had a suspicion that they might have found a faster way.
27:51
SPEAKER_01: And I think this is like somewhat well known,
27:52
SPEAKER_01: but the details they're not going to get into because again,
27:55
SPEAKER_01: but like, yeah, so they're
27:57
SPEAKER_00: probably balancing stuff in space.
27:58
SPEAKER_00: Yeah, yeah.
27:59
SPEAKER_00: So, I mean,
27:59
SPEAKER_00: one of the things that they do in the high frequency train is like between New York
28:03
SPEAKER_00: and Chicago,
28:03
SPEAKER_00: it's actually not I mean,
28:05
SPEAKER_00: there's so many layers you have to be paying attention to in terms of performance.
28:08
SPEAKER_00: The rabbit hole just goes so deep there.
28:10
SPEAKER_00: You're almost certainly running on multithreaded systems, right?
28:13
SPEAKER_00: And you're like, oh, I need to have a multithreaded program.
28:15
SPEAKER_01: I need to have synchronization in there.
28:34
SPEAKER_00: Well, you get the best performance if you just kind of avoid the synchronization.
28:37
SPEAKER_00: And part of this is lock-free programming,
28:39
SPEAKER_00: but part of it is arranging so you don't need locks at all.
28:41
SPEAKER_00: You need to carry about your processor caches.
28:43
SPEAKER_00: And there's this whole, like, kind of setup of caches above the CPU.
28:47
SPEAKER_00: Like, you can think of registers as a cache, then you have L1, L2, L3.
28:51
SPEAKER_00: Even your memory and onto disk.
28:53
SPEAKER_00: And there's just, you have to pay attention to all those levels.
28:56
SPEAKER_00: And if you do, your performance gets way better.
28:59
SPEAKER_00: And if you ignore it and you're like, oh,
29:00
SPEAKER_00: I'm not worrying about kind of,
29:01
SPEAKER_00: kind of,
29:02
SPEAKER_00: you know,
29:02
SPEAKER_00: kind of the cache accesses,
29:04
SPEAKER_00: I'm just accessing data all over, your program will just be way, way slower.
29:09
SPEAKER_00: Some folks pay a ton of attention to this.
29:11
SPEAKER_00: The high frequency trading, I mean,
29:12
SPEAKER_00: they do this all day long and in so many other places,
29:15
SPEAKER_00: like no one pays attention to it and you get this kind of gradual degradation
29:19
SPEAKER_00: the performance of the hardware or the software.
29:22
SPEAKER_01: I did want to talk a bit more about low level stuff,
29:24
SPEAKER_01: but not about the speed of light,
29:25
SPEAKER_01: but low level data structures and programming language features.
29:30
SPEAKER_01: You made some contributions to the standard library, right?
29:33
SPEAKER_00: Well, I've done a couple, not quite standard libraries.
29:36
SPEAKER_00: So, I mean, I just like have been always fascinated by data structures.
29:39
SPEAKER_00: It's kind of awesome.
29:41
SPEAKER_00: I mean, I think just the algorithms in general,
29:43
SPEAKER_00: you're just like sorting algorithms, they're just kind of awesome.
29:45
SPEAKER_00: You could probably explain, you know, insertion sort and, you know, to kind of everything.
29:49
SPEAKER_01: Could you explain quick sort as well?
29:52
SPEAKER_00: Quick sort's a little bit harder.
29:53
SPEAKER_00: Right, that's the thing.
29:54
SPEAKER_01: Yeah.
29:55
SPEAKER_01: No, but it's a smart one.
29:56
SPEAKER_01: Yeah.
29:56
SPEAKER_00: And then you get to these levels of like,
29:58
SPEAKER_00: you know, it's like, oh, wait, someone really smart came up with this.
30:02
SPEAKER_00: So one of the things I worked on,
30:03
SPEAKER_00: just as a little bit of a side at Google at some point,
30:06
SPEAKER_00: was a colleague came to me and he was like, you know what?
30:09
SPEAKER_00: We're using the STL map structure all over the place.
30:12
SPEAKER_00: And the STL map structure is a balanced binary tree.
30:15
SPEAKER_00: I can't remember if it was red,
30:16
SPEAKER_00: black trees or one of these other balancing algorithms
30:19
SPEAKER_00: that many CS college students implement.
30:21
SPEAKER_00: And he came to me, he's like, I think we can do better.
30:23
SPEAKER_00: Because, you know, there's actually a cache problem here.
30:26
SPEAKER_00: Every time every node you're traversing down, you're going to a different cache line.
30:30
SPEAKER_00: And he was thinking about using something else,
30:32
SPEAKER_00: a skip list to do this,
30:33
SPEAKER_00: which that's another awesome data structure everybody should kind of look at.
30:37
SPEAKER_00: But at some point I was like, actually, this feels more like a B-tree.
30:40
SPEAKER_00: So I implemented B-trees a couple of times before and figured out, like,
30:45
SPEAKER_00: how to implement a B-tree that implemented almost all the semantics of the STL map.
30:50
SPEAKER_00: It couldn't quite do it perfectly.
30:51
SPEAKER_00: And the reason is...
30:53
SPEAKER_00: When you insert into a Btree node,
30:56
SPEAKER_00: you have to shift stuff around so you don't get pointer stability.
30:58
SPEAKER_00: This is just kind of fundamental.
31:00
SPEAKER_00: But if you can,
31:00
SPEAKER_00: you know, you don't need that for your use case, you actually can pack more data in.
31:04
SPEAKER_00: So the thing about a Btree,
31:05
SPEAKER_00: like the real easy way to describe this is like you just have a small list of items,
31:10
SPEAKER_00: like eight items.
31:11
SPEAKER_00: The best way to store that,
31:12
SPEAKER_00: if you want to,
31:13
SPEAKER_00: you know, kind of fast access in sorted order, is just to sort the items, right?
31:17
SPEAKER_00: Literally not to have a tree at all.
31:18
SPEAKER_01: Yeah.
31:18
SPEAKER_00: For like eight items.
31:19
SPEAKER_01: Just have it in a very simple list.
31:20
SPEAKER_00: Very, very,
31:21
SPEAKER_00: just an array,
31:22
SPEAKER_00: sort the array,
31:22
SPEAKER_00: and then you can either do a linear scan over it,
31:24
SPEAKER_00: you can do a binary search, and oftentimes linear scan is faster.
31:27
SPEAKER_00: And then you think about that,
31:27
SPEAKER_00: I'm like, well, if I want to store more and eight items, I can just have...
31:31
SPEAKER_00: One node that has eight items,
31:32
SPEAKER_00: and then I have another node,
31:33
SPEAKER_00: and then you have a parent node that connects them together.
31:36
SPEAKER_00: And that is essentially like the, you know, you build it, you think about building it bottom up.
31:39
SPEAKER_00: You start with just one node of eight items.
31:41
SPEAKER_00: Oh, I need to insert the ninth thing.
31:44
SPEAKER_00: I'll split into two chunks.
31:46
SPEAKER_00: And the two chunks,
31:47
SPEAKER_00: one will have four,
31:47
SPEAKER_00: the other will have five, and then you have a parent node that points to them.
31:50
SPEAKER_00: And then you just kind of recurse on that.
31:52
SPEAKER_00: That's the B-tree algorithm in a nutshell.
31:54
SPEAKER_00: Everybody go implement it.
31:55
SPEAKER_00: Actually,
31:55
SPEAKER_00: nobody should implement this anymore because nowadays we have something else
31:58
SPEAKER_00: that will implement this in all the optimizations because there's a crap ton
32:02
SPEAKER_00: of optimizations that you can do on a B-tree.
32:04
SPEAKER_01: But I just want to go back to this.
32:06
SPEAKER_01: There was already an existing implementation for maps.
32:09
SPEAKER_01: And then so your colleague looked at the code and said, I think we can do better.
32:14
SPEAKER_01: What I want to figure out is in my mind,
32:17
SPEAKER_01: someone sitting outside of a non I'm not involved in how some of these libraries
32:23
SPEAKER_01: or data structures are built.
32:26
SPEAKER_01: I always thought,
32:27
SPEAKER_01: and again,
32:27
SPEAKER_01: this might be naive,
32:28
SPEAKER_01: but really smart people sit down,
32:30
SPEAKER_01: they kind of look at the state of the art,
32:32
SPEAKER_01: they implement it, and there's no way it can be faster.
32:34
SPEAKER_01: In fact, I've had arguments in the past saying, like, oh, let's write a faster sorting thing.
32:38
SPEAKER_01: Like, surely it is the fastest.
32:40
SPEAKER_01: But if you could bring us a little bit of,
32:42
SPEAKER_01: like, how you've been inside, how it happens and how...
32:48
SPEAKER_01: Other people like yourself and your colleague can say like,
32:51
SPEAKER_01: oh, what, what if we, what if
32:52
SPEAKER_00: we try something else?
32:53
SPEAKER_00: Yeah.
32:53
SPEAKER_00: Yeah.
32:54
SPEAKER_00: So my recollection here is he was working on this kind of the big internal system.
32:59
SPEAKER_00: I think it was called Gaia that actually had the mapping from,
33:02
SPEAKER_00: you know,
33:02
SPEAKER_00: you log in,
33:03
SPEAKER_00: you have your user ID and you have to look this up and they were storing,
33:07
SPEAKER_00: you know,
33:08
SPEAKER_00: all like this,
33:08
SPEAKER_00: the map from user ID and email to whatnot, to the metadata about the user.
33:13
SPEAKER_00: In STL Maps,
33:14
SPEAKER_00: you just know it's like,
33:15
SPEAKER_00: well, there's a lot of memory usage here and it shows up on profiles.
33:17
SPEAKER_00: And then we're like, well, what can we do to do better?
33:20
SPEAKER_00: And that was kind of the genesis of it.
33:21
SPEAKER_00: And, you know, he happened to be working on it and he happened to be working with me.
33:25
SPEAKER_00: Like, we just started kind of noodling on this problem.
33:27
SPEAKER_00: Like, oh, can we do something better?
33:28
SPEAKER_00: And it's not one of these things like,
33:29
SPEAKER_00: I think now with Google,
33:30
SPEAKER_00: they have a whole team working on their kind of internal libraries.
33:34
SPEAKER_00: At the time,
33:34
SPEAKER_00: it was more of like,
33:35
SPEAKER_00: you know,
33:35
SPEAKER_00: everybody working on their own systems and contributing to a shared
33:39
SPEAKER_01: But I guess it still goes back to what you were just saying of like,
33:42
SPEAKER_01: just go down the layers, try to understand.
33:45
SPEAKER_01: And if something just doesn't add up,
33:46
SPEAKER_01: like suddenly like, oh,
33:47
SPEAKER_01: there's this big explosion of memory usage,
33:49
SPEAKER_01: like, you know, just ask the questions, why is this?
33:52
SPEAKER_01: And if you're able to, or you're happy to be like, oh, can we do something about it?
33:56
SPEAKER_01: Right.
33:56
SPEAKER_00: And one of the things that,
33:57
SPEAKER_00: you know,
33:57
SPEAKER_00: he observed early on,
33:58
SPEAKER_00: I think part of one of the things was it was like a map from integer ID
34:02
SPEAKER_00: to something else.
34:03
SPEAKER_00: And like,
34:04
SPEAKER_00: if you look at red black tree,
34:05
SPEAKER_00: every node,
34:06
SPEAKER_00: you have your kind of value that you're showing in the map,
34:09
SPEAKER_00: and then you have two pointers.
34:10
SPEAKER_00: You might have an energy ID that's like four ray pipes, and then two...
34:13
SPEAKER_00: So it's a waste, right?
34:14
SPEAKER_00: Yeah, and you look at that and you're like, ooh, that seems like a lot of overhead.
34:17
SPEAKER_01: You could use a bit.
34:18
SPEAKER_00: Well, the B tree actually has a lot better,
34:20
SPEAKER_00: it has better spatial locality, and that's what made it faster.
34:23
SPEAKER_00: But it was actually smaller as well at the same time, because you had less pointers involved.
34:28
SPEAKER_01: You also contributed to Go, right?
34:29
SPEAKER_01: Yeah.
34:30
SPEAKER_00: Well, that came later.
34:31
SPEAKER_00: Yeah.
34:31
SPEAKER_00: It came a lot later, but can we talk about that?
34:33
SPEAKER_00: Yeah.
34:34
SPEAKER_00: I mean, just one of these other things,
34:35
SPEAKER_00: you know, I,
34:35
SPEAKER_00: I pay attention to like,
34:36
SPEAKER_00: you know,
34:36
SPEAKER_00: when there's research papers coming out about new data structures
34:39
SPEAKER_00: and like hash tables,
34:41
SPEAKER_00: hash tables are like the,
34:42
SPEAKER_00: the, one of the earliest things you've learned about in college and data structures.
34:46
SPEAKER_00: Like how do I map keys to values where The ordering is unimportant.
34:50
SPEAKER_00: That's when the hash tables come in.
34:52
SPEAKER_00: And there is like, you know, the very earliest ways to do this.
34:54
SPEAKER_00: I've implemented hash tables multiple times.
34:56
SPEAKER_00: It's like,
34:57
SPEAKER_00: you take your key and it might be a string and you,
34:59
SPEAKER_00: you put through a function and it spits out an integer,
35:01
SPEAKER_00: and then you map that into an array of buckets.
35:04
SPEAKER_00: And if multiple things map to the same bucket, you have to have a link list.
35:07
SPEAKER_01: You have a link list.
35:07
SPEAKER_01: Yeah.
35:08
SPEAKER_01: This is naive implementation, right?
35:09
SPEAKER_00: Naive implementation used quite frequently.
35:11
SPEAKER_00: And over time, people discovered like a lot better ways to do hash tables.
35:15
SPEAKER_00: There's very, like that's called a chaining of your hash.
35:17
SPEAKER_00: There's another technique called open addressing,
35:19
SPEAKER_00: where instead of actually having a linked list,
35:21
SPEAKER_00: you just kind of hash it again and move on or kind of walk down
35:25
SPEAKER_00: to subsequent buckets to find out like,
35:27
SPEAKER_00: or subsequent slots to find out where you should be.
35:30
SPEAKER_00: And I remember reading about this new technique.
35:32
SPEAKER_00: It came out of some folks at Google.
35:33
SPEAKER_00: I believe it came out of their Swiss office because it was called Swiss tables.
35:39
SPEAKER_00: I believe that's where the naming came from.
35:42
SPEAKER_00: I'm not 100% sure about that, but I remember reading about it.
35:45
SPEAKER_00: And then I was working on Go for a long period of time, and Go has this built-in map structure.
35:50
SPEAKER_00: And it's a hash table.
35:51
SPEAKER_00: It's a very highly optimized hash table because the Go team is very competent,
35:56
SPEAKER_00: the Go runtime team.
35:57
SPEAKER_00: And various folks have taken an attempt at putting together a Swiss table implementation for Go.
36:03
SPEAKER_00: And I tested some of them and I was like, this is kind of fascinating what Swiss tables do.
36:06
SPEAKER_00: And I'll explain how it works in just a second.
36:08
SPEAKER_00: But I looked at it, it's like, well, it's really hard to beat the performance of the runtime.
36:12
SPEAKER_00: The runtime was really good.
36:14
SPEAKER_00: And I kept on a new lift on this for a little while.
36:16
SPEAKER_00: And eventually I ended up having to take this business trip to To India, to Bangalore.
36:23
SPEAKER_01: And so I was on a long flight.
36:24
SPEAKER_01: Long flight?
36:24
SPEAKER_01: No.
36:25
SPEAKER_01: Yeah.
36:25
SPEAKER_01: It always starts like this.
36:27
SPEAKER_00: And I'm just like, I'm just going to try to pull on this.
36:29
SPEAKER_00: I pulled on it sufficiently that I can get some of the benchmarks to be faster.
36:33
SPEAKER_01: Wow.
36:33
SPEAKER_00: And then I'm like,
36:34
SPEAKER_00: you know,
36:34
SPEAKER_00: that like is kind of like catnip for an engineer, like kind of make it all faster.
36:37
SPEAKER_00: Can I figure all the rest of it, you know, got some help from the runtime folks.
36:41
SPEAKER_00: There's a,
36:42
SPEAKER_00: An issue on the Go issue tracker that,
36:44
SPEAKER_00: you know,
36:44
SPEAKER_00: where other people have been attempting this because people propose like,
36:47
SPEAKER_00: hey, let's use Swiss tables.
36:48
SPEAKER_00: And like the Go folks were like, well, you know, we don't quite know all the details.
36:51
SPEAKER_00: We're going to have to navigate this and that.
36:52
SPEAKER_00: And there were some ideas there that combined them together and got to the point
36:56
SPEAKER_00: where it had kind of a complete implementation that was faster on most benchmarks,
37:00
SPEAKER_00: not quite all of them, but most of them.
37:02
SPEAKER_00: And then the Go folks eventually picked this up and pushed it over the finish line.
37:05
SPEAKER_01: And then so you came with the idea,
37:09
SPEAKER_01: you got to the point where you were able to show an implementation
37:12
SPEAKER_01: that showed how some of the benchmarks were faster,
37:14
SPEAKER_01: and then you started to work with some folks on the Go team too.
37:17
SPEAKER_00: Well, it wasn't quite that.
37:19
SPEAKER_00: I came up with an implementation that we ended up using at my company.
37:23
SPEAKER_00: It was good for our use case, but actually putting it into the runtime is a whole other...
37:27
SPEAKER_01: But then you just showed like,
37:29
SPEAKER_01: here's this implementation and then they kind of took the inspiration and the ideas.
37:33
SPEAKER_01: Yeah.
37:34
SPEAKER_00: They're like, well, this is great.
37:35
SPEAKER_00: We want to make all, they always are looking for ways to make it runtime faster.
37:38
SPEAKER_00: And there's like, you know.
37:39
SPEAKER_01: Oh, wait,
37:40
SPEAKER_01: and then so you did this in this contribution,
37:42
SPEAKER_01: this was a lot of years after you left Google, right?
37:44
SPEAKER_01: Yeah, yeah.
37:45
SPEAKER_01: This was from the outside.
37:46
SPEAKER_01: This is from the outside.
37:46
SPEAKER_01: That's awesome.
37:47
SPEAKER_00: Yeah, and other people contribute stuff to the outside as well.
37:49
SPEAKER_00: You know,
37:50
SPEAKER_00: we had another colleague,
37:52
SPEAKER_00: he contributed one of the CRC implementations, you know, adapting some stuff.
37:55
SPEAKER_00: CRC?
37:56
SPEAKER_00: Cyclic redundancy checksum.
37:58
SPEAKER_00: You know,
37:59
SPEAKER_00: Intel published some papers about here's how to do the CRC and assembly very fast
38:03
SPEAKER_00: and contributed one of the implementations.
38:04
SPEAKER_00: You see a number of those things where, you know, people just like are contributing externally.
38:08
SPEAKER_00: It's not a lot, honestly.
38:09
SPEAKER_00: I mean, I actually don't know the full details,
38:11
SPEAKER_00: but, you know, people are regularly contributing to these things.
38:14
SPEAKER_01: Peter just described how the Swiss table work came together using an issue tracker
38:17
SPEAKER_01: in the GoTracker with different people contributing to the work and then
38:21
SPEAKER_01: the Go team pushing all of this over the finish line.
38:24
SPEAKER_01: This is where I need to mention our season sponsor Linear,
38:26
SPEAKER_01: which is a place to coordinate work between humans as well as agents.
38:30
SPEAKER_01: One thing I've noticed about how most of us work with agents is how it's
38:34
SPEAKER_01: a pretty single player thing.
38:35
SPEAKER_01: You open a terminal UI, go back and forth with an agent, and it usually produces a PR.
38:40
SPEAKER_01: But the rest of your team has no idea what happened in that chat
38:43
SPEAKER_01: unless you tell them or copy the whole history.
38:46
SPEAKER_01: And when everyone in the team works like this,
38:47
SPEAKER_01: a lot of work happens that's invisible to the rest of the team.
38:50
SPEAKER_01: Linear's take is that agent work should be teamwork.
38:53
SPEAKER_01: Even today, teams already use Linear to define the work to be done.
38:56
SPEAKER_01: Now, Linear can already delegate an issue to a coding agent.
38:59
SPEAKER_01: This agent could be an AI agent that Linear integrates with,
39:02
SPEAKER_01: like Codex or Cursor, or Linear's own agent, or a custom agent.
39:06
SPEAKER_01: Either way, the engineer in delegating stays responsible for the outcome.
39:09
SPEAKER_01: What I really like about how Linear works is how the work stays visible.
39:13
SPEAKER_01: Your teammates can follow the session of the agent,
39:15
SPEAKER_01: check out the PR producers, and join the review.
39:17
SPEAKER_01: We've gone from single player work to multiplayer engineering work with agents.
39:21
SPEAKER_01: Oh, and one more thing I like about linear, a focus on costs.
39:24
SPEAKER_01: Linear agents auto routing chooses a model that is the best suited for the task.
39:29
SPEAKER_01: Teams can also inspect usage and set limits,
39:30
SPEAKER_01: track usage,
39:31
SPEAKER_01: so you can use capable agents without having costs balloon out of control.
39:35
SPEAKER_01: Hop on board at linear.app slash pragmatic.
39:38
SPEAKER_01: Peter also previously mentioned Gaia, Google's internal system that mapped logins to users.
39:43
SPEAKER_01: It's not surprising that Google custom built all systems,
39:45
SPEAKER_01: including this one, but most of us won't build our internal Gaia.
39:49
SPEAKER_01: This brings us to our season sponsor, WorkOS.
39:52
SPEAKER_01: You can think of WorkOS as something like Gaia for the rest of us,
39:55
SPEAKER_01: identity infrastructure you've otherwise spent quarters building yourself.
39:59
SPEAKER_01: WorkOS includes single sign-on,
40:00
SPEAKER_01: skim directory sync,
40:01
SPEAKER_01: audit logs,
40:02
SPEAKER_01: role-based access control,
40:03
SPEAKER_01: basically everything a big customer security team asks for,
40:06
SPEAKER_01: delivered as a handful of clean APIs.
40:08
SPEAKER_01: It's how companies go from,
40:10
SPEAKER_01: we have a login, to,
40:11
SPEAKER_01: we can sell to a Fortune 500
40:13
SPEAKER_01: without standing up their own internal identity platform.
40:15
SPEAKER_01: And WorkOS is already building for the next version of the agentic authorization problem.
40:19
SPEAKER_01: Their newest product is Airlock, the authorization layer for AI agents.
40:23
SPEAKER_01: Think about what happens when you hand an agent a task like clean up
40:27
SPEAKER_01: the sale opportunities in our pipeline.
40:29
SPEAKER_01: The last thing you want is for this thing to have standing permission
40:32
SPEAKER_01: to delete whatever it likes.
40:34
SPEAKER_01: Airlock sits between your agent and the tools they call.
40:38
SPEAKER_01: It evaluates every request against the agent's intent and your rules and
40:42
SPEAKER_01: then it allows it, denies it, or routes it to a human for approval.
40:46
SPEAKER_01: You write down the policies in plain language.
40:48
SPEAKER_01: The agent never sees And every call and verdict gets logged.
40:53
SPEAKER_01: It works with coding agents like Cloud Code and Codex and with MCP Gateways.
40:57
SPEAKER_01: So if you're working out how to let agents do real work in production without
41:01
SPEAKER_01: over permissioning them, take a look at WorkOS Airlock at workos.com slash airlock.
41:06
SPEAKER_01: And with this, let's get back to Peter and why he left Google after building Colossus.
41:11
SPEAKER_01: You're at Google, you're building Colossus distributed file systems.
41:14
SPEAKER_01: You're at this point probably working on probably the largest system on the planet, honestly.
41:20
SPEAKER_01: Why did you even consider leaving?
41:22
SPEAKER_00: Yeah, yeah.
41:23
SPEAKER_00: Well, after Colossus,
41:24
SPEAKER_00: I kind of dabbled in this other project called Google Goggles for a little while.
41:28
SPEAKER_00: Remember the glass holes?
41:29
SPEAKER_00: Yeah.
41:30
SPEAKER_01: It keeps coming back, the idea, by the way.
41:32
SPEAKER_00: Yeah, yeah.
41:32
SPEAKER_00: Now it's still here and present.
41:34
SPEAKER_01: Seems like Google's early.
41:35
SPEAKER_00: Yeah, yeah.
41:36
SPEAKER_00: And, you know, I think that was a technology before its time.
41:39
SPEAKER_00: I don't think it was ready to do at that point.
41:40
SPEAKER_00: It looks like actually doing the glasses is quite a bit harder.
41:44
SPEAKER_00: You know, the Android phones we were trying to power it on were, you know, not powerful enough.
41:49
SPEAKER_00: And then,
41:49
SPEAKER_00: you know,
41:49
SPEAKER_00: I just kind of got wanderlust,
41:50
SPEAKER_00: you know, like,
41:51
SPEAKER_00: you know,
41:51
SPEAKER_00: am I just kind of stagnating here at Google,
41:53
SPEAKER_00: which is a strange thing to say, but, you know, some other people feel it as well.
41:57
SPEAKER_00: And decided to go off and try my hand at another startup.
42:00
SPEAKER_00: That didn't work out.
42:01
SPEAKER_00: We got aqua hired by Square.
42:02
SPEAKER_01: So I want to pause for a second.
42:05
SPEAKER_01: So this company, what's the company name?
42:07
SPEAKER_00: The company that we founded
42:08
SPEAKER_01: is called Viewfinder.
42:09
SPEAKER_01: Viewfinder, yeah.
42:11
SPEAKER_00: It was in the mobile photo sharing space, which sounds familiar.
42:14
SPEAKER_00: This is like Instagram.
42:15
SPEAKER_00: This is like Snapchat.
42:16
SPEAKER_01: This was in 2012.
42:17
SPEAKER_01: Right.
42:17
SPEAKER_01: Right, as Instagram and all of them were taking off.
42:20
SPEAKER_01: yeah.
42:20
SPEAKER_00: No,
42:20
SPEAKER_00: we were right there in the play and we just didn't have the right go-to-market kind
42:24
SPEAKER_00: of like how to attract the users, how to get viral growth kind of thing.
42:27
SPEAKER_01: Because from the outside,
42:28
SPEAKER_01: like what I read when I checked,
42:30
SPEAKER_01: you know, the story, just like, oh, you know, like you co-founded a startup.
42:34
SPEAKER_01: It got acquired by Square.
42:36
SPEAKER_01: Hooray.
42:36
SPEAKER_01: Like it sounds like you had bigger ambitions.
42:38
SPEAKER_01: Yeah.
42:39
SPEAKER_01: This was a decent outcome, but not the dream, right?
42:43
SPEAKER_01: yeah.
42:43
SPEAKER_00: No, it wasn't the dream at all.
42:45
SPEAKER_00: So the term I used was acquihired.
42:47
SPEAKER_00: So sometimes a company will get acquired,
42:49
SPEAKER_00: get bought for their IP, for their product, for their business.
42:54
SPEAKER_00: And other times they get bought just for the talent.
42:56
SPEAKER_01: The people.
42:56
SPEAKER_00: The people.
42:57
SPEAKER_00: And we got bought just for the talent.
42:58
SPEAKER_00: They acquired the IP, but I don't think Square ever did anything with it.
43:01
SPEAKER_00: It wasn't kind of like where they were working.
43:03
SPEAKER_00: But we built up a kind of strong technical team, and that's what we were hired for.
43:07
SPEAKER_00: And, you know, like, I can't remember the details.
43:09
SPEAKER_00: We'd raised a small amount of money.
43:10
SPEAKER_00: We were able to pay our investors back, make them whole.
43:13
SPEAKER_00: Maybe they got a little bit of a haircut, but maybe they actually got a little bit of a...
43:16
SPEAKER_00: But it was...
43:18
SPEAKER_00: Essentially,
43:18
SPEAKER_00: they got their money back,
43:19
SPEAKER_00: which is like, you know, as a founder, you know, investors are big boys.
43:25
SPEAKER_00: They're used to losing their money,
43:26
SPEAKER_00: but you kind of feel bad if you lose a lot of money for them.
43:28
SPEAKER_00: So, you know, getting them paid back kind of makes you feel a little bit better.
43:31
SPEAKER_01: Yeah.
43:32
SPEAKER_01: Yeah.
43:32
SPEAKER_01: And then you spent a little time at the company that acquired, you were just square.
43:36
SPEAKER_01: Yeah.
43:36
SPEAKER_01: And then...
43:37
SPEAKER_01: You started itching that a little bit again?
43:38
SPEAKER_01: Yeah, yeah, yeah.
43:39
SPEAKER_00: Because,
43:39
SPEAKER_00: you know,
43:40
SPEAKER_00: we've been working on these,
43:40
SPEAKER_00: you know, distributed file systems and storage systems, you know, Colossus.
43:44
SPEAKER_00: One of the kind of sister projects to Colossus is Spanner.
43:49
SPEAKER_01: How is Spanner different to Colossus?
43:51
SPEAKER_00: Well, Spanner is essentially a distributed database.
43:55
SPEAKER_00: Colossus is a distributed storage system.
43:57
SPEAKER_00: And the way I think about the difference between a distributed storage system and
44:01
SPEAKER_00: a distributed database, you might think, oh, they're both storing data.
44:04
SPEAKER_01: I was about to ask, because a distributed database will at some point be a storage system.
44:10
SPEAKER_01: Right, right.
44:10
SPEAKER_01: So what's the difference?
44:11
SPEAKER_00: Yeah.
44:11
SPEAKER_00: So for Colossus, you know, it was targeting large files, large append-only files.
44:15
SPEAKER_00: You can't update them.
44:16
SPEAKER_01: Append-only, yeah.
44:17
SPEAKER_00: Yeah, large append-only files.
44:19
SPEAKER_00: So, you know, 64 megabytes, maybe up to gigabytes in size.
44:23
SPEAKER_00: But if you're a database,
44:24
SPEAKER_00: you want to be storing like kind of small, like,
44:26
SPEAKER_00: you know,
44:27
SPEAKER_00: kind of,
44:27
SPEAKER_01: you know,
44:27
SPEAKER_01: if you're
44:28
SPEAKER_00: using SQL or like the relational data,
44:30
SPEAKER_00: you might have a table with, you know, billions or billions of, you know, rows.
44:35
SPEAKER_00: Those rows are broken up into columns.
44:36
SPEAKER_00: The columns are typed.
44:38
SPEAKER_00: It just has a very different nature to the engineering challenge for a database
44:41
SPEAKER_00: than it does for a distributed storage system.
44:44
SPEAKER_00: And usually distributed databases are implemented on top of some kind
44:47
SPEAKER_00: of distributed storage system.
44:49
SPEAKER_00: And that was the relationship.
44:50
SPEAKER_00: So Spanner was implemented on top of Colossus.
44:52
SPEAKER_00: And some of the design decisions in Spanner kind of directly fell out of
44:57
SPEAKER_00: the append-only nature of the files in...
45:00
SPEAKER_00: Colossus.
45:00
SPEAKER_00: You can't update a file in place,
45:02
SPEAKER_00: so you have to, you know, make the files immutable in your database.
45:05
SPEAKER_00: And this is where, like, you know, log structured merge trees, you know, come into play.
45:09
SPEAKER_00: And they weren't invented at Google,
45:11
SPEAKER_00: but Google really popularized them with LevelDB,
45:14
SPEAKER_00: which emerged So how did you decide to found Cockroach Labs?
45:16
SPEAKER_00: While we were working at Square,
45:17
SPEAKER_00: my co-founder and I,
45:18
SPEAKER_00: it was actually three of us,
45:19
SPEAKER_00: we were all at Square,
45:20
SPEAKER_00: and one of them is Spencer,
45:21
SPEAKER_00: and I mentioned earlier, he was working on the GIMP, he's my college roommate,
45:45
SPEAKER_00: He was also at Google.
45:46
SPEAKER_00: He was also at Google.
45:47
SPEAKER_00: This other guy,
45:48
SPEAKER_00: Ben Darnell,
45:49
SPEAKER_00: who was also at Google, he joined us at Viewfinder and ended up at Square.
45:52
SPEAKER_00: We were kind of just noodling on a project to do,
45:55
SPEAKER_00: and we'd actually had this design back in Viewfinder.
45:59
SPEAKER_00: We looked around for a data space to be using in order to build Viewfinder on top of.
46:05
SPEAKER_00: We didn't really like the things that were out there.
46:07
SPEAKER_00: The technologies inside Google looked better.
46:09
SPEAKER_00: We had the big table, we had spanner and whatnot.
46:10
SPEAKER_00: And we were looking around and, you know, like HBase existed, but I wasn't quite happy.
46:14
SPEAKER_00: And there's some other systems like React and others.
46:16
SPEAKER_00: And, you know,
46:18
SPEAKER_00: at one point we were just kind of like,
46:19
SPEAKER_00: you know, came up with the design for Cockroach to be an initial design.
46:23
SPEAKER_00: And I was like, no, no, guys, we're doing a mobile photo sharing site.
46:26
SPEAKER_00: We shouldn't build a distributed, you know, database.
46:29
SPEAKER_00: Put it on the back burner.
46:30
SPEAKER_00: Which I think was absolutely the right thing, maybe.
46:32
SPEAKER_00: Or maybe we should have just pivoted away from doing a mobile photo sharing site,
46:35
SPEAKER_00: given the way things worked out.
46:36
SPEAKER_00: And then we got to Square and we saw some of the same problems
46:38
SPEAKER_00: that they were experiencing with data storage systems.
46:40
SPEAKER_00: And Spencer was very convincing.
46:41
SPEAKER_00: Miners convinced some of the management that, like,
46:43
SPEAKER_00: hey, can you just work on this part-time to see if it had life behind the design?
46:47
SPEAKER_00: And then he kind of conscripted Ben and I into it.
46:50
SPEAKER_00: And eventually he started getting attention externally.
46:53
SPEAKER_00: And we're like, hey, can we go and spin this off into a company?
46:55
SPEAKER_00: And that's what ended up happening.
46:57
SPEAKER_01: And so you started a company, but I understand you didn't raise VC funding initially, right?
47:05
SPEAKER_01: That was the case at Viewfinder.
47:08
SPEAKER_00: We did it differently.
47:09
SPEAKER_00: At Viewfinder, we kind of eschewed the VC money.
47:12
SPEAKER_00: And, you know, in hindsight, I wouldn't recommend that.
47:15
SPEAKER_01: You wouldn't recommend?
47:16
SPEAKER_00: Yeah, I wouldn't recommend taking the VC money.
47:18
SPEAKER_00: Because in my experience, the VCs are very, very intelligent.
47:21
SPEAKER_00: They can help you navigate a lot of challenges.
47:23
SPEAKER_00: You know,
47:23
SPEAKER_00: I think sometimes there's this perception,
47:25
SPEAKER_00: you know, the VCs will push you into various areas.
47:27
SPEAKER_00: And maybe there's some bad ones out there that do.
47:29
SPEAKER_00: The VCs I've had experience with, just like some of the Sherpas people, you know, I've ever met.
47:34
SPEAKER_01: So you kind of have like an extra person helping you on the team, pretty much.
47:39
SPEAKER_00: Mentoring you, giving you guidance, telling you what they're seeing.
47:41
SPEAKER_00: They give you advice seeing what they're seeing in the market,
47:43
SPEAKER_00: where things are going that is very hard sometimes for a founder.
47:47
SPEAKER_01: Especially as a technical founder, right?
47:48
SPEAKER_01: You're focused on the engineering part.
47:50
SPEAKER_00: Yeah, yeah.
47:50
SPEAKER_00: No, we actually took money right away for Cockroach Labs.
47:53
SPEAKER_00: It was almost like as soon as we left,
47:55
SPEAKER_00: we did a little roadshow, you know, kind of in the Bay Area.
47:58
SPEAKER_00: We got some interest and, you know, got an investor right away.
48:02
SPEAKER_01: I have to ask you about the name, though.
48:04
SPEAKER_01: How did the name of Cockroach come?
48:07
SPEAKER_00: So, you know, we named the GIMP.
48:10
SPEAKER_00: Yes.
48:11
SPEAKER_00: That was mine.
48:12
SPEAKER_00: Pulp Fiction had come out in college and I'm like, oh, what should we name this thing?
48:15
SPEAKER_00: Oh, the GNU Image Manipulation Program.
48:16
SPEAKER_00: I think we were thinking Image Manipulation Program initially.
48:19
SPEAKER_00: And I'm like, oh, GIMP, it's obvious.
48:21
SPEAKER_00: And it just stuck.
48:22
SPEAKER_00: And at some point,
48:22
SPEAKER_00: you know,
48:22
SPEAKER_00: we were newly on this new database and you kind of want to give things a name.
48:26
SPEAKER_00: You can't just say, oh, we're working on this distributed database.
48:29
SPEAKER_00: You kind of need something.
48:29
SPEAKER_00: And Spencer was like, oh, CockroachDB.
48:32
SPEAKER_00: Like, Cockroaches are unkillable.
48:33
SPEAKER_00: I want these things, this database to be unkillable.
48:35
SPEAKER_00: You know, Cockroaches is going to survive the nuclear apocalypse.
48:39
SPEAKER_00: So that was where the genesis was and just stuck.
48:42
SPEAKER_01: Yeah, whenever a bunch of our nodes goes down, this thing will still be up.
48:45
SPEAKER_01: Yeah, yeah.
48:46
SPEAKER_00: And, you know, this is where we're at today with CockroachDB.
48:48
SPEAKER_00: It's like one of the things that I point out, I'm like, holy crap, this is awesome.
48:52
SPEAKER_00: You kill a node,
48:52
SPEAKER_00: we did this whole campaign last year,
48:54
SPEAKER_00: which was really just to prove out something that had already been present.
48:57
SPEAKER_00: The campaign was performance under adversity,
48:58
SPEAKER_00: but just like you can run a workload against it,
49:00
SPEAKER_00: you can kill a node,
49:01
SPEAKER_00: you can sometimes kill a whole region, and the system keeps on going.
49:05
SPEAKER_00: And it's like stories like that,
49:06
SPEAKER_00: you know, like what we did on the marketing side there, but also...
49:08
SPEAKER_00: We hear this from our customers too.
49:09
SPEAKER_00: They've had fires in data centers and all the other data systems go down
49:13
SPEAKER_00: and CockroachDB keeps on going.
49:14
SPEAKER_00: I'm like, that's awesome.
49:15
SPEAKER_01: When you started out,
49:16
SPEAKER_01: who were companies,
49:19
SPEAKER_01: startups who wanted to use CockroachDB and how has it changed since?
49:24
SPEAKER_01: Because, you know, like just making the case like, okay, I'm starting a startup.
49:28
SPEAKER_01: It's a small startup.
49:29
SPEAKER_01: Like I will need a I'm trying to get a sense of
49:32
SPEAKER_01: at what point did companies tell themselves, we need something distributed.
49:57
SPEAKER_01: Yeah.
49:57
SPEAKER_01: In a database.
49:58
SPEAKER_00: Yeah.
49:59
SPEAKER_00: I mean, oftentimes we have companies calling us up after they've had a disaster.
50:04
SPEAKER_01: So, so like a node went up or a hardware Hard drive fail, that kind of stuff.
50:08
SPEAKER_00: It's not quite like we're ambulance chasers,
50:10
SPEAKER_00: but if you see an outage,
50:12
SPEAKER_00: like a big outage from some company, we'll sometimes be trying to knock them up.
50:16
SPEAKER_00: But also, they will call us.
50:19
SPEAKER_00: There's a very big bank who's now a customer.
50:22
SPEAKER_00: Don't think I can name them, but you can go read.
50:24
SPEAKER_00: They had a very serious outage due to a weather event and
50:29
SPEAKER_01: Which probably knocked down,
50:30
SPEAKER_01: I'm assuming,
50:31
SPEAKER_01: a region or database or a networking cable or a tree fell on something.
50:34
SPEAKER_00: I think it knocked down a whole region.
50:36
SPEAKER_00: It was a region-wide power out.
50:37
SPEAKER_00: It knocked down the region.
50:38
SPEAKER_00: And there's a mandate from the CEO.
50:40
SPEAKER_00: It's like, no, we just have to be able to survive these things.
50:42
SPEAKER_00: And that gets pushed down all the way.
50:44
SPEAKER_00: And you see this in other places where,
50:46
SPEAKER_00: you know,
50:46
SPEAKER_00: one of our early customers,
50:47
SPEAKER_00: they were running on AWS and they just got to the maximum size you can run
50:51
SPEAKER_00: in Aurora instance on.
50:53
SPEAKER_00: And then what would typically happen at that point is then you have to shard your database.
50:57
SPEAKER_00: You know, this is very standard practice.
50:59
SPEAKER_00: But your single node database, you create 10 or 20 or 100 charts.
51:03
SPEAKER_00: And this is what Google done for some period of time.
51:05
SPEAKER_00: And that's a heavy burden on the application developer.
51:08
SPEAKER_00: And the way we always phrase this is like,
51:10
SPEAKER_00: I mean, the application developer is becoming a database developer at that point.
51:13
SPEAKER_00: And they're doing it poorly.
51:14
SPEAKER_00: You know, they're trying to implement distributed transactions or indexes and whatnot.
51:17
SPEAKER_00: We felt the burden for that belongs on the database developer.
51:20
SPEAKER_01: Can we talk about automatic sharding?
51:22
SPEAKER_01: I think it's safe to some,
51:23
SPEAKER_01: most of us will know what sharding is when you're,
51:26
SPEAKER_01: but actually let's start from like manual sharding and
51:30
SPEAKER_01: then how you can implement automatic sharding.
51:32
SPEAKER_01: And if you can tell us like,
51:33
SPEAKER_01: you know,
51:34
SPEAKER_01: tactics that a database like CockroachDB can do to actually just take that load off
51:38
SPEAKER_01: of you.
51:39
SPEAKER_00: Yeah, yeah.
51:39
SPEAKER_00: So I think the very basic form of sharding is a little bit like the hash table.
51:43
SPEAKER_00: Let's say, you know, you have a fixed number of shards.
51:46
SPEAKER_00: Like, let's say it's just 100 shards.
51:47
SPEAKER_00: Your data model is a user with a lot of data associated with the user.
51:51
SPEAKER_00: You just take the user and you say like, oh, they map them to one of the shards.
51:54
SPEAKER_00: And, you know, you kind of just rely on the hash function to get like fairly even distribution.
51:59
SPEAKER_00: The problem with this is at some point,
52:01
SPEAKER_00: you know, one of your shards will get full and you have to kind of re-shard.
52:04
SPEAKER_00: And that's a very, very onerous process.
52:06
SPEAKER_01: And re-shard,
52:07
SPEAKER_01: I guess,
52:07
SPEAKER_01: a simple way to do is like,
52:08
SPEAKER_01: if it's just a hard drive,
52:09
SPEAKER_01: I don't know,
52:10
SPEAKER_01: per node where you write the user data,
52:12
SPEAKER_01: it gets full and you're like, okay, well, I now need to split it somehow.
52:16
SPEAKER_01: I need to move it.
52:17
SPEAKER_01: I need to remap it.
52:18
SPEAKER_01: I need to rejig my metadata, which knows where this data lives, that kind of stuff.
52:21
SPEAKER_01: Yeah, yeah.
52:22
SPEAKER_00: And it depends on exactly how you're doing that mapping from the user ID
52:25
SPEAKER_00: or whatever your shard key is to the shard.
52:28
SPEAKER_00: You might have to remap them all, right?
52:30
SPEAKER_00: This is very typical.
52:30
SPEAKER_00: Exactly.
52:32
SPEAKER_00: And this happens in hash tables where oftentimes in order to grow the hash table,
52:35
SPEAKER_00: you just have to essentially create a new hash table,
52:37
SPEAKER_00: double the size and copy all the data over.
52:39
SPEAKER_00: Now, that's kind of like the very basic, straightforward way.
52:42
SPEAKER_00: And there's...
52:43
SPEAKER_00: Various levels of complexity you add on it.
52:45
SPEAKER_00: One of them is called consistent hashing, and there's various techniques to do this.
52:49
SPEAKER_00: It's kind of fascinating how they all work.
52:52
SPEAKER_00: But in consistent hashing,
52:53
SPEAKER_00: you can add an additional node,
52:55
SPEAKER_00: and then it only moves a fraction of the data from each shard over there.
52:58
SPEAKER_00: And there's various systems that do that, and I believe this is like Winter Lies Cassandra.
53:02
SPEAKER_00: The way CockroachDB does it is more akin to Bigtable,
53:05
SPEAKER_00: more akin to Spanner,
53:06
SPEAKER_00: more akin to HBase,
53:07
SPEAKER_00: where instead of actually hashing,
53:09
SPEAKER_00: we actually take, you know, you can imagine All your keys in a system.
53:12
SPEAKER_00: And this is always true in any system.
53:14
SPEAKER_00: You can imagine this in one big contiguous key space,
53:16
SPEAKER_00: and then you kind of partition contiguous spans of that.
53:19
SPEAKER_00: And then you have to build up an index on top of those contiguous spans.
53:23
SPEAKER_00: And what I just described there actually sounds a lot like a B-tree.
53:27
SPEAKER_00: So there's this index on top that maps you from,
53:31
SPEAKER_00: you know, like, I need to have this range, which node is it on?
53:34
SPEAKER_00: And this is a little bit like a B-tree.
53:35
SPEAKER_00: You know,
53:36
SPEAKER_00: you kind of squint,
53:36
SPEAKER_00: you know,
53:37
SPEAKER_00: it's like, I think you squint and everything's either a B-tree or it's a hash table.
53:39
SPEAKER_00: And You know, that index structure.
53:41
SPEAKER_01: But it's now starting to make sense because I remember when I read,
53:45
SPEAKER_01: it might have been the Wikipedia article on vtrees,
53:47
SPEAKER_01: it said this is data structure that is frequently used in databases.
53:51
SPEAKER_01: It's now coming back to me because I didn't think too much of it.
53:54
SPEAKER_01: I'm not someone who builds databases.
53:56
SPEAKER_01: But now that we're talking, we just organically keep touching on vtrees again and again.
54:00
SPEAKER_00: And the other place it comes up in databases.
54:02
SPEAKER_00: So this is where it kind of comes up in distributed databases.
54:05
SPEAKER_00: And no one ever really calls it a vtrees.
54:07
SPEAKER_00: I just kind of squint sometimes and I see like, Actually kind of a B-tree.
54:10
SPEAKER_00: But the other place it comes up in databases is for your indexes.
54:14
SPEAKER_00: So if you have a table and you have like an index on your email address and you want
54:19
SPEAKER_00: to be able to scan over those email addresses in order,
54:21
SPEAKER_00: that's a B-tree under the hood in a database.
54:23
SPEAKER_00: And, you know, like any kind of index you have usually provides sorted order.
54:26
SPEAKER_00: There are I think that paper was written back in the 80s, and they're still ubiquitous today.
54:32
SPEAKER_00: They're the foundations of single node databases,
54:34
SPEAKER_00: and pretty much every data system I've worked on has had B-trees
54:37
SPEAKER_00: at some point placed in them.
54:47
SPEAKER_01: I want to ask about strong consistency.
54:50
SPEAKER_01: So CockroachDB offers strong consistency.
54:53
SPEAKER_01: Now for people who are a bit more newcomers to distributed systems,
54:57
SPEAKER_01: can we talk about the consistency models and
54:59
SPEAKER_01: then why strong consistency is important and why it's hard to implement it?
55:03
SPEAKER_00: So, I mean,
55:03
SPEAKER_00: there's multiple ways to kind of approach this,
55:05
SPEAKER_00: but I mean, if you've used a database, you've probably heard of transactions.
55:09
SPEAKER_00: And transactions are a way to perform a whole bunch of mutations atomically.
55:13
SPEAKER_00: So databases talk about atomicity, consistency, isolation, durability.
55:18
SPEAKER_00: The Atomicity is really easy.
55:19
SPEAKER_00: It's like when I write to the database, it has to be deliberately written.
55:21
SPEAKER_00: So if anything crashes, it comes back.
55:24
SPEAKER_00: The Atomicity is just referring to the fact I want to do a whole bunch of changes.
55:28
SPEAKER_00: I want them all committed or all aborted at the same time.
55:31
SPEAKER_00: I don't want to have like some kind of partial operation.
55:33
SPEAKER_00: And why is this important?
55:34
SPEAKER_00: Why is the Atomicity important?
55:36
SPEAKER_00: Well, the Atomicity is what gets you to the point where it's like as an application,
55:39
SPEAKER_00: I can do a bunch of operations.
55:41
SPEAKER_00: And if there's an error,
55:42
SPEAKER_00: an error occurs,
55:43
SPEAKER_00: it all kind of gets rolled back and it's a much simpler And then there's
55:48
SPEAKER_00: the consistency in isolation, which kind of get muddled.
55:51
SPEAKER_00: The isolation is referring to isolation between transactions.
55:54
SPEAKER_00: I don't just want to run one transaction at a time.
55:57
SPEAKER_00: That's easy to do.
55:58
SPEAKER_00: I want to run a lot in parallel and make it so that when So when they're running
56:01
SPEAKER_00: in parallel,
56:02
SPEAKER_00: they are running as concurrently as possible,
56:04
SPEAKER_00: but you want to have the appearance when they're running as concurrently
56:07
SPEAKER_00: as possible, that there is kind of a serial order to them.
56:11
SPEAKER_00: So this is like kind of the whole trick,
56:13
SPEAKER_00: the kind of the gold standard for isolation is called Don't worry about that.
56:18
SPEAKER_00: The step down from that is called serializability,
56:20
SPEAKER_00: and that literally refers to having a serial order of your transactions,
56:24
SPEAKER_00: but you're having to construct that in a way that you're doing everything
56:27
SPEAKER_00: as concurrently as possible.
56:29
SPEAKER_00: And the benefit of this,
56:30
SPEAKER_00: the serializability,
56:31
SPEAKER_00: And the other thing that we're talking about is,
56:32
SPEAKER_00: again, it's a very simple model for the application program.
56:34
SPEAKER_00: They don't have to worry about weird kind of defects occurring in their program.
56:38
SPEAKER_00: And some of the ones that can occur, it's like the classic description is of a bank, right?
56:42
SPEAKER_00: Where I might want to read,
56:45
SPEAKER_00: you know,
56:45
SPEAKER_00: like have an operation that reads and says,
56:47
SPEAKER_00: like, do I have $100 in my bank account to transfer somewhere else?
56:50
SPEAKER_00: And you could like arrange for lesser isolation levels that you might be able
56:54
SPEAKER_00: to subtract that $100 twice.
56:57
SPEAKER_00: And that's bad, right?
56:58
SPEAKER_00: You know,
56:58
SPEAKER_00: we want to keep accurate track of your bank account or,
57:02
SPEAKER_00: you know,
57:02
SPEAKER_00: what's in your shopping cart or,
57:04
SPEAKER_00: you know, it's kind of like the use cases are endless there.
57:07
SPEAKER_00: And you do this wrong and you have very egregious bugs.
57:10
SPEAKER_01: But now going back to weak consistency and strong consistency.
57:14
SPEAKER_00: Anything less than linearizability or serializability might be considered kind
57:18
SPEAKER_00: of weak consistency,
57:18
SPEAKER_00: but there's also like,
57:20
SPEAKER_00: you know, kind of strong consistency and eventual consistency.
57:22
SPEAKER_00: So the eventual consistency is like sometimes I can do an operation
57:26
SPEAKER_00: and it might not be immediately, I might not be able to see all the updates.
57:30
SPEAKER_01: The read results will not necessarily give me the current update.
57:32
SPEAKER_00: But they'll come back,
57:33
SPEAKER_00: you know,
57:33
SPEAKER_00: at some point,
57:34
SPEAKER_00: you know, I've written part of it and the rest of it will show up at some point.
57:37
SPEAKER_01: And oftentimes Easiest thing is your credit card balance, right?
57:41
SPEAKER_01: Yeah, yeah.
57:43
SPEAKER_00: And it's faster to do it that way.
57:45
SPEAKER_00: Faster in terms of just what the performance you can get out of the system.
57:47
SPEAKER_00: But again, it's a little bit harder for the application to deal with.
57:50
SPEAKER_00: One of the places this often comes up in distributed databases or databases
57:53
SPEAKER_00: that have any sort of replication is that I could write to the primary replica and
57:57
SPEAKER_00: then I read from the secondary and it's not there yet.
57:59
SPEAKER_00: That's eventual consistency.
58:01
SPEAKER_00: Yeah, that's eventual consistency.
58:02
SPEAKER_00: And you can often work around this,
58:04
SPEAKER_00: but just like puts bigger burden on the application developer because you have
58:08
SPEAKER_00: to pay attention to that.
58:09
SPEAKER_01: And then with CockroachDB,
58:11
SPEAKER_01: you have strong consistency,
58:13
SPEAKER_01: meaning as soon as you're writing it,
58:15
SPEAKER_01: when you're reading from the database, you already get the written value back.
58:18
SPEAKER_00: You get the written value back.
58:19
SPEAKER_00: You know, it's like you read whatever you wrote.
58:22
SPEAKER_00: It doesn't matter if you're reading from the same node you wrote it to.
58:25
SPEAKER_00: If you read it from another node, you still actually get the data you just wrote.
58:28
SPEAKER_01: Is the trade-off logically not that you would have higher latency?
58:32
SPEAKER_01: Because clearly to implement strong consistency,
58:35
SPEAKER_01: you would somehow need to,
58:36
SPEAKER_01: in the naive approach, you would need to write all replicas, right?
58:39
SPEAKER_01: Yeah.
58:40
SPEAKER_01: What are you doing inside, CockroachDB?
58:42
SPEAKER_01: Yeah, yeah.
58:42
SPEAKER_00: Well, we are writing to all the replicas, so.
58:44
SPEAKER_00: Well, yeah.
58:45
SPEAKER_00: Yeah, yeah.
58:46
SPEAKER_00: You're doing
58:46
SPEAKER_01: it fast.
58:47
SPEAKER_00: You're just doing it fast.
58:47
SPEAKER_00: You're making that efficient.
58:48
SPEAKER_00: I think this is one of the things that kind of also fascinates me about
58:51
SPEAKER_00: the software industry is we keep on finding ways to be more and more sophisticated
58:54
SPEAKER_00: in order to provide,
58:55
SPEAKER_00: you know,
58:55
SPEAKER_00: like do things that make it easier to write the applications,
58:58
SPEAKER_00: but do it in a very high performance way.
59:00
SPEAKER_00: And we've gotten, you know, Very, very good at this over the years.
59:03
SPEAKER_00: And this is the area I know about the databases.
59:05
SPEAKER_00: This is happening everywhere.
59:07
SPEAKER_00: Like, I'm just fascinated by how fast graphics have gotten.
59:09
SPEAKER_00: Where when I entered the industry,
59:11
SPEAKER_00: like, you were literally, like, writing out each individual pixel.
59:14
SPEAKER_00: And now you have these GPUs that are doing,
59:16
SPEAKER_00: like, billions of triangles per second, whatever the current numbers are.
59:19
SPEAKER_00: And just, like,
59:19
SPEAKER_00: kind of astounding, like,
59:20
SPEAKER_00: how much sophistication has gotten into every area of computer science,
59:24
SPEAKER_00: wherever you look at it.
59:25
SPEAKER_01: One more thing on CockroachDB I want to ask about is the RAF consensus.
59:28
SPEAKER_01: What is the RAF consensus?
59:29
SPEAKER_00: Yeah.
59:30
SPEAKER_00: Yeah.
59:30
SPEAKER_00: I mean, consensus protocols,
59:32
SPEAKER_00: the original consensus protocol is called Paxos,
59:34
SPEAKER_00: and it was famously hard to implement.
59:36
SPEAKER_00: Raft was, you might think of it as a variant of Paxos.
59:39
SPEAKER_00: It was kind of like an alternative to Paxos,
59:41
SPEAKER_00: but in my mind today...
59:43
SPEAKER_01: And the consensus algorithm being that you have like a number of nodes,
59:46
SPEAKER_01: like three to a lot more, and then...
59:49
SPEAKER_01: How do you get them to agree on?
59:51
SPEAKER_01: What do you typically agree on?
59:52
SPEAKER_01: Yeah, yeah.
59:52
SPEAKER_00: So you agree that the write occurred.
59:54
SPEAKER_00: So like the way to think about consensus.
59:56
SPEAKER_00: So you might think I want to replicate data and I write it to a primary and write it
1:00:01
SPEAKER_00: to a secondary.
1:00:02
SPEAKER_01: And you can't actually have consensus
1:00:04
SPEAKER_00: when you only have two replicas.
1:00:05
SPEAKER_00: And the reason you can't have consensus is if there's a crash and I come up,
1:00:09
SPEAKER_00: like if I'm on the secondary, how do I know if something was written to the primary?
1:00:12
SPEAKER_00: If I'm on the primary, how do I know it was written on the secondary?
1:00:15
SPEAKER_00: Right.
1:00:15
SPEAKER_00: And you're always going to be in this kind of confusing place where it's like,
1:00:18
SPEAKER_00: you either have to roll back a little bit or like, you know, you lose some data.
1:00:22
SPEAKER_00: And Consensus requires at least three,
1:00:24
SPEAKER_00: but you can have consensus across more than three replicas.
1:00:27
SPEAKER_00: And the idea with consensus is I'm going to write to three places.
1:00:30
SPEAKER_00: And normally you don't actually,
1:00:32
SPEAKER_00: when you're doing a read, you don't read from a multiple of them.
1:00:35
SPEAKER_00: But if there's a crash, I have to do recovery.
1:00:37
SPEAKER_00: Then I'm reading O.
1:00:38
SPEAKER_00: I can read from any two of the three,
1:00:41
SPEAKER_00: and I know I can kind of determine what had happened previously.
1:00:44
SPEAKER_00: And it's usually just on the recovery time that you're actually doing that consensus read.
1:00:48
SPEAKER_00: So reads are typically just happening from one replica.
1:00:51
SPEAKER_00: You have to write to all three,
1:00:52
SPEAKER_00: and on a crash during that kind of failure is when the consensus read occurs.
1:00:56
SPEAKER_01: And inside CockroachDB, how many replicas do you choose for either consensus?
1:01:01
SPEAKER_00: It's typically three.
1:01:02
SPEAKER_00: For some system tables, it can be five, and then customers also have control of this.
1:01:06
SPEAKER_00: At the database level, you can write to five, seven.
1:01:08
SPEAKER_00: Five is like, you know, if you're really concerned, you know, about...
1:01:12
SPEAKER_00: The data durability, you might use five, but there's a slowdown.
1:01:15
SPEAKER_00: The more you're running to, it's like the more storage space.
1:01:18
SPEAKER_01: And obviously,
1:01:19
SPEAKER_01: we're talking slowdown with nodes,
1:01:21
SPEAKER_01: but of course,
1:01:22
SPEAKER_01: if they're between regions,
1:01:24
SPEAKER_01: now you have a lot more resilience for,
1:01:26
SPEAKER_01: let's say,
1:01:26
SPEAKER_01: an earthquake or power hours or whatever,
1:01:28
SPEAKER_01: but now you will have additional latency, just speed of light basics, right?
1:01:31
SPEAKER_00: Speed of light latency, right?
1:01:33
SPEAKER_00: And it's tens...
1:01:35
SPEAKER_00: You know, or up to hundreds of milliseconds or even higher if you're going across the globe.
1:01:39
SPEAKER_00: So, you know, you have to be very careful with that in terms of how you architect your queries.
1:01:43
SPEAKER_00: And one of the things that is just a general truism
1:01:45
SPEAKER_00: of distributed databases is you don't want to have a lot of back and forth.
1:01:49
SPEAKER_00: You want to kind of like do all your reads in one kind of parallel read set,
1:01:52
SPEAKER_00: get them back, then do your writes, right?
1:01:54
SPEAKER_00: But if you're doing like kind of serial operations where I read a row,
1:01:57
SPEAKER_00: I write a row, I read a row, I write a row, I mean, the latencies just add up.
1:02:01
SPEAKER_01: We talked about founding CockroachDB, but how has the company grown and where are you today?
1:02:05
SPEAKER_00: Yeah, yeah.
1:02:06
SPEAKER_00: I mean, we're being used in, like, we're powering mission-critical applications.
1:02:09
SPEAKER_00: This is our bread and butter.
1:02:10
SPEAKER_01: But by the way, can you elaborate on mission-critical?
1:02:12
SPEAKER_01: Because it's like,
1:02:14
SPEAKER_01: if you're not,
1:02:14
SPEAKER_01: you are in the industry where you know what this means,
1:02:16
SPEAKER_01: but from the outside, it can feel hard to put a
1:02:19
SPEAKER_00: thumb on what is mission-critical?
1:02:22
SPEAKER_01: Is my SaaS that is, like, showing as mission-critical?
1:02:24
SPEAKER_01: Probably not.
1:02:25
SPEAKER_00: Yeah.
1:02:25
SPEAKER_00: Uh, so mission critical to my mind is like these,
1:02:27
SPEAKER_00: uh, kind of the other term of art is tier zero applications.
1:02:31
SPEAKER_00: The ones that are like just the,
1:02:32
SPEAKER_00: the core crown jewels of what's running a company,
1:02:34
SPEAKER_00: you know, like a trading system, you know, your trading system can't go down.
1:02:38
SPEAKER_00: If the trading system goes down, this is kind of a critical problem.
1:02:41
SPEAKER_00: Or the firm that is running the training system, you know, banking systems.
1:02:45
SPEAKER_00: But also,
1:02:46
SPEAKER_00: you know,
1:02:46
SPEAKER_00: like we work with DoorDash,
1:02:48
SPEAKER_00: you know,
1:02:48
SPEAKER_00: like some people might think that delivering your burrito is a kind of
1:02:52
SPEAKER_00: a mission critical system.
1:02:53
SPEAKER_00: And it certainly is for DoorDash, right?
1:02:55
SPEAKER_00: You know, if that goes down, it's problematic.
1:02:57
SPEAKER_00: We power shopping carts,
1:02:59
SPEAKER_00: you know,
1:03:00
SPEAKER_00: other stuff like this where it's like, Well,
1:03:01
SPEAKER_00: if the shopping cart goes down,
1:03:02
SPEAKER_00: you know,
1:03:02
SPEAKER_00: that company is losing,
1:03:03
SPEAKER_00: you know, hundreds of thousands, millions of dollars per hour.
1:03:07
SPEAKER_00: So that's kind of the criticality you think about.
1:03:08
SPEAKER_01: Yeah, I guess, of course, that they're losing.
1:03:11
SPEAKER_01: But this is like when, yeah,
1:03:12
SPEAKER_01: their customers are also like, they're used to this just working like running water.
1:03:16
SPEAKER_01: And then when it's not the same thing as when your utility breaks, right?
1:03:19
SPEAKER_01: Your water or electricity is out.
1:03:21
SPEAKER_01: You'll survive, but it's not what you expected.
1:03:23
SPEAKER_00: Yeah, yeah, yeah.
1:03:24
SPEAKER_00: And everybody's like, what age are we living in?
1:03:27
SPEAKER_00: The electricity goes out.
1:03:28
SPEAKER_00: We were kind of had this ingrained into our heads at Google.
1:03:31
SPEAKER_00: It's like Gmail cannot go down.
1:03:32
SPEAKER_00: People are relying
1:03:34
SPEAKER_01: on it.
1:03:34
SPEAKER_00: Search cannot go down.
1:03:36
SPEAKER_00: If it goes down too long, people are going to move to other systems.
1:03:39
SPEAKER_00: And it's not like in some ways search isn't as mission critical,
1:03:42
SPEAKER_00: except Oh, wait, every single search is ad dollars behind it.
1:03:45
SPEAKER_00: And you can actually notice the blip in the revenue.
1:03:47
SPEAKER_00: And it's not just the blip in the revenue, it's the blip in reputation as well.
1:03:50
SPEAKER_00: I mean, I think that's the one that really poisons companies is like,
1:03:54
SPEAKER_00: if your bank is down for a serious amount of time,
1:03:56
SPEAKER_00: the reputational damage there will be horrifically bad.
1:03:59
SPEAKER_00: And, you know,
1:03:59
SPEAKER_00: like we often talk about like,
1:04:01
SPEAKER_00: oh, and then grandma won't be able to pay her rent and she'll get evicted.
1:04:03
SPEAKER_00: You have to take this like responsibility really, really seriously.
1:04:05
SPEAKER_01: Yeah.
1:04:06
SPEAKER_01: No, but also, like,
1:04:06
SPEAKER_01: just Gmail being mission critical,
1:04:08
SPEAKER_01: just on the way here,
1:04:09
SPEAKER_01: we only exchanged numbers later,
1:04:11
SPEAKER_01: but we were communicating over email, like, oh,
1:04:13
SPEAKER_01: I was telling you that,
1:04:15
SPEAKER_01: you were telling me that you're here,
1:04:16
SPEAKER_01: I emailed,
1:04:17
SPEAKER_01: and I never for a second thought that it could go down,
1:04:20
SPEAKER_01: and I think we were, like, responding within 30 seconds, right?
1:04:22
SPEAKER_01: And it's just,
1:04:23
SPEAKER_01: I just know it's there,
1:04:24
SPEAKER_01: like, I didn't, like, bother setting up a secondary communication line, so...
1:04:29
SPEAKER_00: Yeah, yeah.
1:04:30
SPEAKER_00: It's like when people just like,
1:04:31
SPEAKER_00: when you have that kind of level of trust with your users,
1:04:34
SPEAKER_00: you got to maintain it and invest in it.
1:04:36
SPEAKER_00: But then it leads to this kind of freedom for the user as well,
1:04:38
SPEAKER_00: where you just don't think about like, I don't have to worry about it.
1:04:41
SPEAKER_00: It's just going to work.
1:04:42
SPEAKER_01: And then in terms of the company, like how many engineers do you have roughly?
1:04:46
SPEAKER_00: We have some hundreds,
1:04:48
SPEAKER_00: I don't actually know the precise engineering number,
1:04:50
SPEAKER_00: 110, but there might be 150 in R&D overall.
1:04:54
SPEAKER_00: There's other folks besides just engineers,
1:04:56
SPEAKER_00: I'm clear as engineering managers, those are engineers as well.
1:04:59
SPEAKER_00: And then, you know, like we've been growing steadily.
1:05:01
SPEAKER_00: It takes quite a while to build a distributed database.
1:05:03
SPEAKER_00: Not for the faint of heart.
1:05:05
SPEAKER_00: You've done that a couple of times.
1:05:07
SPEAKER_00: Yeah.
1:05:08
SPEAKER_00: Well, I did a distributed storage system.
1:05:10
SPEAKER_00: I did a distributed database.
1:05:11
SPEAKER_00: It's not for the faint of heart, right?
1:05:12
SPEAKER_00: So there's a lot of work getting it to a level of stability,
1:05:15
SPEAKER_00: then a level of kind of quality beyond that level of stability,
1:05:19
SPEAKER_00: getting all the bugs out,
1:05:20
SPEAKER_00: and then continue to innovate and put more performance into the system,
1:05:24
SPEAKER_00: adding functionality to integrate better within enterprises.
1:05:27
SPEAKER_00: Our revenue has been kind of steadily growing over the years and You know,
1:05:31
SPEAKER_00: it's at this place now where we see a path to future success as well.
1:05:36
SPEAKER_01: And I want to ask about your coding habits.
1:05:38
SPEAKER_01: So when you co-founded the company, how much code did you write for the first few years?
1:05:44
SPEAKER_00: I wrote a lot.
1:05:45
SPEAKER_00: So I've always been a very prolific coder.
1:05:48
SPEAKER_00: I wrote a lot of code early days.
1:05:50
SPEAKER_00: And early days, I mean,
1:05:51
SPEAKER_00: I was kind of,
1:05:53
SPEAKER_00: we were all technical co-founders,
1:05:55
SPEAKER_00: Ben Spencer and I, and we were already a lot of code, and I was no exception.
1:05:59
SPEAKER_00: But, you know,
1:06:00
SPEAKER_00: if like I look back at my GitHub output,
1:06:02
SPEAKER_00: you know,
1:06:02
SPEAKER_00: it's like kind of peak years,
1:06:04
SPEAKER_00: maybe a hundred thousand lines of code in a year,
1:06:07
SPEAKER_01: which is a lot.
1:06:08
Yeah.
1:06:09
SPEAKER_00: Yeah.
1:06:09
SPEAKER_00: No.
1:06:09
SPEAKER_00: So, I mean, we're talking pre-AI.
1:06:12
SPEAKER_00: Pre-AI, right?
1:06:13
SPEAKER_00: This is when you're back doing this manually, right?
1:06:15
SPEAKER_00: You know,
1:06:16
SPEAKER_00: at some point,
1:06:16
SPEAKER_00: you know, we started out using a system called RocksDB, which is an LSM.
1:06:20
SPEAKER_00: At some point, I think it's back in 2019, we ran into limitations with it.
1:06:25
SPEAKER_00: I decided we wanted to rewrite it and did a big push to, you know, write it.
1:06:29
SPEAKER_00: That might have been like, 40, 50,000 lines of code.
1:06:32
SPEAKER_00: And then a bunch of other people have come up and helped.
1:06:34
SPEAKER_00: And like, you kind of look at that output.
1:06:35
SPEAKER_00: I'm just like, oh my goodness, that was a lot to keep in your head.
1:06:39
SPEAKER_00: It's a lot just to type, you know, 100,000 lines of code.
1:06:42
SPEAKER_00: The average kind of like that the industry talks about is 3,000 lines of code from
1:06:46
SPEAKER_00: an engineer in a month.
1:06:47
SPEAKER_00: And so if you multiply that out, maybe 36,000 in a year, that's good, right?
1:06:52
SPEAKER_00: So I was doing a lot.
1:06:54
SPEAKER_00: I kind of look at that.
1:06:55
SPEAKER_00: It's like, there's kind of a max that you can hold in your head at a time.
1:06:58
SPEAKER_00: The tools have gotten a lot better since I first entered the industry.
1:07:01
SPEAKER_00: We've gotten better debugging techniques,
1:07:03
SPEAKER_00: better testing techniques, but still quite significant.
1:07:06
SPEAKER_01: And you were CTO from the beginning, co-founder of CTO.
1:07:10
SPEAKER_01: But there was a time,
1:07:11
SPEAKER_01: sometime around like 2022,
1:07:12
SPEAKER_01: when you decided to kind of be a bit more hands-off, right?
1:07:16
SPEAKER_00: Yeah, yeah.
1:07:17
SPEAKER_01: Can you tell me about that?
1:07:18
SPEAKER_00: I mean, the general rule of thumb for engineering leaders is,
1:07:22
SPEAKER_00: well, you got to have your team, you got to manage your team.
1:07:24
SPEAKER_00: And we had a VP of engineering.
1:07:26
SPEAKER_00: Um, but I was kind of getting to the point of like, okay, is my, are my coding days done?
1:07:31
SPEAKER_00: You know, like kind of just direct from a higher level.
1:07:33
SPEAKER_00: And, you know,
1:07:34
SPEAKER_00: I got this advice for a long period of time and I pushed back on it,
1:07:37
SPEAKER_00: but you know,
1:07:37
SPEAKER_00: I kind of acquiesced at some point and I think it was the right advice.
1:07:40
SPEAKER_00: I'm not saying that the advice was wrong at the time.
1:07:42
SPEAKER_00: Um, but there was a time period from about 2022 to 2024, whereas like my, my output declined.
1:07:49
SPEAKER_00: I think I actually did the Swiss tables thing in that time period,
1:07:51
SPEAKER_00: but I wasn't doing much on the, the
1:07:53
SPEAKER_01: core, the business.
1:07:54
SPEAKER_00: Yeah, the core business.
1:07:56
SPEAKER_00: You know,
1:07:56
SPEAKER_00: I would get in there and do some work,
1:07:58
SPEAKER_00: but it's really hard that if you're in meetings all day to also do coding.
1:08:02
SPEAKER_00: I think this is the fundamental tension.
1:08:04
SPEAKER_01: So you kind of took on the meeting burden,
1:08:06
SPEAKER_01: the coordination burden,
1:08:09
SPEAKER_01: the stuff that was,
1:08:10
SPEAKER_01: if I'm reading correctly,
1:08:11
SPEAKER_01: before you spent a lot of your head in the code and now you're spending a lot
1:08:14
SPEAKER_01: of your head above the code,
1:08:15
SPEAKER_01: the business, the engineering, the org, the whatever, customers, that kind of stuff.
1:08:20
SPEAKER_00: The customers, and just being an executive as well.
1:08:22
SPEAKER_00: So very hard to wear all those hats simultaneously.
1:08:25
SPEAKER_00: And then I got back into it because AI started to emerge.
1:08:29
SPEAKER_01: So when did you start using AI?
1:08:31
SPEAKER_01: When did you start to find it useful in terms of coding?
1:08:34
SPEAKER_00: Yeah, it was interesting because those initial versions of glorified autocomplete came out.
1:08:40
SPEAKER_01: We're talking about the GitHub Copilot, the cursor.
1:08:43
SPEAKER_00: Yeah, GitHub Copilot.
1:08:44
SPEAKER_00: That was the one I had the first exposure to.
1:08:46
SPEAKER_00: We dabbled with cursor at the time, but they're all like kind of glorified autocomplete.
1:08:50
SPEAKER_00: And it was kind of crazy that you could just start typing something,
1:08:52
SPEAKER_00: you know, like film the rest of the function.
1:08:54
SPEAKER_00: You look at it, and I'm like, wait, you kind of got that right.
1:08:57
SPEAKER_00: This is crazy, right?
1:08:58
SPEAKER_00: And, you know, we're trying to encourage our engineers to use this.
1:09:01
SPEAKER_00: And at some point,
1:09:02
SPEAKER_00: you know,
1:09:03
SPEAKER_00: I can't remember if this is my idea or my co-founders or someone basically like,
1:09:07
SPEAKER_00: you know, like,
1:09:08
SPEAKER_00: In order to like really guide people about how to use it,
1:09:11
SPEAKER_00: you have to be a user yourself.
1:09:13
SPEAKER_00: You know, I think this is true of like engineering management in general.
1:09:16
SPEAKER_00: Like you want to like manage engineers, you have to know how to be an engineer.
1:09:19
SPEAKER_00: Like if you,
1:09:19
SPEAKER_00: if you don't know how to be a good engineer,
1:09:21
SPEAKER_00: it's like really hard to manage other engineers.
1:09:23
SPEAKER_01: I feel like you'll have a hard time like just relating to them at the very least.
1:09:27
SPEAKER_00: Exactly, exactly.
1:09:28
SPEAKER_00: So kind of took it on me like, no, I mean,
1:09:30
SPEAKER_00: this is like,
1:09:31
SPEAKER_00: it was clear very early on,
1:09:33
SPEAKER_00: like this is probably going to go somewhere,
1:09:35
SPEAKER_00: but it wasn't quite clear how far, how fast it would go.
1:09:38
SPEAKER_00: And you start dabbling this and I was like, oh, okay, well, it's not quite good enough.
1:09:43
SPEAKER_00: It's not quite good enough, but you know, let me start getting back into the coding.
1:09:45
SPEAKER_00: Very rapidly,
1:09:46
SPEAKER_00: you know,
1:09:47
SPEAKER_00: you started seeing the signs of life,
1:09:48
SPEAKER_00: like, you know, kind of the Opus models coming out.
1:09:51
SPEAKER_00: It was Sonnet first, then the Opus.
1:09:53
SPEAKER_00: And you're like, you're looking at these and you're like, oh, wow, okay.
1:09:56
SPEAKER_00: Well, they seem to be able to do quite a lot,
1:09:58
SPEAKER_00: but you know,
1:09:59
SPEAKER_00: the code is still not great,
1:10:01
SPEAKER_00: but then it was just like the, just this cadence of continuing improvements.
1:10:05
SPEAKER_00: And, you know,
1:10:07
SPEAKER_00: I was starting to do a lot with Sonnet and then starting to use Opus and,
1:10:11
SPEAKER_00: you know, I had that same movement everybody else did.
1:10:14
SPEAKER_00: And this was last year, last, uh, November.
1:10:16
SPEAKER_01: That went November, December winter break, right?
1:10:19
SPEAKER_01: Yeah.
1:10:19
SPEAKER_00: It was Thanksgiving.
1:10:20
SPEAKER_00: I distinctly remember it cause you know,
1:10:23
SPEAKER_01: You were not chilling, you were coding, weren't you?
1:10:25
SPEAKER_01: You were agenting.
1:10:26
SPEAKER_00: I was agenting just like everyone else.
1:10:29
SPEAKER_00: I had this thing I'd wanted to do for a long period of time on CockroachDB,
1:10:34
SPEAKER_00: which is I wanted to test all these configurations of CockroachDB
1:10:38
SPEAKER_00: across different vertical scaling,
1:10:40
SPEAKER_00: like how many CPUs you have on a node,
1:10:41
SPEAKER_00: how many disks you have on a node,
1:10:44
SPEAKER_00: how many nodes you have in the system, and test across all this huge matrix of it.
1:10:47
SPEAKER_00: This is one of these things they couldn't ever quite Get prioritized appropriately
1:10:51
SPEAKER_00: because it never seemed like kind of critical,
1:10:52
SPEAKER_00: but I've like, I always had this intuition.
1:10:54
SPEAKER_00: There was something there.
1:10:55
SPEAKER_00: And then over this like four day span,
1:10:57
SPEAKER_00: the code just like materialized as I was using,
1:11:00
SPEAKER_00: I think that was Opus four, seven or four or five, whatever the number was.
1:11:03
SPEAKER_00: Like the recollection I had is, you know, like a pretty fast typer.
1:11:08
SPEAKER_00: And then I just remember having this feeling of like,
1:11:11
SPEAKER_00: Well, the code is just like materializing before my eyes.
1:11:14
SPEAKER_00: You know, you would ask for these things.
1:11:15
SPEAKER_00: I got out of the habit of actually typing and you just kind of like ask
1:11:19
SPEAKER_00: for something materializes.
1:11:20
SPEAKER_00: If you've ever seen like some of those,
1:11:22
SPEAKER_00: you know,
1:11:23
SPEAKER_00: a movie where they had this kind of archetype of a software engineer gets in front
1:11:27
SPEAKER_00: of the keyboard and they start typing, show the screen.
1:11:29
SPEAKER_00: It's just like, it's going way beyond human speed.
1:11:32
SPEAKER_00: And then it was, it was Good on that, right?
1:11:34
SPEAKER_01: I think as software engineers, we used to laugh at these.
1:11:37
SPEAKER_01: I still remember swordfish,
1:11:39
SPEAKER_01: the thing when they're visualizing the things that are going or the code appearing.
1:11:44
SPEAKER_01: You're seeing that the person's typing and then it's a big line.
1:11:48
SPEAKER_01: And as software engineers, we're laughing because it's not how it is.
1:11:51
SPEAKER_01: But it's crazy that effect, right?
1:11:54
SPEAKER_01: Yeah, yeah.
1:11:54
SPEAKER_00: But now it is.
1:11:56
SPEAKER_00: It's even better than that effect, right?
1:11:57
SPEAKER_01: Because it actually It works this time.
1:11:59
SPEAKER_00: And it's that slow in comparison.
1:12:00
SPEAKER_00: It materializes faster than that.
1:12:03
SPEAKER_00: Like you can literally go, like we've been talking about bee trees a whole bunch.
1:12:06
SPEAKER_00: I implemented another bee tree in the past month.
1:12:09
SPEAKER_01: Of course you did.
1:12:09
SPEAKER_00: Yeah.
1:12:10
SPEAKER_00: And it took about 30 minutes to implement probably 10,000 lines of highly optimized Rust.
1:12:15
SPEAKER_00: I mean, it's just like it boggles the mind.
1:12:17
SPEAKER_00: I mean, we should look up later on 10,000 lines.
1:12:20
SPEAKER_00: It's like a crazy amount.
1:12:22
SPEAKER_00: You physically cannot type that fast.
1:12:24
SPEAKER_01: You started getting back to coding.
1:12:25
SPEAKER_01: Like,
1:12:25
SPEAKER_01: are we talking about kind of vibe coding prototyping or are we actually talking,
1:12:29
SPEAKER_01: you started to contribute like proper production ready code that is up the level
1:12:33
SPEAKER_01: of what you're doing.
1:12:35
SPEAKER_00: Well, it started with this tool that was And I was looking
1:12:39
SPEAKER_00: for places like we can have innovation.
1:12:41
SPEAKER_00: And one of the things that we want to innovate in,
1:12:54
SPEAKER_00: Um, was, you know, better, uh, auto scaling of a CockroachDB cluster.
1:12:58
SPEAKER_00: And kind of in the January timeframe,
1:13:00
SPEAKER_00: I came out with like,
1:13:02
SPEAKER_00: what I would think is like kind of a, uh, a research breakthrough, you might say.
1:13:05
SPEAKER_00: And it came about because I was dabbling in this area and just working with
1:13:08
SPEAKER_00: the models, trying to understand it.
1:13:10
SPEAKER_00: And they're not just good at coding.
1:13:11
SPEAKER_00: They're also good at like helping you explore design ideas.
1:13:13
SPEAKER_00: And I think this is kind of the fascinating thing where it's,
1:13:16
SPEAKER_00: you have to get out of the mindset of like, I know exactly what I'm going to build.
1:13:19
SPEAKER_00: But more like, hey, we have this problem, talk through it, be a partner with me.
1:13:24
SPEAKER_01: Be a sparring partner.
1:13:25
SPEAKER_00: Be a sparring partner.
1:13:26
SPEAKER_00: You know,
1:13:26
SPEAKER_00: like there's this advice you might have heard that like,
1:13:28
SPEAKER_00: you know, if you're stuck on a problem, you should go yellow duck it.
1:13:31
SPEAKER_00: Rubber duck.
1:13:32
SPEAKER_00: Rubber duck it.
1:13:33
SPEAKER_00: I think I've heard it as yellow duck, you know.
1:13:35
SPEAKER_00: Just go talk to something.
1:13:37
SPEAKER_00: You don't even need it to respond.
1:13:38
SPEAKER_00: Now you can talk to this system, this intelligence.
1:13:42
SPEAKER_00: And it will give you back stuff.
1:13:43
SPEAKER_00: And it's not always right.
1:13:45
SPEAKER_00: I mean, this is the thing, even today, these models are fantastic.
1:13:48
SPEAKER_00: Fables, fantastic.
1:13:49
SPEAKER_00: Astros, fantastic.
1:13:50
SPEAKER_00: And they're not always right, but they kind of like, they know so much.
1:13:53
SPEAKER_00: It's just encyclopedic.
1:13:55
SPEAKER_00: And you can explore ideas super, super fast.
1:13:59
SPEAKER_00: And then be pointing out, well, that doesn't sound right to me.
1:14:01
SPEAKER_00: And it'll be like, oh yeah, you're absolutely right.
1:14:03
SPEAKER_00: You know, like, I hate that sick fancy.
1:14:06
SPEAKER_00: It's like, it kills me.
1:14:07
SPEAKER_01: Or you're right to push on back on, you're right to push the back on that.
1:14:09
SPEAKER_00: That's a new episode of Yeah, and yet, like, just able to make such fast progress.
1:14:15
SPEAKER_00: And I find it absolutely incredible.
1:14:17
SPEAKER_00: So it quickly moved from just doing kind of side things,
1:14:21
SPEAKER_00: building up some tools and whatnot,
1:14:22
SPEAKER_00: to building,
1:14:24
SPEAKER_00: you know,
1:14:24
SPEAKER_00: essentially over the last eight months,
1:14:26
SPEAKER_00: it's January, we've been building towards a, you know, a new launch.
1:14:31
SPEAKER_00: And, you know,
1:14:31
SPEAKER_00: like a lot of that has been produced,
1:14:34
SPEAKER_00: the gently powered engineers,
1:14:35
SPEAKER_00: and like the entire company's gone on board with this now,
1:14:38
SPEAKER_00: where I think it's like Probably 100% of engineers are using it.
1:14:41
SPEAKER_00: To a greater or lesser degree, a lot of code is being produced.
1:14:44
SPEAKER_00: I think it's very high quality code.
1:14:46
SPEAKER_00: These models will not test things adequately.
1:14:48
SPEAKER_00: They don't look quite intently enough about performance.
1:14:52
SPEAKER_00: But if you're like, you can guide them in the right way.
1:14:54
SPEAKER_00: And I think there's actually a huge advantage to anybody who's done management before.
1:14:58
SPEAKER_00: It's a little bit like being a manager of people where you're a manager of a large group.
1:15:03
SPEAKER_00: You're not looking at every line of code,
1:15:05
SPEAKER_00: but you're definitely kind of helping architect the system.
1:15:07
SPEAKER_00: And I think there's a very...
1:15:08
SPEAKER_00: Strong analogy there.
1:15:09
SPEAKER_01: I sometimes push back on this analogy.
1:15:11
SPEAKER_01: The reason being, I was an engineering manager.
1:15:13
SPEAKER_01: My take is that working with these agents,
1:15:16
SPEAKER_01: it's not like management, because management has so much of the human stuff.
1:15:19
SPEAKER_01: As a manager, when I think of all the stuff I dealt with, which was the people Yeah.
1:15:53
SPEAKER_01: It's a lot closer to that, if you know what I mean.
1:15:55
SPEAKER_00: Yeah, I do agree.
1:15:56
SPEAKER_00: I use the engineering manager shorthand,
1:15:58
SPEAKER_00: but it's really about being a tech lead for like a 30 or 40 person organization,
1:16:02
SPEAKER_00: you know, or being an architect for one.
1:16:04
SPEAKER_00: I think that the term architect gives me a little bit of a distaste,
1:16:08
SPEAKER_00: but if you're an architect, he's also on the ground.
1:16:10
SPEAKER_00: A hands-on architect.
1:16:11
SPEAKER_00: A hands-on architect.
1:16:12
SPEAKER_00: And you don't have any of the management stuff,
1:16:15
SPEAKER_00: which is a blessing and curse,
1:16:16
SPEAKER_00: but it's kind of remarkable that you can spin these things up if they make
1:16:19
SPEAKER_00: a mistake.
1:16:20
SPEAKER_00: You can keep on correcting them until they get it right.
1:16:22
SPEAKER_00: We've always been able to do that on the human side, and I can do it faster.
1:16:26
SPEAKER_00: I think the ultimate result for me is you just have to be more ambitious
1:16:30
SPEAKER_00: about everything you do.
1:16:31
SPEAKER_00: You can produce more, higher performance, higher quality, more secure.
1:16:36
SPEAKER_00: So our ambitions have to raise up.
1:16:38
SPEAKER_01: You also mentioned that with AI,
1:16:40
SPEAKER_01: like you're ever using and you're building stuff,
1:16:42
SPEAKER_01: but also with CockroachDB, you're now building something that is also related to AI.
1:16:47
SPEAKER_01: Can you talk about that?
1:16:48
SPEAKER_00: Yeah, yeah.
1:16:49
SPEAKER_00: We're building multiple things.
1:16:50
SPEAKER_00: I mean, AI is the future.
1:16:53
SPEAKER_01: Well, it's here to say.
1:16:53
SPEAKER_01: I think that's for sure.
1:16:54
SPEAKER_01: It's here to say.
1:16:55
SPEAKER_00: Yeah.
1:16:55
SPEAKER_00: I mean, like, you know, one of our theses, which is not crazy.
1:17:00
SPEAKER_00: Every application in the future is going to be written by AI.
1:17:02
SPEAKER_00: You know, I think there will be some kind of bespoke software, you know, handcrafted software.
1:17:07
SPEAKER_00: We will see that continue to exist just as people write assembly still,
1:17:11
SPEAKER_00: you know, but it's going to be diminishing in size.
1:17:14
SPEAKER_00: So, you know, asymptotically approaching 100% of software will be written by AI.
1:17:18
SPEAKER_01: Generated, right?
1:17:19
SPEAKER_00: Yeah.
1:17:20
SPEAKER_00: It's hard to say if like when it's going to end where the humans are the agent,
1:17:23
SPEAKER_00: you know, kind of like supplying the agency and the vision behind it.
1:17:25
SPEAKER_00: I think that might exist for many, many years.
1:17:28
SPEAKER_00: But I think the code will fundamentally be written by the AI.
1:17:32
SPEAKER_00: And I think we're going to just see this explosion of applications.
1:17:36
SPEAKER_00: And we're seeing that inside Cockroach Labs.
1:17:38
SPEAKER_00: We're seeing it elsewhere.
1:17:39
SPEAKER_00: Earlier this year,
1:17:40
SPEAKER_00: we kind of rolled out this internal platform where non-engineers could write kind
1:17:45
SPEAKER_00: of mini applications.
1:17:46
SPEAKER_00: I mean, this is like you're hearing this at other companies.
1:17:48
SPEAKER_00: We did the same thing.
1:17:49
SPEAKER_00: And over the course of just a couple months,
1:17:51
SPEAKER_00: you know, 500 applications, a thousand applications.
1:17:55
SPEAKER_01: By non-engineers.
1:17:56
SPEAKER_00: By non-engineers.
1:17:57
SPEAKER_00: Primarily by non-engineers.
1:17:58
SPEAKER_00: And I thought it was awesome.
1:17:59
SPEAKER_00: Like our HR team is building like these little applications.
1:18:02
SPEAKER_00: Like this is the thing I've always dreamed about doing.
1:18:04
SPEAKER_00: And like they were never serviced.
1:18:06
SPEAKER_00: Like your CFOs all over the place and ours is no exception.
1:18:09
SPEAKER_00: Producing dashboards that they can never create it before.
1:18:12
SPEAKER_00: And I think it's very empowering.
1:18:13
SPEAKER_00: I'm married.
1:18:14
SPEAKER_00: I have a wife.
1:18:14
SPEAKER_00: She needs software.
1:18:15
SPEAKER_00: She cannot produce that software on her own.
1:18:17
SPEAKER_00: And I've never actually helped her produce the software, which is my own failing.
1:18:20
SPEAKER_00: But I think there's a world in the future
1:18:23
SPEAKER_00: where everyone's getting custom software built for them.
1:18:26
SPEAKER_00: And you're just going to also see greater and greater applications
1:18:28
SPEAKER_00: and higher quality systems being produced by every company as well.
1:18:33
SPEAKER_01: Today, what is your stack?
1:18:35
SPEAKER_01: What do you work with in terms of harness, model, how you run agents?
1:18:39
SPEAKER_01: Is it one agent?
1:18:40
SPEAKER_01: Is it multiple?
1:18:41
SPEAKER_01: What kind of terminal do you use?
1:18:43
SPEAKER_00: Yeah, yeah.
1:18:44
SPEAKER_00: It's evolved over time.
1:18:45
SPEAKER_00: So, you know,
1:18:46
SPEAKER_00: like when I first started dabbling in the AI stuff again,
1:18:48
SPEAKER_00: it was, you know, GitHub Copilot.
1:18:49
SPEAKER_00: I was an Emacs user for like 20-something years.
1:18:52
SPEAKER_00: I got convinced to move to VS Code.
1:18:54
SPEAKER_00: That's all gone now.
1:18:56
SPEAKER_00: At some point, I moved to using Cloud Code.
1:18:58
SPEAKER_00: That was the thing that, whatever reason, I just got started using it.
1:19:01
SPEAKER_00: It was Cloud Code in the terminal.
1:19:02
SPEAKER_00: Of late, I used Mixture.
1:19:04
SPEAKER_00: So I'll just grab my current setup.
1:19:06
SPEAKER_00: I use the Cloud Desktop app, Cloud Code via the Cloud Desktop app.
1:19:10
SPEAKER_00: It's fantastic.
1:19:11
SPEAKER_00: Good job, Anthropic.
1:19:12
SPEAKER_00: I also sometimes use the Codex desktop app just so I have an alternative model
1:19:17
SPEAKER_00: to turn to for some very critical things we're working on.
1:19:20
SPEAKER_00: I will get, you know, one of those models, sometimes like the best model.
1:19:24
SPEAKER_00: Oftentimes I'm using the best model like Fable.
1:19:26
SPEAKER_00: You know, I've recently started using that.
1:19:27
SPEAKER_00: Sometimes it's Astra, sometimes it's the other ones.
1:19:30
SPEAKER_00: But you like,
1:19:30
SPEAKER_00: you have one producer design,
1:19:32
SPEAKER_00: you have the other one being like, hey, my colleague produced this.
1:19:34
SPEAKER_00: Can you kind of tear it up?
1:19:35
SPEAKER_00: You know, adversary will review it.
1:19:38
SPEAKER_00: And it's not always perfect, right?
1:19:39
SPEAKER_00: You know, but I think there is utility, especially for something that's super, super critical.
1:19:43
SPEAKER_00: We're pushing towards the launch really soon and kind of rushing towards the finish line.
1:19:46
SPEAKER_00: I'm telling folks on the team,
1:19:47
SPEAKER_00: it's like,
1:19:48
SPEAKER_00: you know,
1:19:48
SPEAKER_00: especially our very senior folks, like you should use the best model right now.
1:19:51
SPEAKER_00: It's worthwhile to do that.
1:19:53
SPEAKER_00: I'm generally just using the best model.
1:19:55
SPEAKER_00: And part of the reason is I don't actually know That I'm getting
1:19:58
SPEAKER_00: a lot more intelligence from it,
1:19:59
SPEAKER_00: but I don't want to have the cognitive overhead of deciding on a case-by-case basis.
1:20:04
SPEAKER_00: Should I use Sonnet?
1:20:05
SPEAKER_00: Should I use Opus?
1:20:06
SPEAKER_00: Should I use Fable?
1:20:07
SPEAKER_00: Should I use Sol or Astra?
1:20:09
SPEAKER_00: And I think,
1:20:09
SPEAKER_00: you know, maybe if you, like, I really needed the speed, I would make that decision.
1:20:13
SPEAKER_00: But oftentimes I'm like, I'm doing things in parallel.
1:20:16
SPEAKER_00: So you're asking how many agents I'm spinning up?
1:20:19
SPEAKER_00: Well, it depends.
1:20:20
SPEAKER_00: Like oftentimes there's like a certain number of sessions you might be using.
1:20:23
SPEAKER_00: I find like kind of cognitive overhead is about five to 10 sessions concurrently,
1:20:29
SPEAKER_00: but sometimes those sessions will have many sub-agents doing things.
1:20:33
SPEAKER_01: Yeah.
1:20:33
SPEAKER_01: And they'll be long running.
1:20:34
SPEAKER_00: Yeah.
1:20:35
SPEAKER_00: It'll depend on what I'm doing.
1:20:36
SPEAKER_00: Yeah.
1:20:37
SPEAKER_00: As I was coming here, I was on the train.
1:20:39
SPEAKER_00: I spun up something to do a little bit of research,
1:20:41
SPEAKER_00: and that was just probably using three sub-agents.
1:20:43
SPEAKER_00: Other times, it might be 20 sub-agents.
1:20:46
SPEAKER_00: Anthropic has dynamic workflows.
1:20:48
SPEAKER_00: I can't remember what the codex thing is called,
1:20:50
SPEAKER_00: but they have various ways of spinning up graphs of agents and prosecuting them.
1:20:55
SPEAKER_00: I mean, I think sometimes I'm probably using 100 sub-agents.
1:20:58
SPEAKER_00: Others, it's only like five.
1:20:59
SPEAKER_00: It depends on what's happening.
1:21:01
SPEAKER_00: And sometimes I'm not doing any of that agentic stuff because I'm often thinking
1:21:03
SPEAKER_00: about something else.
1:21:04
SPEAKER_01: Since you were an Emacs user for 20 years, you know, using the terminal, right?
1:21:09
SPEAKER_01: Well, Emacs, but how come you moved to graphical interface with agents?
1:21:13
SPEAKER_01: A lot of people still use terminal UIs.
1:21:16
SPEAKER_00: Yeah, yeah.
1:21:17
SPEAKER_00: Well, so I moved from Emacs to So I moved to VS Code.
1:21:20
SPEAKER_00: I was just like, a colleague was like, hey, these IDs are really good.
1:21:23
SPEAKER_00: You need to try them out.
1:21:24
SPEAKER_00: I was like, Emacs is an ID.
1:21:25
SPEAKER_00: And then I kind of moved and like, you know, some of the stuff was just more seamless.
1:21:29
SPEAKER_00: And like, Emacs keeps on catching up.
1:21:30
SPEAKER_00: It always feels a little bit, you know, six months to a year behind.
1:21:34
SPEAKER_00: And like every time they have a new version, my setup would break.
1:21:36
SPEAKER_00: And I just got tired of that.
1:21:38
SPEAKER_00: So I moved to VS Code.
1:21:39
SPEAKER_00: But then I was like, I'd have many terminals open in VS Code.
1:21:42
SPEAKER_00: My setup would always be like, Uh,
1:21:44
SPEAKER_00: terminals in VS code tabs,
1:21:46
SPEAKER_00: the terminals and tabs,
1:21:47
SPEAKER_00: you know, have like eight of them open and then they started taking over.
1:21:50
SPEAKER_00: It used to be like just cloud code in the terminal.
1:21:52
SPEAKER_00: Yeah.
1:21:53
SPEAKER_00: Yeah.
1:21:53
SPEAKER_00: And then,
1:21:54
SPEAKER_00: you know,
1:21:54
SPEAKER_00: just some point recently is like six weeks ago,
1:21:56
SPEAKER_00: colleague was like, well, have you tried out the cloud desktop app recently?
1:21:59
SPEAKER_00: It's really good.
1:22:00
SPEAKER_00: I was like, yeah, no, no, I'm very, very happy.
1:22:02
SPEAKER_00: And I made the switch and it was a little bit awkward at first.
1:22:04
SPEAKER_00: And then suddenly I'm like, holy crap, this is awesome.
1:22:07
SPEAKER_01: You can like manage the agents a bit better, easier sessions?
1:22:09
SPEAKER_01: Yeah, I mean,
1:22:09
SPEAKER_01: it's like,
1:22:10
SPEAKER_00: you have the sessions there,
1:22:11
SPEAKER_00: the sessions I'm down the side,
1:22:12
SPEAKER_00: you know,
1:22:13
SPEAKER_00: and it's like,
1:22:13
SPEAKER_00: they're kind of like tabs in some regard and shit,
1:22:16
SPEAKER_00: but the whole integration that Anthropic's been doing and OpenAI is doing
1:22:19
SPEAKER_00: the same thing now.
1:22:20
SPEAKER_00: By the way,
1:22:21
SPEAKER_00: like I'm not dissing Cursor and Factory and Cognition,
1:22:23
SPEAKER_00: they're all pushing the same thing.
1:22:25
SPEAKER_00: It's incredible how fast these systems are innovating right now and evolving.
1:22:31
SPEAKER_01: What do you think good software engineering looks today compared to like,
1:22:35
SPEAKER_01: you know,
1:22:35
SPEAKER_01: four years ago before we had a,
1:22:37
SPEAKER_01: has it, has it changed, has good change or that's not really changed?
1:22:40
SPEAKER_00: Yeah.
1:22:41
SPEAKER_00: I think the ambition has to increase, you know, I think the quality has to be higher.
1:22:47
SPEAKER_01: I'm glad you mentioned quality.
1:22:48
SPEAKER_01: Can we talk about that?
1:22:48
SPEAKER_01: Because I'm seeing across the industry just quality declines,
1:22:53
SPEAKER_01: which you cannot fully put your finger on AI,
1:22:55
SPEAKER_01: but oftentimes it is people pushing up more and more and just not paying attention
1:23:00
SPEAKER_01: to Just small regressions here and there.
1:23:04
SPEAKER_01: Again, you're building a database.
1:23:07
SPEAKER_01: Have you noticed any or have you gotten feedback of any quality regressions?
1:23:11
SPEAKER_01: Or if not, how come?
1:23:13
SPEAKER_01: Because this is just basic.
1:23:16
SPEAKER_01: We talk about the law of physics.
1:23:19
SPEAKER_01: This is an observed thing that when you start to have more output,
1:23:23
SPEAKER_01: you increase your deployment frequency.
1:23:26
SPEAKER_01: You often, not always, but you often have more regressions.
1:23:31
SPEAKER_01: They won't be.
1:23:51
SPEAKER_00: They kind of get a little bit lazy on the testing side and you have to make sure
1:23:56
SPEAKER_00: the tests are comprehensive but also they're using all the testing techniques
1:24:00
SPEAKER_00: and there's a lot of testing techniques out there we have a lot of knowledge
1:24:03
SPEAKER_00: about how to do testing well and you know what the agents are lazy Humans are
1:24:07
SPEAKER_00: a little bit lazy.
1:24:08
SPEAKER_00: So getting the humans to actually be very disciplined about their testing is also challenging.
1:24:13
SPEAKER_00: And I think it's actually easier with agents.
1:24:15
SPEAKER_00: You know, you can kind of instill that.
1:24:16
SPEAKER_00: You can get them set up.
1:24:17
SPEAKER_00: You have to give them the guidance.
1:24:18
SPEAKER_00: Use property-based testing.
1:24:20
SPEAKER_00: Use metamorphic testing.
1:24:21
SPEAKER_00: Use, like, these advanced testing techniques, deterministic simulation testing.
1:24:25
SPEAKER_00: You know, there's, like, technique after technique you can use.
1:24:28
SPEAKER_00: And humans have always, like, I'm lazy myself.
1:24:31
SPEAKER_00: Like, I'm pretty good about being disciplined about testing.
1:24:33
SPEAKER_00: At some point, you're just like, Okay, that was enough.
1:24:36
SPEAKER_00: We got a ship.
1:24:36
SPEAKER_00: And like now you can get a little bit like, you know, a little bit stricter and firmer.
1:24:40
SPEAKER_00: I think the same thing applies over on the security side, the performance side.
1:24:44
SPEAKER_00: I mean, we've always had at Cockroach and in the industry,
1:24:47
SPEAKER_00: we've known security coding practices.
1:24:50
SPEAKER_00: You can literally have every single line of code on every commit reviewed by a security expert.
1:24:54
SPEAKER_01: Doing like an adversarial security review by an agent or multiple agents.
1:25:00
SPEAKER_00: Or multiple agents, right?
1:25:01
SPEAKER_00: We're going through a tough time in the industry right now with like kind of hacks
1:25:05
SPEAKER_00: and security leaks and whatnot.
1:25:07
SPEAKER_00: There's only a limited number of bugs that can be in software.
1:25:09
SPEAKER_00: I think we'll get it added on the security side and on the quality side.
1:25:13
SPEAKER_00: And then also just on the,
1:25:14
SPEAKER_00: when I say quality,
1:25:15
SPEAKER_00: it's not just bugs,
1:25:16
SPEAKER_00: but it's also like, you know, just little things like, oh, that UX element is wrong.
1:25:20
SPEAKER_00: And there's no excuse for that right now.
1:25:22
SPEAKER_00: Like fixing it is so, so easy.
1:25:24
SPEAKER_00: And what we're also starting to see evolve is like, you know, designers using Figma.
1:25:29
SPEAKER_00: Well, I think that should be a thing of the past right now.
1:25:31
SPEAKER_00: Our designer just deals with HTML and CSS and JavaScript directly.
1:25:35
SPEAKER_00: And sometimes is even just producing pull requests, you know, PRs, which she loves.
1:25:40
SPEAKER_00: We all love.
1:25:41
SPEAKER_00: Everybody's happy with this.
1:25:42
SPEAKER_00: There's none of this waterfall.
1:25:44
SPEAKER_01: So she's producing pull requests for the production code base.
1:25:48
SPEAKER_00: Yeah.
1:25:49
SPEAKER_00: Yeah.
1:25:49
SPEAKER_00: And you know, this is on the UX side, not on the core database.
1:25:51
SPEAKER_01: Of course, but this is the area that she owns, right?
1:25:54
SPEAKER_00: Yeah.
1:25:56
SPEAKER_00: It's just, I mean, she loves it.
1:25:57
SPEAKER_00: Everybody loves it.
1:25:58
SPEAKER_00: I mean, there's no downside.
1:25:59
SPEAKER_01: This is not new.
1:26:00
SPEAKER_01: Yeah, for sure.
1:26:02
SPEAKER_01: What about code review?
1:26:04
SPEAKER_01: What's your take on code review?
1:26:05
SPEAKER_01: I think it's starting to get a bit controversial.
1:26:08
SPEAKER_01: Is it going to stay or not?
1:26:09
SPEAKER_01: Because there's been a practice that's been around... You worked at Google.
1:26:13
SPEAKER_01: Google has been so big on code review.
1:26:15
SPEAKER_01: As I understand, there are two code reviews, right?
1:26:18
SPEAKER_01: The domain expert reviews it, and then there's language expert reviews it.
1:26:21
SPEAKER_01: So you went through that.
1:26:23
SPEAKER_01: How did you think of it, and how are you thinking of it now?
1:26:26
SPEAKER_00: I used to be part of the, I was one of the early members of the C++ coding review team.
1:26:31
SPEAKER_01: I'm asking the right, okay, you will have strong opinions, tell me.
1:26:34
SPEAKER_00: Yeah, yeah.
1:26:35
SPEAKER_00: I mean, I see that code review,
1:26:37
SPEAKER_00: I think,
1:26:37
SPEAKER_00: was fairly essential before the age of AI,
1:26:40
SPEAKER_00: but now we're getting to the point where I'm reviewing and looking at code.
1:26:44
SPEAKER_00: I look at most of the code that the agent's producing still,
1:26:47
SPEAKER_00: and yet it's like you're not giving it the same level of scrutiny, right?
1:26:51
SPEAKER_00: And this has always been the case.
1:26:53
SPEAKER_00: If you get a pull request from a junior engineer,
1:26:55
SPEAKER_00: you have to give it more scrutiny than if you get it from your most senior engineer.
1:26:58
SPEAKER_00: And you ask any tech lead, any engineer manager, any software engineer, they'll say, yeah, yeah.
1:27:03
SPEAKER_00: So the junior engineer or someone new to the code base You know,
1:27:06
SPEAKER_00: you just have to get more scrutiny.
1:27:08
SPEAKER_00: And what I'm finding is the agents are getting better,
1:27:10
SPEAKER_00: you're having to give less and less scrutiny.
1:27:11
SPEAKER_00: And yet you do still have to give scrutiny to some things.
1:27:13
SPEAKER_00: It's like I said, it's like the testing will be incomplete.
1:27:16
SPEAKER_00: Maybe they didn't follow the security stuff.
1:27:18
SPEAKER_00: Maybe, you know, like there was a performance regression.
1:27:20
SPEAKER_00: And I think it's like, you know, assuring there's kind of constraints on the system.
1:27:24
SPEAKER_00: So it's hard for them to do the wrong thing.
1:27:27
SPEAKER_00: I suspect that,
1:27:28
SPEAKER_00: you know,
1:27:29
SPEAKER_00: I don't know if it's going to be this year or next that we're materially going
1:27:32
SPEAKER_00: to stop looking at the code in the same way that we don't look at assembly anymore.
1:27:37
SPEAKER_01: We trust the compiler to produce pretty good assembly.
1:27:40
SPEAKER_00: We trust the compiler to produce good assembly.
1:27:42
SPEAKER_01: Unless you're one of those people where it's really important,
1:27:45
SPEAKER_01: you're maybe a game developer or someone and you look at it,
1:27:49
SPEAKER_01: but there's fewer and fewer of those folks.
1:27:51
SPEAKER_00: Yeah.
1:27:51
SPEAKER_00: And even then,
1:27:52
SPEAKER_00: I think what we'll be migrating to is why does the human have to look at
1:27:57
SPEAKER_00: the assembly?
1:27:58
SPEAKER_00: The AI should look at the assembly.
1:28:00
SPEAKER_00: Like I'm doing something right now where I want to have a zero overhead abstraction.
1:28:05
SPEAKER_00: You know,
1:28:06
SPEAKER_00: for doing something that is done at test time and when it's
1:28:08
SPEAKER_00: in compression is compiled out.
1:28:09
SPEAKER_00: Fable set up for me a little system where it actually looks at the decompiled code
1:28:13
SPEAKER_00: to verify like that there's only a few extra instructions put in place.
1:28:17
SPEAKER_00: I never would have put that in place,
1:28:18
SPEAKER_00: but now it has this guardrail that every time it changes this,
1:28:21
SPEAKER_00: it can verify that there's no regression.
1:28:23
SPEAKER_00: And I think you're going to see more and more of this where you kind of put guardrails in place.
1:28:27
SPEAKER_00: I think the model kind of likes it because it can work within that guardrail.
1:28:31
SPEAKER_01: Now, for 20 plus years, you were writing so much code.
1:28:36
SPEAKER_01: I'm sure you were in the zone.
1:28:38
SPEAKER_01: Do you remember being in the zone and just churning it out?
1:28:40
SPEAKER_01: You wrote a lot of production-ready code.
1:28:43
SPEAKER_01: Now that you're coding with AI, do you get into the zone?
1:28:47
SPEAKER_01: Yeah, absolutely.
1:28:48
SPEAKER_01: And how is that zone?
1:28:50
SPEAKER_01: Is it the same?
1:28:51
SPEAKER_01: Is it different?
1:28:52
SPEAKER_00: It definitely feels a little bit different.
1:28:54
SPEAKER_00: It's maybe a little bit less intense, but you're managing more things cognitively.
1:29:00
SPEAKER_01: Can you describe me, like, what is this right now when you're in the zone?
1:29:03
SPEAKER_00: Yeah.
1:29:03
SPEAKER_00: Well, I'm thinking of ideas that normally would have taken me a week to experiment with.
1:29:09
SPEAKER_00: And I think of multiple of these experiments, and then I fire them off all simultaneously.
1:29:13
SPEAKER_00: And then I'm kind of, like, reviewing, like, what else should I do while that's being complete?
1:29:18
SPEAKER_00: So Sometimes I'm kind of reviewing,
1:29:19
SPEAKER_00: like they'll be giving me progress updates of like, oh, hey, this is coming in.
1:29:22
SPEAKER_00: We're seeing this stuff.
1:29:23
SPEAKER_00: And I'm being like, well, that doesn't sound right.
1:29:25
SPEAKER_00: Hey, what about this?
1:29:25
SPEAKER_00: Or did we do this, you know, correctly?
1:29:27
SPEAKER_00: You know, maybe I had design and it's not implementing the design quite perfectly.
1:29:31
SPEAKER_00: But I have this little feeling that,
1:29:32
SPEAKER_00: you know,
1:29:33
SPEAKER_00: I haven't been a college professor,
1:29:34
SPEAKER_00: but maybe I was a,
1:29:35
SPEAKER_00: you know,
1:29:36
SPEAKER_00: if I was a college professor and I had a whole swarm of research assistants
1:29:39
SPEAKER_00: and they're all got off doing things and it's coming back,
1:29:41
SPEAKER_00: but it's coming back just really rapidly.
1:29:44
SPEAKER_01: You're not waiting months or weeks.
1:29:45
SPEAKER_00: I'm not waiting months or weeks.
1:29:46
SPEAKER_00: And then I'm iterating.
1:29:47
SPEAKER_00: I'm like, oh, that one failed.
1:29:48
SPEAKER_00: That's fine.
1:29:49
SPEAKER_00: You know, you just got to let go.
1:29:51
SPEAKER_00: And this, this is the nature of the software I build.
1:29:53
SPEAKER_00: I think there's other pieces where it's like,
1:29:55
SPEAKER_00: you just whip out a website,
1:29:56
SPEAKER_00: you can whip out something that doesn't have this level of kind of scrutiny,
1:29:59
SPEAKER_00: you know, very quickly.
1:30:00
SPEAKER_00: But I talk about this in a, I have a whole bunch of analogies about like the feeling debate.
1:30:04
SPEAKER_01: Let's talk about analogies.
1:30:06
SPEAKER_01: What analogies do you have about, Using AI or AI.
1:30:09
SPEAKER_00: The one I was,
1:30:10
SPEAKER_00: you know,
1:30:11
SPEAKER_00: advocating for,
1:30:11
SPEAKER_00: I had kind of two that I was advocating for,
1:30:13
SPEAKER_00: like late last year and this year, which is, you know, AI is coming for us.
1:30:18
SPEAKER_00: It's here, right?
1:30:19
SPEAKER_00: And it's like, you know, you're producing software, you're like walking down the road.
1:30:23
SPEAKER_00: And sometimes,
1:30:24
SPEAKER_00: you know,
1:30:24
SPEAKER_00: someone will pass you,
1:30:25
SPEAKER_00: they're running,
1:30:25
SPEAKER_00: they got an efficient gate and whatnot, but everyone's under their own locomotion.
1:30:30
SPEAKER_00: And these identity coding agents came and it was like a car pulled up next to you.
1:30:35
SPEAKER_00: You get into the car,
1:30:35
SPEAKER_00: you don't know how to drive,
1:30:36
SPEAKER_00: you don't understand the controls,
1:30:37
SPEAKER_00: but you gotta get in,
1:30:38
SPEAKER_00: you start,
1:30:39
SPEAKER_00: you might figure out the gas pedal, and it takes off and crashes into a tree.
1:30:43
SPEAKER_00: But you have to learn how to drive.
1:30:44
SPEAKER_00: I think using all these coding tools, it doesn't just happen naturally.
1:30:48
SPEAKER_00: It's learning how to drive.
1:30:49
SPEAKER_00: I think we might be in the era of F1 driver right now,
1:30:54
SPEAKER_00: which is like really good people can drive these systems a lot harder,
1:30:57
SPEAKER_00: a lot faster than the people who are just picking them up.
1:31:00
SPEAKER_00: If you've never used an authentic coding tool,
1:31:02
SPEAKER_00: there's a vast difference between someone who's like really expert in them,
1:31:06
SPEAKER_00: knows where they break,
1:31:07
SPEAKER_00: can pay attention to that,
1:31:09
SPEAKER_00: versus someone who's just picking it up for the first time.
1:31:11
SPEAKER_01: One analogy I've heard is, we used to talk about the 10x engineer, you remember.
1:31:16
SPEAKER_01: This used to be a debate for a very long time, is it or is it not?
1:31:20
SPEAKER_01: But now what I'm hearing is the 100x engineer.
1:31:23
SPEAKER_01: And so you're saying that you are seeing some folks who,
1:31:27
SPEAKER_01: maybe let's not use the term 100x engineer,
1:31:28
SPEAKER_01: but like this F1 driver who is just really, really good at it.
1:31:32
SPEAKER_01: How would you describe a person who you've seen this?
1:31:35
SPEAKER_01: Is it just like rock solid engineering basics and they picked up Are they leading
1:31:39
SPEAKER_01: into using these tools or what are they like?
1:31:41
SPEAKER_01: Yeah, yeah.
1:31:42
SPEAKER_00: I mean, there is quite a bit of a,
1:31:44
SPEAKER_00: it feels like,
1:31:45
SPEAKER_00: you know, it's directional that they're good at software engineering before.
1:31:48
SPEAKER_00: I mean, I sometimes think it's like,
1:31:50
SPEAKER_00: you know, everyone's in this kind of spectrum of capability of software engineering.
1:31:53
SPEAKER_00: And this is just like, you know, taking that line and spread it out.
1:31:56
SPEAKER_00: And it's not quite true.
1:31:58
SPEAKER_00: You know,
1:31:58
SPEAKER_00: I think it's helped some people more than others,
1:32:00
SPEAKER_00: but, you know, it feels like it's just stretched it out.
1:32:02
SPEAKER_00: So, you know, your ability before is now amplified.
1:32:04
SPEAKER_01: I'm always interested to learn that,
1:32:07
SPEAKER_01: for example,
1:32:08
SPEAKER_01: Boris Churny,
1:32:09
SPEAKER_01: Thibault at OpenAI, they both have been really, really good software engineers.
1:32:15
SPEAKER_01: Boris wrote one of the first TypeScript books, the first TypeScript book for O'Reilly.
1:32:19
SPEAKER_01: He built some massive systems, same with Thibault, who built it.
1:32:22
SPEAKER_01: And now you're kind of seeing, oh, these people are building all these tools and innovating.
1:32:25
SPEAKER_01: Like, yeah, they
1:32:28
SPEAKER_00: I mean, I have two other analogies to give you about what feels like an AI.
1:32:33
SPEAKER_00: You've undoubtedly heard about the term paraprogramming.
1:32:36
SPEAKER_00: Yeah, paraprogramming.
1:32:38
SPEAKER_00: And the idea behind paraprogramming is it's good
1:32:41
SPEAKER_00: to just like know I have one keyboard,
1:32:42
SPEAKER_00: one monitor,
1:32:43
SPEAKER_00: and two engineers at it,
1:32:44
SPEAKER_00: one at the keyboard,
1:32:45
SPEAKER_00: and the other one sitting beside them,
1:32:46
SPEAKER_00: kind of like looking over the shoulder and giving guidance.
1:32:48
SPEAKER_00: And I think there's an aspect of that feeling where I actually got Chachapiti to do
1:32:52
SPEAKER_00: a little image of this where,
1:32:53
SPEAKER_00: you know,
1:32:54
SPEAKER_00: it's like the Android is at the computer typing
1:32:56
SPEAKER_00: and you're just there giving instructions.
1:32:58
SPEAKER_00: But that was maybe the way it felt like a year ago.
1:33:00
SPEAKER_00: I think it feels a little bit different now.
1:33:02
SPEAKER_00: The one I just started recently saying is that I feel like the domain experts,
1:33:07
SPEAKER_00: the people who were really strong before are now massively amplified.
1:33:10
SPEAKER_00: Have you seen all this mathematical stuff coming out like the crazy proofs?
1:33:14
SPEAKER_01: Well, I don't understand it, but I don't understand.
1:33:16
SPEAKER_00: Yeah.
1:33:16
SPEAKER_01: Yeah.
1:33:17
SPEAKER_00: So I told you I wasn't very good at math.
1:33:19
SPEAKER_00: You know, it's not complete.
1:33:20
SPEAKER_00: I just was like, I stopped in the freshman year of college.
1:33:23
SPEAKER_00: Right.
1:33:23
SPEAKER_00: Yeah.
1:33:24
SPEAKER_00: But, you know,
1:33:24
SPEAKER_00: I kind of like watch along with these advancements and,
1:33:27
SPEAKER_00: you know,
1:33:27
SPEAKER_00: Terence Tao,
1:33:28
SPEAKER_00: he's like probably the most famous living mathematician, you know, super genius.
1:33:32
SPEAKER_00: He actually posted this session.
1:33:33
SPEAKER_00: There is this recent breakthrough.
1:33:34
SPEAKER_00: I think it's called the Jacobian conjecture.
1:33:36
SPEAKER_00: I don't even know what it meant, but he posted this chat GPT session where he's interacting.
1:33:40
SPEAKER_00: I think it was chat GPT.
1:33:42
SPEAKER_00: And you can see him interacting with this intelligence.
1:33:45
SPEAKER_00: And it was crazy because he's talking to it as a peer colleague and it's responding.
1:33:50
SPEAKER_00: And it honestly looks like, I'd encourage everyone to go look this up.
1:33:54
SPEAKER_00: It looks like, you know, almost a foreign language.
1:33:56
SPEAKER_00: It's like his domain expertise is getting amplified by the system he's interacting with.
1:34:01
SPEAKER_00: And you can see how he's like kind of learning and exploring ideas just really rapidly.
1:34:05
SPEAKER_00: Now there's all this controversy about AI mathematics,
1:34:07
SPEAKER_00: but I feel like the The analogy that comes to mind to me, though,
1:34:11
SPEAKER_00: is the domain experts,
1:34:13
SPEAKER_00: they're a little bit like sorcerers in their particular domain.
1:34:16
SPEAKER_00: You know, you have the earth sorcerers, the earth wizards, the water ones, whatnot.
1:34:20
SPEAKER_00: And if you know the magic incantations,
1:34:22
SPEAKER_00: the right words to say the right order, you actually get something kind of magical.
1:34:26
SPEAKER_00: And if you don't, you just get sparkable stuff that doesn't have anything there behind it.
1:34:30
SPEAKER_00: You know, you would probably admit you don't know much about distributed databases.
1:34:33
SPEAKER_00: If you're going to ask Fable or Astra,
1:34:35
SPEAKER_00: build me a distributed database like Cocker TV,
1:34:37
SPEAKER_00: you will get something out, but it'll kind of be ultimately hollow inside.
1:34:41
SPEAKER_00: But if you're an expert in databases and you have to build a distributed database
1:34:44
SPEAKER_00: and you can point out all the various things you have to know about
1:34:46
SPEAKER_00: a distributed database.
1:34:47
SPEAKER_00: Here's what you have to worry about, the storage layer, the networking layer.
1:34:49
SPEAKER_00: Here's the various data structures, runtime inside.
1:34:52
SPEAKER_00: You can actually get something quite magical very, very rapidly out of it.
1:34:56
SPEAKER_01: What would your advice be for engineers with mid-level to senior level
1:35:03
SPEAKER_01: who have been figuring out how to do coding to become strong engineers in this age
1:35:10
SPEAKER_01: of AI?
1:35:11
SPEAKER_00: Well, the first off is you have the most amazing tutor kind of readily at hand.
1:35:16
SPEAKER_00: And I mean,
1:35:18
SPEAKER_00: one of the things that I would always do,
1:35:20
SPEAKER_00: you know,
1:35:20
SPEAKER_00: throughout my career,
1:35:22
SPEAKER_00: and now I've kind of stopped doing it,
1:35:23
SPEAKER_00: but it's like the reason why is going to become obvious,
1:35:25
SPEAKER_00: which is like, I would always look at other people's code.
1:35:28
SPEAKER_00: So I was at, you know, Google early on.
1:35:31
SPEAKER_00: You probably heard of Jeff Dean.
1:35:33
SPEAKER_00: Jeff Dean was an amazing coder.
1:35:35
SPEAKER_00: His colleague, Sanjay Gamalat, also just an incredible coder.
1:35:38
SPEAKER_00: And, you know, I would be looking at their pull requests.
1:35:41
SPEAKER_00: I'd be looking at their changes.
1:35:42
SPEAKER_00: It wasn't called pull requests at Google.
1:35:43
SPEAKER_00: There's a different name for it, but I'd be looking at my- CL, right?
1:35:46
SPEAKER_00: Yeah.
1:35:47
SPEAKER_00: CL, this is P4, it's a different version control system.
1:35:51
SPEAKER_00: But I'd be looking at their changes, be like, how did they do what they did, right?
1:35:54
SPEAKER_00: I'd be looking at their code,
1:35:55
SPEAKER_00: oh my goodness,
1:35:56
SPEAKER_00: Sanjay's code is really always very elegant,
1:35:58
SPEAKER_00: you know,
1:35:58
SPEAKER_00: Jeff says high performance,
1:35:59
SPEAKER_00: how's he doing that, you know, like, how's he going about it?
1:36:01
SPEAKER_00: And it's almost just like you acquire it via osmosis.
1:36:05
SPEAKER_00: But now what you can do is,
1:36:06
SPEAKER_00: not just acquire it via osmosis,
1:36:07
SPEAKER_00: but you can literally, like, I mean,
1:36:09
SPEAKER_00: I would be encouraging,
1:36:10
SPEAKER_00: if you're a junior engineer,
1:36:11
SPEAKER_00: and you know there's a senior engineer nearby,
1:36:13
SPEAKER_00: Like you could ask him how they're doing what they're doing,
1:36:15
SPEAKER_00: but you can just ask the AI to dissect what they've done and explain it to you
1:36:20
SPEAKER_00: and explain to you at various different levels.
1:36:22
SPEAKER_00: Like, how does this code work?
1:36:23
SPEAKER_00: What is it doing?
1:36:25
SPEAKER_00: Give me a diagram,
1:36:26
SPEAKER_00: explain it to me like I'm five,
1:36:27
SPEAKER_00: explain it to me like I'm 10, explain it to me in French, whatever you want.
1:36:30
SPEAKER_00: Like, I mean, fundamentally, to some degree, AI is a translation tool.
1:36:34
SPEAKER_00: They translate it from whatever, you know, kind of language or understanding it's in.
1:36:38
SPEAKER_00: And keep on interrogating it until it gets it, you
1:36:40
SPEAKER_01: know, increases your understanding.
1:36:42
SPEAKER_01: You were saying how domain experts are very much amplified.
1:36:46
SPEAKER_01: I guess one strategy as a software engineer is like obviously become
1:36:49
SPEAKER_01: a great software engineer and use it as a tool.
1:36:51
SPEAKER_01: You can get a lot faster.
1:36:55
SPEAKER_01: I'm not a distributed databases expert at all.
1:36:58
SPEAKER_01: But I use AI to explain a few things for me to understand upfront, which was very helpful.
1:37:03
SPEAKER_01: And it would have taken me a lot longer time beforehand.
1:37:05
SPEAKER_01: So I can use this.
1:37:07
SPEAKER_01: But I wonder if there's another part of like,
1:37:08
SPEAKER_01: as a software engineer,
1:37:09
SPEAKER_01: you can use it to become more of a domain expert wherever you're working.
1:37:12
SPEAKER_01: If it's a payments company, I mean, use it to learn about payments as well.
1:37:15
SPEAKER_01: So you can help the business, you can help your team.
1:37:18
SPEAKER_01: And honestly, you'll just learn more, right?
1:37:20
SPEAKER_01: Yeah, yeah.
1:37:21
SPEAKER_00: I think, you know, I would encourage everyone, you have to have a little curiosity, right?
1:37:25
SPEAKER_00: Don't be bound in by, you know, the area you're working on.
1:37:28
SPEAKER_00: Explore outside of it.
1:37:29
SPEAKER_00: You know,
1:37:30
SPEAKER_00: I was working on Gmail,
1:37:31
SPEAKER_00: but I was fascinated about how, like, the main Google search engine worked.
1:37:35
SPEAKER_00: I was fascinated by, like,
1:37:36
SPEAKER_00: how the internals of Bigtable worked,
1:37:37
SPEAKER_00: even though I wasn't directly working on Bigtable.
1:37:39
SPEAKER_00: Just explore in.
1:37:40
SPEAKER_00: Look at those things.
1:37:41
SPEAKER_00: And now it's so much easier
1:37:43
SPEAKER_00: because you have this super advanced patient intelligence there to explain to you,
1:37:48
SPEAKER_00: like, why do you think it was done this way?
1:37:50
SPEAKER_00: And then, like, I mean, once you become an expert, you can be integrated.
1:37:53
SPEAKER_00: Well, I see it's done that way.
1:37:54
SPEAKER_00: Why don't we change this?
1:37:55
SPEAKER_00: Would this be helpful?
1:37:56
SPEAKER_00: And that's where you go from just kind of learning to actually contributing back.
1:38:00
SPEAKER_00: I think everyone has to have the personal agency to do this.
1:38:03
SPEAKER_00: If you're just sitting there waiting for someone to educate you on how to do this,
1:38:06
SPEAKER_00: it's going to be really hard right now because anyone who's coming in explaining how
1:38:10
SPEAKER_00: to use AI or explaining how to be a better software engineer,
1:38:12
SPEAKER_00: they're going to be out of date.
1:38:14
SPEAKER_00: You just got to get in there, be using these tools all the time yourself, and use it to learn.
1:38:20
SPEAKER_00: I feel like I've learned more in the past probably even year than
1:38:24
SPEAKER_00: the previous five years combined, which is weird.
1:38:27
SPEAKER_01: Given your trajectory and given the environment you were working in, right?
1:38:32
SPEAKER_00: I mean, anybody who's been in the industry for a while, I'm definitely a better coder.
1:38:36
SPEAKER_00: I was a better coder 10 years ago than when I first got into the industry.
1:38:38
SPEAKER_00: I can look back every decade and realize I got a lot better.
1:38:42
SPEAKER_00: And I feel like I just got a lot better over this past year.
1:38:44
SPEAKER_01: And this was also one of the reasons I was really excited to talk to you because
1:38:47
SPEAKER_01: when we started to just exchange messages,
1:38:50
SPEAKER_01: the first thing you wrote to me and I asked you like,
1:38:52
SPEAKER_01: hey, you know, how are things going?
1:38:54
SPEAKER_01: You said like,
1:38:54
SPEAKER_01: you wouldn't believe,
1:38:55
SPEAKER_01: but my coding output is insane and it's high quality and it's database quality.
1:38:59
SPEAKER_01: You are one example, right?
1:39:00
SPEAKER_01: Hopefully one of many.
1:39:15
SPEAKER_00: Yeah, yeah.
1:39:16
SPEAKER_00: No, and I'm not the only one in Cockroach Labs.
1:39:18
SPEAKER_00: We have other people doing this as well.
1:39:20
SPEAKER_00: I find it very exciting.
1:39:21
SPEAKER_00: You know, it's a little bit exhausting right now, but it's very exciting.
1:39:24
SPEAKER_00: Like, I got into software engineering because I like building stuff.
1:39:28
SPEAKER_00: I can build stuff faster.
1:39:30
SPEAKER_00: You know,
1:39:30
SPEAKER_00: the stuff you might have had to compromise on in the past,
1:39:33
SPEAKER_00: and you can take away some of those compromises.
1:39:34
SPEAKER_00: I mean, you see this in the UX of software coming out.
1:39:36
SPEAKER_00: I think the UX is a lot higher.
1:39:38
SPEAKER_00: You see all the fancy, like, web animations and whatnot.
1:39:41
SPEAKER_00: That's only just, like, the surface level.
1:39:43
SPEAKER_00: It just extends way, way below that.
1:39:45
SPEAKER_01: Peter, this was awesome.
1:39:46
SPEAKER_01: Thanks for coming on the podcast.
1:39:47
SPEAKER_00: Yeah, this is wonderful.
1:39:48
SPEAKER_00: Thanks for having me.
1:39:49
SPEAKER_01: One reason I was excited to talk to Peter is because he's been a very high profile
1:39:53
SPEAKER_01: and productive engineer,
1:39:54
SPEAKER_01: pre-AI, building some of the most resilient distributed systems in production.
1:39:58
SPEAKER_01: CockroachDB is known for its resilience and how even if several nodes are destroyed,
1:40:02
SPEAKER_01: the database still operates with a data loss.
1:40:05
SPEAKER_01: Basically, it's as hard to get rid of as cockroaches are, hence the name.
1:40:09
SPEAKER_01: One interesting part
1:40:09
SPEAKER_01: of our conversation was how Peter built more efficient data structures than
1:40:12
SPEAKER_01: the standard coding libraries had,
1:40:14
SPEAKER_01: thanks to him and colleagues paying attention to parts of the library
1:40:17
SPEAKER_01: that seemed slow.
1:40:18
SPEAKER_01: He did it for the C++ STL map and in Go for the switch table implementation.
1:40:23
SPEAKER_01: Both stories are a good reminder that you can improve the existing library or even
1:40:27
SPEAKER_01: the language, especially if you measure which parts feel slow.
1:40:30
SPEAKER_01: Another part of his conversation that I liked was how Peter came a bit of a full circle.
1:40:35
SPEAKER_01: He used to write 100,000 lines of code per year, being a very productive engineer and CTO.
1:40:40
SPEAKER_01: He didn't stop writing code aiming to coach engineers between 2022 and 2024.
1:40:45
SPEAKER_01: And then he started to code again because with AI tools,
1:40:48
SPEAKER_01: he wanted to coach his engineers better,
1:40:50
SPEAKER_01: but it's hard to do if you don't use the tools yourself.
1:40:53
SPEAKER_01: And now he finds himself being extremely productive.
1:40:56
SPEAKER_01: And this time the team around him is productive as well.
1:40:58
SPEAKER_01: And we're not talking about Vibe-coded software,
1:41:00
SPEAKER_01: but database worthy, high quality code generated and committed to production.
1:41:05
SPEAKER_01: Peter is convinced that AI amplifies existing expertise,
1:41:08
SPEAKER_01: and this is one reason why he probably learned more this last year,
1:41:11
SPEAKER_01: building with AI, than the previous five years combined.
1:41:14
SPEAKER_01: And I find it a valuable reminder that learning and building deep expertise
1:41:17
SPEAKER_01: in software engineering, this is very valuable.
1:41:19
SPEAKER_01: And as closing,
1:41:20
SPEAKER_01: I appreciated that Peter said that not only is he excited, but he's also exhausted.
1:41:25
SPEAKER_01: There's a lot to learn, but it's tiring, and neither him nor anyone I know is immune to this.
1:41:30
SPEAKER_01: So if you're also exhausted with all of the things going on with AI, know that you're not alone.
1:41:36
SPEAKER_01: Check the show notes for more of The Pragmatic Engineer deep dives
1:41:39
SPEAKER_01: on Google's engineering culture and on distributed systems.
1:41:41
SPEAKER_01: If you liked this episode,
1:41:43
SPEAKER_01: please make sure you're subscribed in your podcast player,
1:41:45
SPEAKER_01: and a special thank you if you leave a rating.
1:41:47
SPEAKER_01: Thanks, and I'll see you in the next one.