Latest / The Pragmatic Engineer / Software engineering at Google & CS education with Titus Winters

Transcript

Software engineering at Google & CS education with Titus Winters · The Pragmatic Engineer

• 0:00
  SPEAKER_00: What if an engineer who helped define the contents
• 0:02
  SPEAKER_00: of today's software engineering curriculum said that most
• 0:04
  SPEAKER_00: of this should be burned down?
• 0:05
  SPEAKER_00: An ex-Google software engineer says exactly this.
• 0:08
  SPEAKER_00: In 2023,
• 0:09
  SPEAKER_00: Titus Winters was the only active software engineer on the steering committee for
• 0:12
  SPEAKER_00: the ACM and IEEE undergraduate CS curriculum.
• 0:15
  SPEAKER_00: Before, he spent more than 10 years at Google,
• 0:18
  SPEAKER_00: was the lead author of the book Software Engineering at Google,
• 0:20
  SPEAKER_00: and ran the company's readability program.
• 0:23
  SPEAKER_00: In today's conversation,
• 0:24
  SPEAKER_00: we discuss why Google made test manager around 2006 after
• 0:28
  SPEAKER_00: the flagship search product could no longer be released on a regular cadence due
• 0:32
  SPEAKER_00: to persistent regressions,
• 0:34
  SPEAKER_00: ways to measure developer productivity,
• 0:36
  SPEAKER_00: and why the value of AI does not show up in the metrics that we measure today,
• 0:40
  SPEAKER_00: ways that Titus would change computer science education,
• 0:43
  SPEAKER_00: including why they'd want students to be graded on their prompts and tests,
• 0:46
  SPEAKER_00: not the code, and more.
• 0:48
  SPEAKER_00: If you want to know what Google learned about engineering at scale,
• 0:51
  SPEAKER_00: or how modern computer science education should look like in the age of AI,
• 0:55
  SPEAKER_00: this episode is for you.
• 0:57
  SPEAKER_00: This episode is presented by TurboPuffer, vector and full textures built on object storage.
• 1:01
  SPEAKER_00: It's fast, cheap, and extremely scalable.
• 1:04
  SPEAKER_00: Before we start, I'd like to talk about Linear, our season sponsor.
• 1:08
  SPEAKER_00: Until recently,
• 1:08
  SPEAKER_00: I was thinking of Linear as a tool that helps plan and organize the work,
• 1:12
  SPEAKER_00: but it's much more than that.
• 1:13
  SPEAKER_00: A lot of teams use Linear to ship changes from IDEA to merge in GitHub all
• 1:18
  SPEAKER_00: without leaving the tool.
• 1:20
  SPEAKER_00: I sat down with Dima,
• 1:21
  SPEAKER_00: an engineer on the Linear team to see how devs inside of Linear use Linear
• 1:25
  SPEAKER_00: to build all of Linear.
• 1:26
  SPEAKER_00: So here's a real issue.
• 1:27
  SPEAKER_00: When you submit a review with unsent draft comments,
• 1:29
  SPEAKER_00: Linear warns you about these comments,
• 1:32
  SPEAKER_00: but it doesn't tell you where those drafts are.
• 1:34
  SPEAKER_00: A customer complained and Dima decided to fix it.
• 1:37
  SPEAKER_00: But for the fix, Dima did not open his ID.
• 1:40
  SPEAKER_00: Instead, he opened up linear and described the issue to the agent.
• 1:44
  SPEAKER_00: The agent built a fix, created a PR, and attached storybook screenshots for easy verification.
• 1:49
  SPEAKER_00: You can see the fix with the draft comments linked on this screenshot.
• 1:52
  SPEAKER_00: In the PR, there's a versatile preview.
• 1:55
  SPEAKER_00: You can open the preview with one click from linear to play around with the change,
• 1:58
  SPEAKER_00: and we see that it worked.
• 2:00
  SPEAKER_00: And here's my favorite part, guided reviews, which is work in progress.
• 2:04
  SPEAKER_00: Guided reviews is a new code review experience where the important stuff just pops out.
• 2:09
  SPEAKER_00: The top of the review has a summary and diagram of the change.
• 2:12
  SPEAKER_00: And the pull request is divided into chapters that walk you through what's changed.
• 2:17
  SPEAKER_00: These blocks highlight the important changes in a way that's easier to understand and review.
• 2:21
  SPEAKER_00: Oh, and when Dima is finished with all of this,
• 2:23
  SPEAKER_00: he just self-approves the code and is merged into GitHub,
• 2:25
  SPEAKER_00: all without leaving Linear.
• 2:27
  SPEAKER_00: If you take one thing away from this,
• 2:29
  SPEAKER_00: Linear is no longer just a place for planning tickets and organizing the work.
• 2:32
  SPEAKER_00: It's where all the work happens,
• 2:34
  SPEAKER_00: creating the issue,
• 2:35
  SPEAKER_00: kicking out the agent,
• 2:36
  SPEAKER_00: checking out the preview, doing the review, and merging to production.
• 2:39
  SPEAKER_00: Linear is the product development system for teams and for AI agents,
• 2:43
  SPEAKER_00: and I'm loving this evolution.
• 2:45
  SPEAKER_00: Check it out and get a six-month trial at linear.app.pagmatic.
• 2:49
  SPEAKER_00: Titus, welcome to the podcast.
• 2:50
  SPEAKER_00: Thank you so much for having me.
• 2:51
  SPEAKER_00: I'm so excited to have this one.
• 2:53
  SPEAKER_00: It's awesome to do this in
• 2:55
  SPEAKER_01: person again.
• 2:57
  SPEAKER_00: Yeah, it's a joy.
• 2:58
  SPEAKER_00: So how did you get into tech?
• 3:01
  SPEAKER_01: Growing up,
• 3:01
  SPEAKER_01: did you know that
• 3:03
  SPEAKER_00: you're going to do something with
• 3:04
  SPEAKER_01: I mean,
• 3:04
  SPEAKER_01: my earliest memories,
• 3:05
  SPEAKER_01: I thought I was going to be a scientist,
• 3:08
  SPEAKER_01: but like,
• 3:08
  SPEAKER_01: you know,
• 3:08
  SPEAKER_01: as a three-year-old,
• 3:09
  SPEAKER_01: that means like lab coats and test tubes and I don't know what else, right?
• 3:13
  SPEAKER_01: My dad very insightfully knew that computers would be important.
• 3:18
  SPEAKER_01: Like that
• 3:18
  SPEAKER_01: was not his field
• 3:19
  SPEAKER_00: at all.
• 3:19
  SPEAKER_00: He was kind of in carpentry and set design, things like that.
• 3:22
  SPEAKER_01: So there was always a computer in the house.
• 3:24
  SPEAKER_01: It was usually kind of a low-end thing of,
• 3:26
  SPEAKER_01: you know, whatever we could scrape together and afford.
• 3:29
  SPEAKER_01: And then I remember I really got my start.
• 3:32
  SPEAKER_01: I think I was around 11 or 12.
• 3:34
  SPEAKER_01: And instead of buying candy for everyone for Valentine's one year, he went to the bookstore.
• 3:40
  SPEAKER_01: And I don't know what my brother got, but I got a book on LearnQBasic.
• 3:45
  SPEAKER_01: And it was all kind of downhill from there.
• 3:47
  SPEAKER_01: Like I spent probably the next five,
• 3:50
  SPEAKER_01: six years like programming, teaching myself to program was my hobby.
• 3:54
  SPEAKER_01: Occasionally I would wind up like meeting,
• 3:56
  SPEAKER_01: you know, other kids my age that were a little into it.
• 3:59
  SPEAKER_01: We moved eventually to Las Vegas and I went to a magnet school that had
• 4:03
  SPEAKER_01: a computer science program that is actually
• 4:06
  SPEAKER_01: a really well regarded high school these days.
• 4:08
  SPEAKER_01: I had no idea.
• 4:09
  SPEAKER_01: And yeah, like just kind of self-taught initially.
• 4:12
  SPEAKER_01: Pre-internet, back when you had to really fight for it.
• 4:17
  SPEAKER_01: But yeah, that set me on my path, that book.
• 4:20
  SPEAKER_00: And then you, after high school, it was kind of obvious you went to college for it.
• 4:26
  SPEAKER_00: Yeah.
• 4:27
  SPEAKER_01: So I went to Harvey Mudd,
• 4:27
  SPEAKER_01: which is a great school with a mix of,
• 4:30
  SPEAKER_01: you know,
• 4:31
  SPEAKER_01: tech and liberal arts education,
• 4:33
  SPEAKER_01: the kind of place that'll teach you to build weapons and teach you not to use them,
• 4:38
  SPEAKER_01: that sort of thing.
• 4:39
  SPEAKER_01: And then I met my wife there,
• 4:42
  SPEAKER_01: and she was a little bit more junior than I was, so I didn't want to move away.
• 4:46
  SPEAKER_01: She was still in school,
• 4:48
  SPEAKER_01: so I did my master's at UC Riverside, actually in computer science education.
• 4:55
  SPEAKER_01: I did a kind of weird,
• 4:57
  SPEAKER_01: I had a very strange graduate experience,
• 4:59
  SPEAKER_01: but yeah,
• 5:00
  SPEAKER_01: I spent four years at Riverside doing a PhD in CSED,
• 5:04
  SPEAKER_01: kind of applied AI, educational data mining, computer science education.
• 5:10
  SPEAKER_01: So some of the earliest conferences on CSED research I published there.
• 5:17
  SPEAKER_01: And my advisor was the department chair.
• 5:21
  SPEAKER_01: He never generally ran grad students,
• 5:23
  SPEAKER_01: like the research area was not his focus, but he was constantly the chair.
• 5:28
  SPEAKER_01: He was this like charming, you know, social, political sort of guy that knew all the stories.
• 5:34
  SPEAKER_01: And so my graduate work was a lot of learn all of the stories of how we got here
• 5:39
  SPEAKER_01: and teach all of the classes whenever there was a hole in the Yeah,
• 5:43
  SPEAKER_00: and we'll keep coming back to probably this,
• 5:46
  SPEAKER_00: like this education, computer science thing throughout this thing.
• 5:49
  SPEAKER_00: So we've talked this before, so I know.
• 5:52
  SPEAKER_00: But let's jump back.
• 5:53
  SPEAKER_00: So like you now graduated from college.
• 5:57
  SPEAKER_00: You were looking for a job.
• 5:57
  SPEAKER_00: Rob?
• 5:58
  SPEAKER_00: Yep.
• 5:59
  SPEAKER_00: What did you find?
• 6:01
  SPEAKER_01: So first I did a sort of consulting group.
• 6:04
  SPEAKER_01: Again, two-body problem.
• 6:06
  SPEAKER_01: My wife went to grad school in San Diego.
• 6:08
  SPEAKER_01: I did not get a professorship after my PhD.
• 6:11
  SPEAKER_01: So I did this consulting stuff.
• 6:13
  SPEAKER_01: I was at a small startup consulting group, about 30, 50 people.
• 6:18
  SPEAKER_01: Mostly I worked for, indirectly for DirecTV, doing the embedded system stuff on set-top boxes.
• 6:24
  SPEAKER_01: Mm-hmm.
• 6:24
  SPEAKER_00: And then after four years of that, we were done with grad schools.
• 6:28
  SPEAKER_00: We were done with all of that.
• 6:30
  SPEAKER_01: I took the offer to do a Google interview,
• 6:34
  SPEAKER_01: got the offer, decided not to go to the Bay Area, and they moved us to New York.
• 6:40
  SPEAKER_01: So we've been here ever since.
• 6:42
  SPEAKER_00: This is when Google was opening an office here.
• 6:44
  SPEAKER_01: The office was already open and reasonably big.
• 6:47
  SPEAKER_01: This was 2010.
• 6:48
  SPEAKER_01: So I think there were a few thousand people there.
• 6:51
  SPEAKER_00: And then in 2010, so Google was founded in 98, they went public in 2004.
• 6:58
  SPEAKER_00: I remember that they started to really accumulate a global brand as like,
• 7:01
  SPEAKER_00: wow, Google is the place to work at around like 2005 to 2010.
• 7:05
  SPEAKER_00: So right around when you were applying,
• 7:07
  SPEAKER_00: it was actually one of the hardest places to get into a bit like today,
• 7:10
  SPEAKER_00: Entropic or last year, OpenAI or the You know, like in the mid-2010s Uber.
• 7:18
  SPEAKER_00: How did you find the interview experience and getting an offer?
• 7:21
  SPEAKER_00: Because it must have been pretty competitive at that point.
• 7:23
  SPEAKER_00: It was competitive.
• 7:24
  SPEAKER_01: I was a little surprised that they gave me the offer.
• 7:27
  SPEAKER_01: I felt like I didn't do that great.
• 7:29
  SPEAKER_01: I remember...
• 7:31
  SPEAKER_01: For some stupid reason,
• 7:32
  SPEAKER_01: I really, really didn't want to spend a bunch of like precious vacation days.
• 7:36
  SPEAKER_01: So I flew up from San Diego the morning of instead of sensibly like, you know.
• 7:41
  SPEAKER_01: They would have paid for the hotel and everything.
• 7:43
  SPEAKER_00: Yeah, I just didn't want like,
• 7:45
  SPEAKER_01: I don't know.
• 7:45
  SPEAKER_01: I'm an anxious person.
• 7:46
  SPEAKER_01: I like my habits and routines and stuff.
• 7:49
  SPEAKER_01: Getting up at five for a big interview is not a good plan, folks.
• 7:53
  SPEAKER_01: Hot tip.
• 7:54
  SPEAKER_01: So there are a couple of things like I remember one interview was like
• 7:57
  SPEAKER_01: a relatively simple like clock math sort of thing.
• 8:01
  SPEAKER_01: It was technically camera angles on a Mars rover or something.
• 8:06
  SPEAKER_01: And there was a point in there was like,
• 8:08
  SPEAKER_01: I'm tired and having interview brain and this is just basic math.
• 8:12
  SPEAKER_01: So here's a function that has the right inputs and outputs.
• 8:16
  SPEAKER_01: And if I wasn't stupid this morning, that would work.
• 8:18
  SPEAKER_01: And I just did it all in terms of that.
• 8:20
  SPEAKER_01: And I think my interviewer actually really liked that approach of like,
• 8:25
  SPEAKER_01: yeah, this is not the important part.
• 8:26
  SPEAKER_01: Like, let's just talk.
• 8:28
  SPEAKER_01: And so, yeah, eventually they gave me the offer and, yeah, moved to New York and started up.
• 8:33
  SPEAKER_00: And when you joined, you got the nuclear hat.
• 8:35
  SPEAKER_01: Uh-huh.
• 8:36
  SPEAKER_01: Yep, absolutely.
• 8:38
  SPEAKER_01: And so what did you work on
• 8:39
  SPEAKER_00: at Google
• 8:40
  SPEAKER_01: when you joined?
• 8:41
  SPEAKER_01: So briefly I was on and like I don't think that Google had the best placement
• 8:47
  SPEAKER_01: for new hires necessarily.
• 8:49
  SPEAKER_01: Onboarding process that we say.
• 8:51
  SPEAKER_01: That was a little rough because like I said the things that I'm interested
• 8:54
  SPEAKER_01: in are like Developer efficiency, problem solve.
• 8:58
  SPEAKER_01: Like I have a background in networks.
• 9:00
  SPEAKER_01: I have a background in like API design, those sorts of things.
• 9:03
  SPEAKER_01: They said, oh, you know, networks.
• 9:06
  SPEAKER_01: So let's put you on this team that monitors the amount of traffic and where
• 9:13
  SPEAKER_01: that is crossing the edge of the Google network,
• 9:16
  SPEAKER_01: which is actually a databases problem.
• 9:20
  SPEAKER_01: With networks as the topic of all of the data that you're doing.
• 9:23
  SPEAKER_01: I'm like, I don't know anything about databases.
• 9:25
  SPEAKER_01: That was never my thing.
• 9:26
  SPEAKER_01: So it wasn't a great fit.
• 9:28
  SPEAKER_01: And then almost immediately after that, like two months in.
• 9:31
  SPEAKER_00: I want to push a little bit back here already.
• 9:35
  SPEAKER_00: It's like,
• 9:35
  SPEAKER_00: come on,
• 9:36
  SPEAKER_00: you're joining Google, you had college, you had four or five years of experience.
• 9:41
  SPEAKER_00: When you joined a company,
• 9:43
  SPEAKER_00: especially a hot company,
• 9:44
  SPEAKER_00: they said, all right, do X, and you're like, I'm not really there.
• 9:48
  SPEAKER_00: Would you not want to say,
• 9:50
  SPEAKER_00: let me just go deep into this to see what's important for the business?
• 9:54
  SPEAKER_00: I am pushing back on you here.
• 9:56
  SPEAKER_01: Yeah, no, please do, always.
• 9:58
  SPEAKER_01: I...
• 9:59
  SPEAKER_01: Did what I could to learn, you know, from that team, for sure.
• 10:02
  SPEAKER_01: There was a little like interpersonal tension in some of those things.
• 10:06
  SPEAKER_00: Okay, so it already started a bit like that.
• 10:08
  SPEAKER_01: You could have had this nagging feeling.
• 10:09
  SPEAKER_01: Yeah, like it was just like, yeah, this wasn't quite right.
• 10:12
  SPEAKER_01: And then there was a like email that went out to all the engineers
• 10:16
  SPEAKER_01: in New York saying like, hey,
• 10:17
  SPEAKER_01: we're thinking about spinning up a new team that handles all of
• 10:22
  SPEAKER_01: the old like C++ nuts and bolts infrastructure code.
• 10:26
  SPEAKER_01: That was written by people whose employee IDs are like two digits.
• 10:29
  SPEAKER_01: They have more important things to do than to handle like patch requests for all
• 10:34
  SPEAKER_01: of those components.
• 10:36
  SPEAKER_01: So we would like people to maintain that.
• 10:39
  SPEAKER_01: And me and I don't know,
• 10:41
  SPEAKER_01: nine or 10 other people like raised our hands and I was the only person
• 10:44
  SPEAKER_01: that showed up to like the second meeting.
• 10:46
  SPEAKER_01: So I just kind of got onto that team.
• 10:49
  SPEAKER_00: I get it.
• 10:50
  SPEAKER_00: So basically,
• 10:51
  SPEAKER_00: I think the context is just important because I think,
• 10:53
  SPEAKER_00: you know, folks hearing, oh, I'm joining this team.
• 10:55
  SPEAKER_00: Like, I didn't really like the first thing.
• 10:57
  SPEAKER_00: I think the difference,
• 10:58
  SPEAKER_00: the context that was missing is like,
• 10:59
  SPEAKER_00: first of all, Google was growing really quickly.
• 11:01
  SPEAKER_00: New teams were starting up left and right.
• 11:03
  SPEAKER_00: And also like, Moving teams was kind of fine, right?
• 11:07
  SPEAKER_00: And you were more looking for like,
• 11:09
  SPEAKER_00: all right,
• 11:09
  SPEAKER_00: like you knew that this team,
• 11:12
  SPEAKER_00: both for what they were doing and also sometimes there's not a great people fit.
• 11:15
  SPEAKER_00: It would probably not be the one where you're going to be staying for many years.
• 11:19
  SPEAKER_00: You did the best you can.
• 11:20
  SPEAKER_00: And then when you saw an opportunity, you knew it was better for your strengths.
• 11:23
  SPEAKER_00: You're like, let's go for it.
• 11:24
  SPEAKER_00: So for you, C++, low level stuff.
• 11:27
  SPEAKER_01: Yeah, I mean,
• 11:28
  SPEAKER_01: back when I had been doing the consulting at DirecTV,
• 11:30
  SPEAKER_01: there was stuff like build an API to do UPnP, like Universal Plug-in Plays.
• 11:36
  SPEAKER_01: It was like local network RPC sorts of stuff.
• 11:40
  SPEAKER_01: Remote positive call.
• 11:41
  SPEAKER_00: Yeah, right.
• 11:42
  SPEAKER_01: And so it had been a lot of people,
• 11:46
  SPEAKER_01: teams were manually assembling like the XML structure for every request
• 11:52
  SPEAKER_01: to like implement a single client call or something like that.
• 11:55
  SPEAKER_01: It was like, we could do better folks.
• 11:57
  SPEAKER_01: And like,
• 11:58
  SPEAKER_01: I was probably a little too clever in the way that I assembled all of that API,
• 12:02
  SPEAKER_01: but it was like, yeah, it was a thing that I liked.
• 12:06
  SPEAKER_00: Was it more of an initiative of like, hey,
• 12:09
  SPEAKER_00: Google has a bunch of C++ here and there,
• 12:11
  SPEAKER_00: kind of like roots up, like,
• 12:13
  SPEAKER_00: let's get together and initially in our free time, let's see if we can do something?
• 12:17
  SPEAKER_00: Or was it more some manager type said,
• 12:19
  SPEAKER_00: like, I'm actually Do you think of assembling a team, a full-time team on this?
• 12:22
  SPEAKER_00: So this was,
• 12:23
  SPEAKER_00: I think,
• 12:23
  SPEAKER_00: like the last
• 12:25
  SPEAKER_01: year before Google got actually pretty formal about you're embedded to
• 12:30
  SPEAKER_01: a particular team and headcount is limited.
• 12:32
  SPEAKER_01: It was a little chaotic.
• 12:33
  SPEAKER_01: It's a little loosey-goosey about how those things work.
• 12:36
  SPEAKER_01: I could actually be wrong about that because I was still,
• 12:39
  SPEAKER_01: you know,
• 12:39
  SPEAKER_01: new enough that the internals of how headcount works might have been just not clear.
• 12:44
  SPEAKER_01: But yeah, like moved over pretty quick and boy, that was exactly the place for me.
• 12:49
  SPEAKER_00: All right, tell me about it.
• 12:50
  SPEAKER_00: Why was it the place?
• 12:52
  SPEAKER_00: What kind of work can we imagine of, like, I mean,
• 12:55
  SPEAKER_00: because it sounds like when you describe to me, like,
• 12:58
  SPEAKER_00: taking random C++ stuff that people wrote at Google, like,
• 13:01
  SPEAKER_00: that sounds like the version of hell for me,
• 13:03
  SPEAKER_00: like, you know, like, as an engineer, especially pre-AI, right?
• 13:05
  SPEAKER_00: Like,
• 13:06
  SPEAKER_00: Understanding what another engineer at some point wrote who might have left and
• 13:10
  SPEAKER_00: a team is using and it's working for them.
• 13:12
  SPEAKER_00: It sounds like it's a lose-lose because if you take it over and you make a change
• 13:17
  SPEAKER_00: and you break someone, they'll be mad at you.
• 13:19
  SPEAKER_00: And if you do it and you do a great job,
• 13:22
  SPEAKER_00: they'll probably not even think about it because it's always worked.
• 13:24
  SPEAKER_00: It still works.
• 13:25
  SPEAKER_00: Great.
• 13:27
  SPEAKER_01: Luckily,
• 13:28
  SPEAKER_01: the fear and the safety net for making changes had been pretty well established
• 13:33
  SPEAKER_01: a couple of years before this played out.
• 13:35
  SPEAKER_01: Specifically, and this goes to topics that we'll get to in a second, I'm sure.
• 13:40
  SPEAKER_01: Specifically, Google already went through the phase of, yes, you have to write tests.
• 13:46
  SPEAKER_01: In 2010, this was not an obvious thing in the industry.
• 13:49
  SPEAKER_01: Right.
• 13:49
  SPEAKER_01: But like,
• 13:50
  SPEAKER_01: I think the story is 2006,
• 13:53
  SPEAKER_01: 2007, Search lost the ability to push a release on a regular cadence.
• 14:00
  SPEAKER_01: Like, it was down to, like,
• 14:03
  SPEAKER_01: we were trying for monthly and they were maybe only get once a quarter because...
• 14:08
  SPEAKER_00: Oh, so this is why it's Google obsession with tests and quality and... Uh-huh.
• 14:13
  SPEAKER_00: So the core money-making product was kind of handcuffed,
• 14:17
  SPEAKER_00: not being able to push around through 2006.
• 14:19
  SPEAKER_01: Wow.
• 14:20
  SPEAKER_01: And circa 2006,
• 14:22
  SPEAKER_01: almost anything that you did had some tendrils in like the front end or search
• 14:27
  SPEAKER_01: or stuff like that,
• 14:28
  SPEAKER_01: the whole architecture and separation and like multi-app,
• 14:32
  SPEAKER_01: multi-products, like world.
• 14:35
  SPEAKER_01: That was still not, right?
• 14:36
  SPEAKER_01: So if you were making changes to Gmail,
• 14:38
  SPEAKER_01: there were still chances that you were making changes to search.
• 14:41
  SPEAKER_01: And so everything was holed up.
• 14:45
  SPEAKER_01: So the release engineers,
• 14:47
  SPEAKER_01: they banded together and they simply just said,
• 14:51
  SPEAKER_01: you will not merge this change if there is not tests, period.
• 14:53
  SPEAKER_01: We do not care what VP says anything.
• 14:56
  SPEAKER_01: We do not care how important this feature is.
• 14:58
  SPEAKER_01: You will not do that.
• 15:00
  SPEAKER_01: We will roll that back.
• 15:02
  SPEAKER_01: And as soon as you put a line in the sand,
• 15:05
  SPEAKER_01: Even the people that are like, I'm very smart, how dare you ask me to write tests?
• 15:08
  SPEAKER_01: Like, this is beneath me.
• 15:10
  SPEAKER_00: And then so basically taking over or taking on maintenance
• 15:13
  SPEAKER_00: or even changing any library,
• 15:15
  SPEAKER_00: C++ library,
• 15:16
  SPEAKER_00: you had the test so you could have high confidence if you refactor these things,
• 15:19
  SPEAKER_00: all the tests pass.
• 15:21
  SPEAKER_00: Yeah.
• 15:21
  SPEAKER_00: Either it works or...
• 15:23
  SPEAKER_00: There was never enough test coverage.
• 15:25
  SPEAKER_00: Right, exactly.
• 15:26
  SPEAKER_00: And either way, it's a win.
• 15:28
  SPEAKER_00: Because it's better if you expose the lack of test coverage.
• 15:32
  SPEAKER_00: Okay, you have a crash today,
• 15:33
  SPEAKER_00: but tomorrow or a year from now, you'll have twice as much traffic anyway.
• 15:36
  SPEAKER_01: Right, exactly.
• 15:37
  SPEAKER_00: And like...
• 15:39
  SPEAKER_01: It's one of the things that I believe very deeply is if you are held back
• 15:43
  SPEAKER_01: from making changes to a thing out of fear, you have to go in and fix that.
• 15:47
  SPEAKER_01: Like fear is not an acceptable basis for engineering.
• 15:51
  SPEAKER_01: And I think it's in the first two or three pages of the book.
• 15:55
  SPEAKER_01: I've said now for 10 years,
• 15:57
  SPEAKER_01: you know,
• 15:59
  SPEAKER_01: your code base is sustainable if for any valuable change that comes along,
• 16:04
  SPEAKER_01: you are capable of making that change safely, right?
• 16:07
  SPEAKER_01: You may choose not to make that change, right?
• 16:10
  SPEAKER_01: That could be technical debt or, you know, different product priorities or whatever.
• 16:15
  SPEAKER_01: But if you're incapable of reacting to change,
• 16:18
  SPEAKER_01: then you're placing a high bet or high risk bet that the change isn't critical
• 16:22
  SPEAKER_01: or isn't important.
• 16:24
  SPEAKER_01: And I hate to break it to people,
• 16:26
  SPEAKER_01: but the security world has gotten real, real bad in the last few years.
• 16:31
  SPEAKER_01: There is nothing that is not going to change.
• 16:34
  SPEAKER_01: The strategic advantage of being able to change is incredible.
• 16:39
  SPEAKER_00: Is it safe to say that if I join a company,
• 16:43
  SPEAKER_00: a startup,
• 16:44
  SPEAKER_00: a large company,
• 16:45
  SPEAKER_00: whatever company,
• 16:46
  SPEAKER_00: I joined a team and I look at the code base and my first weeks,
• 16:51
  SPEAKER_00: my first months,
• 16:53
  SPEAKER_00: I'm just being afraid of pushing code changes because I worry this thing will break.
• 16:58
  SPEAKER_00: I'll break production.
• 17:00
  SPEAKER_00: That I'm working on a system that is just, let's say, poorly engineered.
• 17:04
  SPEAKER_01: It could be the system is poorly engineered or it could be there's some onboarding
• 17:08
  SPEAKER_01: and mentoring that didn't happen there.
• 17:10
  SPEAKER_01: So one or the other probably.
• 17:11
  SPEAKER_01: One or the other for sure.
• 17:12
  SPEAKER_01: And I think, and this goes to some of the educational stuff, like we...
• 17:18
  SPEAKER_01: In undergrad,
• 17:19
  SPEAKER_01: you know,
• 17:19
  SPEAKER_01: education and like the way that we kind of portray what the job is like,
• 17:24
  SPEAKER_01: there is a correct answer.
• 17:25
  SPEAKER_01: There is right and wrong.
• 17:27
  SPEAKER_01: It is not just the tradeoffs.
• 17:29
  SPEAKER_01: And you're graded anytime someone looks at what you have done, right?
• 17:34
  SPEAKER_01: And I think it's going to be, especially for new grads, it's really hard.
• 17:38
  SPEAKER_01: It's really easy to get into a fear of like, I don't know if I did this right.
• 17:42
  SPEAKER_01: I don't know if this was the answer you were looking for
• 17:44
  SPEAKER_01: because they've been falsely given this perspective that there's
• 17:48
  SPEAKER_01: a right answer instead of, of course, it's just a tradeoff, right?
• 17:51
  SPEAKER_01: It's at some point in the design space.
• 17:53
  SPEAKER_00: And I guess you're also saying that these new grads are also used
• 17:56
  SPEAKER_00: to you do something that is a meaningful effort,
• 17:59
  SPEAKER_00: typically a class or an exam, and you get feedback.
• 18:02
  SPEAKER_00: You get a scale, you get a grade, you get a thumbs up, a pass, or a fail at the very least.
• 18:08
  SPEAKER_00: And I know I should remember when I went to my first workplace,
• 18:11
  SPEAKER_00: I should remember this really weird,
• 18:13
  SPEAKER_00: like, join the company, build this project, and then nothing.
• 18:19
  SPEAKER_00: And then I, you know, I do another thing and then nothing.
• 18:23
  SPEAKER_00: And then I have a,
• 18:24
  SPEAKER_00: I have a one-on-one with the CEO of the company and I'm thinking they're going
• 18:27
  SPEAKER_00: to fire me because like clearly like I,
• 18:30
  SPEAKER_00: I knew that I don't know,
• 18:31
  SPEAKER_00: like middle of the day,
• 18:32
  SPEAKER_00: like I was sometimes slacking off because I finished my stuff.
• 18:34
  SPEAKER_00: In hindsight,
• 18:35
  SPEAKER_00: I was pretty efficient,
• 18:36
  SPEAKER_00: but I just finished my stuff and then I was just browsing the internet.
• 18:38
  SPEAKER_00: I told them I'm ready.
• 18:40
  SPEAKER_00: And I was pretty sure like, Maybe I'll be re-imprated or something.
• 18:43
  SPEAKER_00: And instead, I'll be like, oh, amazing job.
• 18:45
  SPEAKER_00: Like you've onboarded so quickly.
• 18:47
  SPEAKER_00: We're so happy with you.
• 18:49
  SPEAKER_00: But the difference was that up until then,
• 18:51
  SPEAKER_00: I've had years of getting direct feedback and it didn't connect to my mind.
• 18:55
  SPEAKER_00: Yeah.
• 18:56
  SPEAKER_01: The other part of that is like the only time that a human is going to look
• 19:01
  SPEAKER_01: at your code if you're an undergrad is to grade you, is to judge you.
• 19:06
  SPEAKER_01: Whereas when you become a more experienced engineer,
• 19:11
  SPEAKER_01: you're asking for feedback regularly, but it's just a check-in.
• 19:16
  SPEAKER_01: Are we going in the right direction?
• 19:17
  SPEAKER_01: Does this make sense to you?
• 19:19
  SPEAKER_01: And the more of that feedback that you can get that isn't judgment,
• 19:22
  SPEAKER_01: The more effective you will be.
• 19:24
  SPEAKER_01: I'm like, oh,
• 19:26
  SPEAKER_01: I hate that we've like given all of the new grads and all
• 19:30
  SPEAKER_01: the junior engineers exactly the wrong instinct.
• 19:32
  SPEAKER_01: So going back to Google, what is Abseil?
• 19:38
  SPEAKER_00: How do you pronounce it?
• 19:38
  SPEAKER_00: Abseil.
• 19:40
  SPEAKER_01: Yeah.
• 19:41
  SPEAKER_01: So Abseil was starting about 10 years ago,
• 19:44
  SPEAKER_01: shortly after the initial pivot to cloud,
• 19:46
  SPEAKER_01: there was a motion to Hey,
• 19:49
  SPEAKER_01: we have all of these amazing developer tools in addition
• 19:51
  SPEAKER_01: to all this amazing production stuff.
• 19:54
  SPEAKER_01: Maybe we should try to externalize the developer tools as well.
• 19:57
  SPEAKER_01: And at the same time, there was also a little bit of Google shifting into Alphabet.
• 20:04
  SPEAKER_01: So for both the developer tooling aspects at one level of like,
• 20:09
  SPEAKER_01: how do we share developer tools amongst different companies within
• 20:13
  SPEAKER_01: the Alphabet umbrella?
• 20:14
  SPEAKER_01: And how do we share high value,
• 20:17
  SPEAKER_01: like internal interfaces and APIs and, you know, tech components?
• 20:21
  SPEAKER_01: How do we share those when it's not one, you know, single corporate, one single company?
• 20:28
  SPEAKER_01: And also not one monorepo, right?
• 20:30
  SPEAKER_01: Increasingly, yeah, exactly.
• 20:33
  SPEAKER_01: And so there was a bunch of work that was kind of the normal, like, okay,
• 20:36
  SPEAKER_01: what does it take to make these developer tools like multi-tenant and like,
• 20:39
  SPEAKER_01: you know, slightly less...
• 20:42
  SPEAKER_01: Completely tied into that ecosystem.
• 20:44
  SPEAKER_00: And just multi-tenant,
• 20:46
  SPEAKER_00: for those who are not as familiar, what does multi-tenant mean inside of Google?
• 20:51
  SPEAKER_01: Well, so this would have been multi-tenant of other companies within
• 20:55
  SPEAKER_01: the Alphabet umbrella, and then maybe
• 20:57
  SPEAKER_00: extend that out to
• 20:57
  SPEAKER_01: cloud customers.
• 20:58
  SPEAKER_00: Oh, got it.
• 20:59
  SPEAKER_00: Yeah, so like a Waymo and the Google could theoretically use it, something like that.
• 21:03
  SPEAKER_01: That became not the hottest topic, and they kind of dropped that idea after a year or two.
• 21:09
  SPEAKER_01: But...
• 21:10
  SPEAKER_01: The low level components,
• 21:11
  SPEAKER_01: like if you wanted to share these things that were written internally in
• 21:15
  SPEAKER_01: the monorepo, what would it take to do that effectively?
• 21:19
  SPEAKER_01: And Abseil was basically, here is the layer of common C++ idiom.
• 21:27
  SPEAKER_01: This is the vernacular that everything at Google is written out of.
• 21:30
  SPEAKER_01: Let's open source that so that there's a common substrate for all Google tech.
• 21:36
  SPEAKER_01: At the same time, the question was, some of this vernacular is gross and tech daddy.
• 21:43
  SPEAKER_01: Should we clean that up before we commit to it forever?
• 21:46
  SPEAKER_01: And the answer was yes, to their great credit.
• 21:48
  SPEAKER_01: So the Abseil open source project is the public like projection of the stuff
• 21:54
  SPEAKER_01: that got cleaned up.
• 21:56
  SPEAKER_01: Most of the work was that was I would defend pretty strongly.
• 22:00
  SPEAKER_01: That was probably the largest refactoring task in human history.
• 22:04
  SPEAKER_00: No way.
• 22:05
  SPEAKER_01: Yeah, we had like 10 or 12 people working full-time with world-class tools
• 22:12
  SPEAKER_01: that no one else understands yet,
• 22:13
  SPEAKER_01: just cleaning up and making sure that idiomatic code externally would be compatible
• 22:19
  SPEAKER_01: with idiomatic code internally.
• 22:21
  SPEAKER_00: And obviously,
• 22:22
  SPEAKER_00: when you were talking about cleaning up,
• 22:23
  SPEAKER_00: you're talking about cleaning up utilities and components that are in the middle
• 22:27
  SPEAKER_00: of some of the highest used services in the world.
• 22:30
  SPEAKER_00: Google Search, Gmail, Maps, you name it.
• 22:35
  SPEAKER_00: If you would break something,
• 22:36
  SPEAKER_00: you would
• 22:36
  SPEAKER_01: probably see at some point an outage in one of those services.
• 22:39
  SPEAKER_01: It was at the time,
• 22:41
  SPEAKER_01: I think it was 200 or 250 million lines of C++ code with 250 million lines of C++?
• 22:48
  SPEAKER_01: First party, not counting external dependencies.
• 22:52
  SPEAKER_01: And Google had that kind of,
• 22:55
  SPEAKER_01: I think that was the public number at the time,
• 22:57
  SPEAKER_00: 10,000 engineers probably at the time,
• 22:59
  SPEAKER_01: making edits to that code base in a month.
• 23:01
  SPEAKER_01: And so our job was, we joked about, that's a New York reference.
• 23:06
  SPEAKER_01: We joked about it as our job is the logical equivalent of Google has
• 23:10
  SPEAKER_01: a new technology for the skyscraper foundations.
• 23:14
  SPEAKER_01: We're replacing the foundation under the Empire State Building.
• 23:17
  SPEAKER_01: We're not allowed to close the building or block traffic.
• 23:20
  SPEAKER_01: And we discovered nuclear waste in the basement.
• 23:22
  SPEAKER_01: It's okay, guys, we got this.
• 23:24
  SPEAKER_01: Like it was, it was a magically, it was a great team.
• 23:27
  SPEAKER_00: I just want to put the 250 million lines of code in perspective because today
• 23:32
  SPEAKER_00: with AI, AI can generate like, you know, we have crazy numbers.
• 23:36
  SPEAKER_00: It can generate tens of thousands of lines of code in an hour, et cetera.
• 23:39
  SPEAKER_00: And you can get to millions if you want to.
• 23:41
  SPEAKER_00: Honestly, just an agent can do it.
• 23:42
  SPEAKER_00: However, pre-AI,
• 23:45
  SPEAKER_00: Uber's mobile application was about a million lines of code on iOS and on Android,
• 23:50
  SPEAKER_00: and that was each written by 200 engineers for years.
• 23:53
  SPEAKER_00: Bunn, I find it rare to find examples of a project that has more than pre-AI,
• 23:58
  SPEAKER_00: more than let's say 3 million lines of code.
• 24:10
  SPEAKER_00: And that will typically be hundreds of engineers writing it for several years.
• 24:13
  SPEAKER_00: So the reason I was like 250 million,
• 24:17
  SPEAKER_00: that is about 100 times more than I've heard for code base.
• 24:21
  SPEAKER_00: And we're talking about like, clearly, this is written for production.
• 24:24
  SPEAKER_00: You're not like just goofing around and pushing some scaffolding here and there and right.
• 24:30
  SPEAKER_00: And when you said first party,
• 24:32
  SPEAKER_00: it's easy to have a lot of code when you take on an open source component.
• 24:36
  SPEAKER_00: But no, no.
• 24:37
  SPEAKER_00: Nope.
• 24:37
  SPEAKER_01: That was all Googler authored.
• 24:39
  SPEAKER_00: Yep.
• 24:40
  SPEAKER_01: And so we have some insight into how things grow.
• 24:46
  SPEAKER_00: Okay.
• 24:46
  SPEAKER_00: So the largest refactoring history at that point in human history might have,
• 24:50
  SPEAKER_00: you're not extricating.
• 24:51
  SPEAKER_00: I don't think so.
• 24:52
  SPEAKER_01: I mean, I can't prove it, but I would be surprised.
• 24:56
  SPEAKER_00: So how, okay.
• 24:57
  SPEAKER_00: When you have this type,
• 24:58
  SPEAKER_00: you were in the middle of this 10 to 12 people,
• 24:59
  SPEAKER_00: you have this massive,
• 25:00
  SPEAKER_00: massive Massive task that seems giant and almost, how do you go about it?
• 25:05
  SPEAKER_00: How do you say like, how do you break it down?
• 25:08
  SPEAKER_00: How do you plan for it?
• 25:10
  SPEAKER_00: How do you speed it up?
• 25:11
  SPEAKER_00: Because you clearly, if you just did it by hand, this would have not fully worked.
• 25:15
  SPEAKER_00: Right.
• 25:15
  SPEAKER_01: So one of the things that had enabled any of this and that came out of having been
• 25:21
  SPEAKER_01: on that low level C++ infrastructure team to start with, Was, okay,
• 25:25
  SPEAKER_01: if we own the interfaces that are kind of the nuts and bolts that everything
• 25:29
  SPEAKER_01: at Google is written out of,
• 25:30
  SPEAKER_01: like way below the database level,
• 25:32
  SPEAKER_01: just like caches,
• 25:33
  SPEAKER_01: containers,
• 25:34
  SPEAKER_01: string manipulation routines, command line flags, like that sort of thing.
• 25:39
  SPEAKER_01: If we own those things, maybe it's valuable to make them better.
• 25:42
  SPEAKER_01: But how do you do that when there's 50,000 call sites or 100,000 edits that need
• 25:47
  SPEAKER_01: to be made to change that API?
• 25:48
  SPEAKER_01: Yeah.
• 25:49
  SPEAKER_01: And so luckily I was friends and org chart cousins eventually with
• 25:54
  SPEAKER_01: the compiler team, especially Clang LLVM people.
• 25:58
  SPEAKER_01: And so they started doing stuff in the AST transformation sort of space.
• 26:02
  SPEAKER_01: AST.
• 26:03
  SPEAKER_01: Abstract syntax tree.
• 26:04
  SPEAKER_01: So like you take a parse tree and, you know, this is Google, like MapReduce.
• 26:09
  SPEAKER_01: So if you do a MapReduce over the code base where the mapper parses all of
• 26:15
  SPEAKER_01: the translation units of your C++,
• 26:17
  SPEAKER_01: like all of the build targets, And produces, here is the parse tree.
• 26:22
  SPEAKER_01: Then you can match the parse tree for specific, like, it's not text match.
• 26:28
  SPEAKER_01: It's, I guarantee you are calling this particular function in this particular context.
• 26:34
  SPEAKER_01: And then you can apply some code to rewrite based on that.
• 26:39
  SPEAKER_01: And so we had compiler assisted,
• 26:41
  SPEAKER_01: you know, semi-automated refactoring to generate all of the text edits.
• 26:46
  SPEAKER_01: Turns out one of the things that was really hard at first was we could generate
• 26:51
  SPEAKER_01: the edit, but we couldn't get the white space in the right places.
• 26:53
  SPEAKER_01: No way.
• 26:54
  SPEAKER_01: Yeah, seriously.
• 26:55
  SPEAKER_01: So Clang format exists because I wished for it.
• 26:59
  SPEAKER_01: This is like the auto formatter for many like C++ sorts of languages at this point.
• 27:05
  SPEAKER_01: But it's like I literally cannot accomplish this refactoring
• 27:09
  SPEAKER_01: until someone figures out a way
• 27:11
  SPEAKER_00: to like hack together a auto white space for C++.
• 27:14
  SPEAKER_01: And so you do that.
• 27:15
  SPEAKER_01: You generate one, you know, big edit.
• 27:18
  SPEAKER_01: It is too large to fit in the codebase because for any given codebase,
• 27:23
  SPEAKER_01: there's kind of a natural background level of how much change is happening.
• 27:27
  SPEAKER_01: And if you're trying to edit 10,000 files,
• 27:30
  SPEAKER_01: 20,000 files,
• 27:31
  SPEAKER_01: it is likely the case that you will never Be able to test and re-sync
• 27:36
  SPEAKER_01: before something has gone out of sync.
• 27:39
  SPEAKER_01: Someone else edited something.
• 27:40
  SPEAKER_01: Yeah, exactly.
• 27:40
  SPEAKER_01: Merge conflicts, parallel changes, that kind of stuff.
• 27:42
  SPEAKER_01: Yeah, exactly.
• 27:44
  SPEAKER_01: You do not want at a 10,000 plus person company to be like,
• 27:48
  SPEAKER_01: everyone stop touching the repo for an hour.
• 27:50
  SPEAKER_01: Let me do this thing.
• 27:51
  SPEAKER_01: That's not going to scale.
• 27:53
  SPEAKER_01: And so the tech is you figure out how to do your refactoring so that the old API and
• 27:59
  SPEAKER_01: the new one can coexist in spelling.
• 28:02
  SPEAKER_01: Yeah.
• 28:03
  SPEAKER_01: And then you parallelize, like just cut by like directory structure.
• 28:08
  SPEAKER_01: Like, all right,
• 28:09
  SPEAKER_01: I'll send you this batch of updates,
• 28:11
  SPEAKER_01: this batch of updates, and you wind up, you know, going like that.
• 28:14
  SPEAKER_01: And then once everything is clean,
• 28:16
  SPEAKER_01: then there's a race to make sure no one copy and pastes the old thing.
• 28:21
  SPEAKER_01: And then you delete the old thing.
• 28:23
  SPEAKER_01: The new thing is the default.
• 28:24
  SPEAKER_01: You move on with life.
• 28:24
  SPEAKER_01: You just do that a few hundred times and your code base is in much better shape.
• 28:28
  SPEAKER_00: Yeah.
• 28:28
  SPEAKER_00: Wow.
• 28:29
  SPEAKER_00: And then you're doing this all the while other engineers, they're just doing their thing.
• 28:33
  SPEAKER_00: They're using these things they don't even know about.
• 28:35
  SPEAKER_00: They don't want to know about you.
• 28:37
  SPEAKER_00: I mean, they might be aware of you,
• 28:38
  SPEAKER_00: but they shouldn't have to change their, as you say, slow down or.
• 28:42
  SPEAKER_00: Yep.
• 28:42
  SPEAKER_01: But we did get amazing feedback on some of this, like.
• 28:46
  SPEAKER_01: At first, people are like, what are you doing?
• 28:48
  SPEAKER_01: Why are you changing my code?
• 28:50
  SPEAKER_01: I'm not okay with this.
• 28:52
  SPEAKER_01: And very quickly, they're like, oh, this is great.
• 28:55
  SPEAKER_01: I go away.
• 28:56
  SPEAKER_01: My code just gets better.
• 28:59
  SPEAKER_01: And I don't need to worry about looking up, is this the right way to do a thing?
• 29:04
  SPEAKER_01: Because if it was the wrong way to do a thing, you would have gotten rid of it already.
• 29:08
  SPEAKER_01: When you have someone that is going through and just janitoring the code base like that.
• 29:12
  SPEAKER_01: Copy and paste becomes the correct answer for how do I do this?
• 29:16
  SPEAKER_01: Like find someone else that did it once,
• 29:18
  SPEAKER_01: you copy it,
• 29:19
  SPEAKER_01: you move on with life and you have evidence that it's probably a reasonable answer.
• 29:24
  SPEAKER_00: Because if it was wrong, we would have fixed it already.
• 29:26
  SPEAKER_00: It's very interesting because today,
• 29:28
  SPEAKER_00: obviously,
• 29:28
  SPEAKER_00: if you did a big refactoring,
• 29:30
  SPEAKER_00: the obvious tool that you would reach for is AI,
• 29:33
  SPEAKER_00: as we've seen with so many companies having lots of success with.
• 29:36
  SPEAKER_00: Well, in the sense that AI is capable of the challenge
• 29:39
  SPEAKER_00: that you had is you had this massive asset code base, rewriting it.
• 29:43
  SPEAKER_00: By hand would have not been feasible,
• 29:44
  SPEAKER_00: but at Google,
• 29:45
  SPEAKER_00: you came up with a very clever semi-automated approach,
• 29:48
  SPEAKER_00: which was based on pretty hardcore language parsing and coming up with new ways
• 29:54
  SPEAKER_00: where you could formally,
• 29:55
  SPEAKER_00: maybe informally,
• 29:56
  SPEAKER_00: or at least prove that this thing will be correct,
• 29:59
  SPEAKER_00: which is what most teams would not have the availability,
• 30:02
  SPEAKER_00: capacity, even expertise to do.
• 30:04
  SPEAKER_00: What everyone has the capability to do right now is like,
• 30:07
  SPEAKER_00: oh, AI, go and do this and do tests and all this.
• 30:11
  SPEAKER_01: Where I wanted
• 30:12
  SPEAKER_00: to get to,
• 30:12
  SPEAKER_00: like rewriting with AI still feels so much more janky than what you did,
• 30:17
  SPEAKER_00: which was so much more deterministic.
• 30:20
  SPEAKER_00: And I wonder if this technique that you used at Google might at some point spread
• 30:27
  SPEAKER_00: or people will just put up with like AI kind of,
• 30:30
  SPEAKER_00: you know, like it might or might not work.
• 30:32
  SPEAKER_01: I think if you're looking for it, it's already out there.
• 30:35
  SPEAKER_01: In the Java space in particular, tools like Open Rewrite, like you basically...
• 30:40
  SPEAKER_01: There's a little configuration per run for it.
• 30:43
  SPEAKER_01: And you say like, this is the pattern I'm looking for in the Java code.
• 30:47
  SPEAKER_01: This is how the rewrite would look.
• 30:49
  SPEAKER_01: And it goes through and blasts through all your Java source files and does that update.
• 30:55
  SPEAKER_01: Turns out AI tools are really,
• 30:57
  SPEAKER_01: really good at generating open rewrite recipes,
• 31:01
  SPEAKER_01: which is way cheaper on token consumption and way more accurate and much faster
• 31:07
  SPEAKER_01: and like auditable.
• 31:09
  SPEAKER_01: It is a substantially better pattern.
• 31:10
  SPEAKER_01: And we've seen talks from,
• 31:14
  SPEAKER_01: I want to say,
• 31:15
  SPEAKER_01: Netflix and Uber,
• 31:16
  SPEAKER_01: I could be wrong,
• 31:18
  SPEAKER_01: specifically on like everyone seems to be coming to the same conclusion
• 31:22
  SPEAKER_01: of what you need is a deterministic tool that declares the intent of the rewrite.
• 31:27
  SPEAKER_01: You can have the AI...
• 31:28
  SPEAKER_01: Write a little skill to train the AI how to create the config for those tools,
• 31:33
  SPEAKER_01: and then you move on with life.
• 31:35
  SPEAKER_01: In the C++ space, actually, some good friends might have a startup right now.
• 31:39
  SPEAKER_01: It's called BrontoSource that is specifically just about that sort
• 31:43
  SPEAKER_01: of declarative rewrite pattern at a very high level of intent,
• 31:48
  SPEAKER_01: very powerful for C++.
• 31:49
  SPEAKER_00: Yeah, I like the idea of using AI to help generate deterministic Software,
• 31:57
  SPEAKER_00: tools, transformations.
• 31:58
  SPEAKER_00: Yeah.
• 32:00
  SPEAKER_01: Kelsey Hightower gave a great talk recently on like zero token architecture
• 32:06
  SPEAKER_01: and like his whole thesis.
• 32:07
  SPEAKER_01: And I think everyone is coming to the same conclusion.
• 32:10
  SPEAKER_01: Don't use AI automatically.
• 32:12
  SPEAKER_01: For constant inference,
• 32:15
  SPEAKER_01: use AI to generate the deterministic tool that will actually do the work and
• 32:19
  SPEAKER_01: then have AI fill in the gaps on the parts of the problem that you can't solve or
• 32:24
  SPEAKER_01: where to invoke the tool, things like that.
• 32:27
  SPEAKER_01: Like if your whole architecture is token consumption for all time,
• 32:32
  SPEAKER_01: it's probably a cheaper and more effective way to do it.
• 32:35
  SPEAKER_00: I just recently heard a head of engineering tell me
• 32:38
  SPEAKER_00: that they're using these AI code review tools,
• 32:41
  SPEAKER_00: which are pretty good, but very expensive and slow.
• 32:46
  SPEAKER_00: And this person said,
• 32:47
  SPEAKER_00: hang on,
• 32:48
  SPEAKER_00: what if we told the AI, okay,
• 32:50
  SPEAKER_00: let's do a thorough code review every now and then,
• 32:52
  SPEAKER_00: but when it finds something, let's have it generate a lint rule.
• 32:56
  SPEAKER_00: Yes, please.
• 32:57
  SPEAKER_00: Which is cheap and And it will no longer take 30 minutes, the cost.
• 33:02
  SPEAKER_00: And also it's just non-deterministic.
• 33:04
  SPEAKER_00: It might or might not find it next time.
• 33:06
  SPEAKER_00: That was frustrating.
• 33:07
  SPEAKER_00: Like this person was like, look, when we know we have certain things.
• 33:10
  SPEAKER_00: So now what they're doing is every now and then they still run some of these reviews.
• 33:14
  SPEAKER_00: And of course, but whenever they find any issues, then they fire up.
• 33:19
  SPEAKER_00: Inference, AI,
• 33:20
  SPEAKER_00: several agents,
• 33:21
  SPEAKER_00: and it's like,
• 33:22
  SPEAKER_00: all right,
• 33:22
  SPEAKER_00: can you come up with an improvement in our deterministic linting that can help it?
• 33:27
  SPEAKER_00: And the inspiration came actually from this person heard about how Uber,
• 33:30
  SPEAKER_00: actually, I think it was, heard me talk about when I joined Yeah.
• 33:33
  SPEAKER_00: Yeah.
• 33:34
  SPEAKER_00: Yeah.
• 33:48
  SPEAKER_00: I think it makes perfect sense.
• 33:49
  SPEAKER_00: Like, let's try to learn our hard lessons only once,
• 33:51
  SPEAKER_00: codify those rules somewhere to Germanistic.
• 34:13
  SPEAKER_01: Apply them equally and appropriately and fairly and leave the interesting parts of code review.
• 34:21
  SPEAKER_01: Make sure that the parts of code review that we're focusing on are only the things
• 34:25
  SPEAKER_01: that humans need to be doing, which was true then.
• 34:28
  SPEAKER_01: I don't think everyone invested in linting and static analysis quite to that extent.
• 34:33
  SPEAKER_01: They should have.
• 34:34
  SPEAKER_01: And then it's equally true now.
• 34:37
  SPEAKER_01: We can't review everything,
• 34:39
  SPEAKER_01: but there are places that require this human attention that is well-spent.
• 34:45
  SPEAKER_00: Titus just made the case for learning the hard lessons once,
• 34:48
  SPEAKER_00: then writing the damals rules and saving the human attention for the things
• 34:51
  SPEAKER_00: that really needed it.
• 34:52
  SPEAKER_00: That is pretty much the same way you want permissions for AI agents to work,
• 34:55
  SPEAKER_00: which brings us to our season sponsor, WorkOS.
• 34:58
  SPEAKER_00: WorkOS provides the auth infrastructure that B2B and AI-native companies need
• 35:02
  SPEAKER_00: to sell to enterprise customers with SSO's Kim Arbak and now Authorization
• 35:06
  SPEAKER_00: for AI Agents.
• 35:07
  SPEAKER_00: WorkOS Airlock is intent-based access control.
• 35:10
  SPEAKER_00: Most access control only asks who you are.
• 35:13
  SPEAKER_00: Airlock also looks at what the agent is trying to do at the moment it tries to do it.
• 35:18
  SPEAKER_00: Clear-cut rules like no agent deletes cloud resources are run as plain code.
• 35:22
  SPEAKER_00: For the fuzzier calls, an AI just evaluates the policy at runtime.
• 35:26
  SPEAKER_00: You write those policies in plain English and Claude can pull
• 35:28
  SPEAKER_00: the ones you already have out of Notion or Gyra and import them for you.
• 35:32
  SPEAKER_00: Your agents,
• 35:32
  SPEAKER_00: whether that's Claude or Devin,
• 35:34
  SPEAKER_00: connect to a single access point and your MCP servers are behind this access point.
• 35:38
  SPEAKER_00: Airlock is a new layer on top of your existing permissions,
• 35:41
  SPEAKER_00: which means that an agent can never get more access than the person who invoked
• 35:44
  SPEAKER_00: the agent.
• 35:45
  SPEAKER_00: When something risky comes up, Airlock can escalate to human for approval, say over Slack.
• 35:49
  SPEAKER_00: It's the same idea Titus described.
• 35:51
  SPEAKER_00: Write the rules once and spend human attention only on the calls that need it.
• 35:56
  SPEAKER_00: You can request early access at workerwas.com slash airlock.
• 35:59
  SPEAKER_00: I also want to mention our presenting sponsor,
• 36:01
  SPEAKER_00: TurboPuffer,
• 36:02
  SPEAKER_00: a ridiculously scalable,
• 36:03
  SPEAKER_00: fast and cheap hybrid search engine built on top of object storage.
• 36:06
  SPEAKER_00: How often do you get to see a rewrite playing out in real time?
• 36:09
  SPEAKER_00: With the Teepuff team, we finally can.
• 36:11
  SPEAKER_00: They are completely redesigning their storage architecture from first principles
• 36:15
  SPEAKER_00: to make search faster, cheaper, and more reliable at scale.
• 36:18
  SPEAKER_00: The new system called TurboPuffer v3 is not yet in prod, but it already passes CI.
• 36:23
  SPEAKER_00: And when the rewrite started in September,
• 36:25
  SPEAKER_00: it was not nearly as performant as the current production version, v2.
• 36:29
  SPEAKER_00: The T5 team had been perf-mapsing query plans until they are fast or faster than v2,
• 36:34
  SPEAKER_00: doing all this in public all the way up to the v3 production launch.
• 36:38
  SPEAKER_00: And it's so cool to see their progress.
• 36:40
  SPEAKER_00: The new system went from being 10 to 100 times slower across most metrics
• 36:44
  SPEAKER_00: to being several times faster on things like cold vector search,
• 36:46
  SPEAKER_00: order by attitude, or hybrid search.
• 36:49
  SPEAKER_00: The V3 system is still slower in areas like full text search,
• 36:52
  SPEAKER_00: but only one to two times slower versus a hundred times slower
• 36:56
  SPEAKER_00: when this perf maxing started.
• 36:58
  SPEAKER_00: And the TPF team is sharing exactly how they optimize things
• 37:01
  SPEAKER_00: in their development logs with things like group count aggregation,
• 37:04
  SPEAKER_00: compressed column encodings, batch scoring, and many more.
• 37:08
  SPEAKER_00: A lot of these wins are basic foundational work right now,
• 37:11
  SPEAKER_00: but the team will be sharing some very cool systems work over the next few weeks.
• 37:14
  SPEAKER_00: But hang on, why is the team doing a full rewrite?
• 37:18
  SPEAKER_00: It's because search is changing, and so is TurboBuffer.
• 37:20
  SPEAKER_00: TPUF started as a vector database, but it is no longer a vector database.
• 37:25
  SPEAKER_00: It has evolved into a high-performance database for search.
• 37:27
  SPEAKER_00: The V3 discards the vector primary index,
• 37:29
  SPEAKER_00: and all indexes will share the same columnar storage system.
• 37:33
  SPEAKER_00: Every query will run on vectorized query engines,
• 37:35
  SPEAKER_00: which is something you don't usually see in search databases,
• 37:39
  SPEAKER_00: but which will result in both very fast search and just generally
• 37:42
  SPEAKER_00: a very fast database.
• 37:43
  SPEAKER_00: The TurboPuffer team is keeping a work log of their journey and you can follow along
• 37:47
  SPEAKER_00: at turbopuffer.com.
• 37:48
  SPEAKER_01: It's a blast to see how it's being built.
• 37:51
  SPEAKER_00: Speaking of hard,
• 37:52
  SPEAKER_00: hard lessons and documenting them,
• 37:54
  SPEAKER_00: can we talk about Software Engineering Google,
• 37:55
  SPEAKER_00: the book that you're quite well known for?
• 37:58
  SPEAKER_00: How did the idea come?
• 37:59
  SPEAKER_00: How did it come together?
• 38:01
  SPEAKER_00: What was your motivation to start with?
• 38:03
  SPEAKER_01: So this was a little bit the tail end of, hey, maybe we'll ship the developer tools.
• 38:10
  SPEAKER_01: Maybe we'll need to explain why Google's developer tools are the way they are.
• 38:15
  SPEAKER_01: Because otherwise it'll be a little bit of like aliens came down and you're trying
• 38:18
  SPEAKER_01: to give you something that doesn't make any sense in your world.
• 38:20
  SPEAKER_01: Part of the reason that we had the hubris to like tackle that sort of task was
• 38:25
  SPEAKER_01: because we had been doing this refactoring at scale, you know, everywhere.
• 38:30
  SPEAKER_01: And when you...
• 38:32
  SPEAKER_01: Run into everything that any engineer thought was a clever solution to their problem.
• 38:37
  SPEAKER_01: It meant that my cadre of weirdos have run into so many,
• 38:44
  SPEAKER_01: oh, yep, I can see how you thought that was a good idea.
• 38:48
  SPEAKER_01: Don't do that.
• 38:50
  SPEAKER_01: Like more than anyone else out there.
• 38:53
  SPEAKER_00: You're talking some of the most kind of battle-scarred people in the industry,
• 38:58
  SPEAKER_00: just because you're working on these large-scale systems for such a long period
• 39:02
  SPEAKER_00: of time.
• 39:03
  SPEAKER_01: And it's not the production battle scars, it's the like...
• 39:06
  SPEAKER_01: Testing and local code design battle scars.
• 39:10
  SPEAKER_01: Everyone else in the industry has all this production experience and my job ends at Merge.
• 39:16
  SPEAKER_01: I know more about development and API design and test and keep the code base as
• 39:23
  SPEAKER_01: an artifact working than most, but it does mean...
• 39:28
  SPEAKER_01: I'm kind of an outsider in that sense.
• 39:30
  SPEAKER_00: Well, interesting enough, I mean,
• 39:31
  SPEAKER_00: the book is very,
• 39:32
  SPEAKER_00: very thorough on how it gets to production,
• 39:35
  SPEAKER_00: but it does not cover the operational part, which I guess is the SRE book.
• 39:39
  SPEAKER_00: But there is an SRE book, which, well, then this book would be like twice as long.
• 39:45
  SPEAKER_00: How did you pull the book together?
• 39:47
  SPEAKER_00: Because you wrote some parts of it.
• 39:50
  SPEAKER_00: Other folks inside of Google wrote different chapters of it.
• 39:53
  SPEAKER_01: So it's very much in the spirit of the SRE books.
• 39:56
  SPEAKER_01: Yeah.
• 39:57
  SPEAKER_01: Like try to get the expert on or an expert on Gavin Topics to contribute something useful.
• 40:04
  SPEAKER_01: And then the three names on the front of the book, myself, Hiram Wright, and Tom Mantrek.
• 40:08
  SPEAKER_01: Hiram and I were kind of the technical leadership for that.
• 40:11
  SPEAKER_01: And Tom is plenty technical, but has been our tech writer friend for many, many years.
• 40:18
  SPEAKER_01: So he handled a lot of the like editorial, structural, organizational stuff.
• 40:22
  SPEAKER_01: But Hiram and I had been the tag team of...
• 40:26
  SPEAKER_01: Hiram generated millions of edits and sent them to me for review,
• 40:32
  SPEAKER_01: and I had the permissions to review almost everything in the codebase.
• 40:36
  SPEAKER_01: And so everyone knew us as those two guys that clean everything up.
• 40:41
  SPEAKER_01: So we had those scars, and so we led that.
• 40:44
  SPEAKER_01: And then the trick was finding the experts to write something high quality.
• 40:48
  SPEAKER_01: Some of those were great from the start.
• 40:52
  SPEAKER_01: A lot of that took a bit of ghostwriting stuff.
• 40:56
  SPEAKER_01: The whole task was actually almost two years.
• 41:00
  SPEAKER_01: Certainly not full-time.
• 41:01
  SPEAKER_01: We were at most probably putting in a quarter to half-time most of that time.
• 41:07
  SPEAKER_01: But yeah, we wrangled together way too much text from way too many people.
• 41:14
  SPEAKER_01: I don't think I realized just how much we overshot until I saw a printed copy of that book.
• 41:19
  SPEAKER_01: I was like, wow, that's huge.
• 41:21
  SPEAKER_00: Yeah, and even today,
• 41:23
  SPEAKER_00: I do recommend the book because it gives a really good understanding of how
• 41:28
  SPEAKER_00: the Google at the time worked.
• 41:29
  SPEAKER_00: Obviously,
• 41:30
  SPEAKER_00: there's been evolving since,
• 41:31
  SPEAKER_00: but how a company going from zero to building important production systems came up
• 41:36
  SPEAKER_00: with a bunch of stuff that might not make as much sense.
• 41:39
  SPEAKER_00: And you talk about things like Like monorepo,
• 41:42
  SPEAKER_00: like the stringent code review process, how there's a readability process at Google.
• 41:48
  SPEAKER_00: And maybe we can talk about that.
• 41:50
  SPEAKER_00: Google is the only company I know where for code review back in the day,
• 41:54
  SPEAKER_00: they didn't just have code review like an engineer who owns this part of the system
• 41:57
  SPEAKER_00: and understands, reviews it and says it's okay.
• 42:00
  SPEAKER_00: But Google had a readability review.
• 42:02
  SPEAKER_00: What is readability review?
• 42:04
  SPEAKER_01: So for any change that's going in,
• 42:08
  SPEAKER_01: Someone that is involved,
• 42:09
  SPEAKER_01: either the author or the reviewer or author or a reviewer,
• 42:13
  SPEAKER_01: has to have two different permission bits.
• 42:17
  SPEAKER_01: First is that ownership thing.
• 42:18
  SPEAKER_01: So someone has to be in the list of people that is authorized to change these files.
• 42:23
  SPEAKER_01: And the second is someone has to actually be certified in,
• 42:28
  SPEAKER_01: we believe you understand how to write idiomatic code in this code base in
• 42:32
  SPEAKER_01: that language.
• 42:32
  SPEAKER_01: And this is a step towards trying to make sure that everything is relatively consistent.
• 42:37
  SPEAKER_01: And that there's common patterns that you're not like reinventing the wheel because
• 42:43
  SPEAKER_01: the strategic value of consistency is incredible.
• 42:46
  SPEAKER_01: And so the readability process is someone has to be certified for that.
• 42:50
  SPEAKER_01: How do you get certified for that?
• 42:52
  SPEAKER_01: And initially that came as employee number three sat down with every new employee
• 42:59
  SPEAKER_01: and walked them through their first change.
• 43:01
  SPEAKER_01: That stopped scaling.
• 43:02
  SPEAKER_01: So they got a bunch of other people to try to proxy Bing Craig.
• 43:07
  SPEAKER_01: And they walk you through your first big substantive change.
• 43:11
  SPEAKER_01: And over time,
• 43:12
  SPEAKER_01: it got to the point where most change was not like I am writing a new component
• 43:17
  SPEAKER_01: from scratch.
• 43:18
  SPEAKER_01: It's an existing product,
• 43:20
  SPEAKER_01: like you're making little edits here and there, you're adding a feature or whatever.
• 43:23
  SPEAKER_01: So finding something that was meaty enough to like demonstrate in one go
• 43:27
  SPEAKER_01: that you knew everything was kind of hard.
• 43:30
  SPEAKER_01: And the whole thing felt a little bit more like a hazing ritual.
• 43:33
  SPEAKER_01: And so over the years, I think the people in Java started it.
• 43:39
  SPEAKER_01: It moved from that sort of one shot to being explicitly kind of a mentoring program
• 43:44
  SPEAKER_01: of you do your normal work.
• 43:47
  SPEAKER_01: And after your team has signed off on it,
• 43:49
  SPEAKER_01: then we will round robin assign someone from the pool of like Java mentors to go
• 43:55
  SPEAKER_01: through and see like.
• 43:56
  SPEAKER_01: Is there anything like substantively like interesting worth commenting on here
• 44:00
  SPEAKER_01: to kind of up the quality of the code and make sure that it matches the rest of
• 44:04
  SPEAKER_01: the norms?
• 44:04
  SPEAKER_01: And you might notice that means if you're on a team
• 44:08
  SPEAKER_01: that already has great standards, there's probably very little to comment on.
• 44:12
  SPEAKER_01: Right.
• 44:12
  SPEAKER_01: And you'll probably graduate pretty quick.
• 44:15
  SPEAKER_01: And if you're not and you choose not to learn any of the lessons,
• 44:18
  SPEAKER_01: you will probably graduate very slowly,
• 44:21
  SPEAKER_00: if at all.
• 44:22
  SPEAKER_01: And so the whole thing kind of grew into that.
• 44:25
  SPEAKER_01: And we extended that sort of mentorship approach into all the other readability languages.
• 44:31
  SPEAKER_01: And that's been the norm for a bunch of years.
• 44:33
  SPEAKER_01: I haven't checked up on it since,
• 44:35
  SPEAKER_00: you know, for at least a couple of years.
• 44:36
  SPEAKER_00: But it's very interesting because it feels to me like this turns code review into...
• 44:41
  SPEAKER_00: Into just education in the depth of a programming language.
• 44:45
  SPEAKER_00: Yep.
• 44:45
  SPEAKER_01: Was this by design or was it just a byproduct?
• 44:47
  SPEAKER_01: I think it's absolutely by design.
• 44:49
  SPEAKER_01: Certainly at the parts of it where I was pulling any of the strings was definitely by design.
• 44:54
  SPEAKER_01: But yeah,
• 44:54
  SPEAKER_01: when I left Google,
• 44:56
  SPEAKER_01: you know,
• 44:57
  SPEAKER_01: almost three years ago,
• 44:58
  SPEAKER_01: readability was like I was kind of the overarching owner for
• 45:02
  SPEAKER_01: the readability program.
• 45:04
  SPEAKER_01: I would argue that might be the largest educational program in tech or maybe in
• 45:08
  SPEAKER_01: the world because there were Tens of thousands of people participating in that,
• 45:13
  SPEAKER_00: like at any given moment.
• 45:14
  SPEAKER_00: And then you just participated by joining Google and then you had no choice.
• 45:18
  SPEAKER_00: It wasn't like voluntary.
• 45:20
  SPEAKER_00: Right.
• 45:20
  SPEAKER_01: And so certainly some people were super pissy about it.
• 45:23
  SPEAKER_00: But I guess there was also a goal where it wasn't just about like, OK,
• 45:26
  SPEAKER_00: so step one is you need to address the comments that a feasibility expert puts
• 45:31
  SPEAKER_00: on your.
• 45:31
  SPEAKER_00: Step two,
• 45:32
  SPEAKER_00: you probably say like,
• 45:33
  SPEAKER_00: I'd like to get fewer comments on them if you have any sense of pride,
• 45:37
  SPEAKER_00: because it's kind of embarrassing.
• 45:38
  SPEAKER_00: And then step three is like, you know what?
• 45:41
  SPEAKER_00: Okay, I now get zero comments.
• 45:42
  SPEAKER_00: I actually want to become a readability expert myself.
• 45:45
  SPEAKER_00: And then that's a different level.
• 45:47
  SPEAKER_00: And then step four is now You are giving feedback,
• 45:51
  SPEAKER_00: and I guess due to this feedback,
• 45:52
  SPEAKER_00: maybe you meet people,
• 45:53
  SPEAKER_00: maybe you start to form some, I don't know, mentorship, friendships, who knows.
• 45:57
  SPEAKER_01: Yeah, and you see other parts of the code base,
• 45:59
  SPEAKER_01: you see stuff that other people are doing,
• 46:01
  SPEAKER_01: you learn about other tech that might be happening.
• 46:04
  SPEAKER_01: It was a huge, subtle, below-the-waterline information-sharing program.
• 46:10
  SPEAKER_00: In the book, you write about the importance of a team, and I'll quote you.
• 46:14
  SPEAKER_00: In the realm of programming,
• 46:16
  SPEAKER_00: lone craftspeople are extremely rare,
• 46:17
  SPEAKER_00: and even when they do exist, they don't perform superhuman achievements in a vacuum.
• 46:21
  SPEAKER_00: Their world-changing accomplishment is almost always the result of a spark
• 46:24
  SPEAKER_00: of inspiration followed by a heroic team effort.
• 46:27
  SPEAKER_00: A great team makes brilliant use of its superstars,
• 46:30
  SPEAKER_00: but the whole is always greater than the sum of the parts.
• 46:33
  SPEAKER_00: But creating a superstar team is fiendishly difficult.
• 46:37
  SPEAKER_00: Yeah.
• 46:38
  SPEAKER_00: I don't think I wrote that chapter.
• 46:40
  SPEAKER_00: It's in the book that you are a co-author of, so...
• 46:43
  SPEAKER_00: I wanted to reflect on this,
• 46:46
  SPEAKER_00: especially with AI,
• 46:47
  SPEAKER_00: because now I'm starting to hear a lot more people say,
• 46:51
  SPEAKER_00: yeah, in the past we talked about a 10x engineer up to one, up to 22, 23.
• 46:56
  SPEAKER_00: Don't get me started on that.
• 46:58
  SPEAKER_00: So I just wanted to
• 47:15
  SPEAKER_00: Ask you how the concept of the team holds up as we have more AI tools and
• 47:21
  SPEAKER_00: the concept of a standout team.
• 47:23
  SPEAKER_01: Yeah.
• 47:23
  SPEAKER_01: So I think regardless of where you fit on the spectrum of like AI believerism,
• 47:30
  SPEAKER_01: I think the team is still critical.
• 47:32
  SPEAKER_01: Like if you believe that the agents are the only answer
• 47:36
  SPEAKER_01: in another direction forward, We're still managing that team of agents.
• 47:41
  SPEAKER_01: And I've heard an increasing number of voices talking about it as
• 47:45
  SPEAKER_01: the problem right now is we need to make every engineer into an engineering manager
• 47:52
  SPEAKER_01: for a team of agents.
• 47:54
  SPEAKER_01: And we don't have the training or skill set or background to have done that.
• 47:57
  SPEAKER_01: Like we're asking them to do a thing with no scaffolding and guidance.
• 48:01
  SPEAKER_01: The alternative is you think of the AI and the agents as your tools,
• 48:07
  SPEAKER_01: your IDE, but it is still you in conjunction with the team of humans around you.
• 48:13
  SPEAKER_01: It's still team behavior.
• 48:16
  SPEAKER_01: I think there are certainly differences in the social dynamics and things like that
• 48:21
  SPEAKER_01: between you and the agents.
• 48:23
  SPEAKER_01: Which I'm going to go out on a limb and say the agents don't have feelings.
• 48:27
  SPEAKER_01: They definitely do not.
• 48:28
  SPEAKER_01: Right?
• 48:29
  SPEAKER_01: Should not.
• 48:31
  SPEAKER_01: Let's hold that line for a bit.
• 48:33
  SPEAKER_01: But yeah,
• 48:35
  SPEAKER_01: like I think understanding clear communication and like being able
• 48:41
  SPEAKER_01: to write things down and get it out of your head is still,
• 48:44
  SPEAKER_01: you know, a substantial important thing.
• 48:46
  SPEAKER_01: Like I...
• 48:48
  SPEAKER_01: Yeah, I think it is still unquestionably is still teams at the root of it.
• 48:53
  SPEAKER_01: Software engineering is a team sport, period.
• 48:55
  SPEAKER_00: Also, I guess just one like pretty like naive way to just prove
• 49:00
  SPEAKER_00: that this is probably still true is if we just look at some
• 49:03
  SPEAKER_00: the leading AI companies, the leading AI labs, the Entropics, the Open AIs.
• 49:07
  SPEAKER_00: When I talk to people there.
• 49:10
  SPEAKER_00: Yeah.
• 49:27
  SPEAKER_00: And that team, when I met them, you can tell they gel.
• 49:30
  SPEAKER_00: They work together.
• 49:31
  SPEAKER_00: They trust each other.
• 49:32
  SPEAKER_00: At some point, Boris will be on holiday and you will not know because things just happen.
• 49:38
  SPEAKER_01: You said the important keyword, trust.
• 49:42
  SPEAKER_01: A substantial part of this is the psychological safety aspect of it.
• 49:46
  SPEAKER_01: The other thing,
• 49:47
  SPEAKER_01: which I think is natural for the AI labs,
• 49:49
  SPEAKER_01: but hasn't been nearly as dominant for the rest of the industry yet,
• 49:53
  SPEAKER_01: Is you're learning as you go.
• 49:56
  SPEAKER_01: And if you look at all of the research,
• 49:58
  SPEAKER_01: you know,
• 49:59
  SPEAKER_01: like Dora research on stuff or any research on productivity or team efficacy or any
• 50:05
  SPEAKER_01: of those things,
• 50:07
  SPEAKER_01: like at root,
• 50:07
  SPEAKER_01: it always boils down to culture,
• 50:10
  SPEAKER_01: psychological safety and the the permission to learn.
• 50:15
  SPEAKER_01: It's the permission to admit that I don't know something.
• 50:18
  SPEAKER_01: It is the incentive to learn from each other.
• 50:21
  SPEAKER_01: It is the incentive to share your mistakes so that the team only has to do that once.
• 50:26
  SPEAKER_01: Your lint rules or SRE postmortem culture,
• 50:29
  SPEAKER_00: every effective structure
• 50:32
  SPEAKER_01: and organization in tech.
• 50:33
  SPEAKER_01: Is actually about information flow and learning.
• 50:37
  SPEAKER_01: And I think a really great perspective on that.
• 50:41
  SPEAKER_01: I love Kat Hicks, the psychology researcher for tech.
• 50:45
  SPEAKER_01: Like her book, The Psychology of Software Teams is beautiful.
• 50:49
  SPEAKER_01: And like she talks about, you know, these things a lot, right?
• 50:53
  SPEAKER_01: Similarly,
• 50:54
  SPEAKER_01: there's a platform engineering at Google book that some
• 50:56
  SPEAKER_01: of my friends are putting out as early release chapters on O'Reilly.
• 50:59
  SPEAKER_01: I might have written the chapter on education for that.
• 51:04
  SPEAKER_01: Very surprisingly slash not surprisingly.
• 51:06
  SPEAKER_01: Right.
• 51:06
  SPEAKER_01: But yeah,
• 51:07
  SPEAKER_01: like education is the powerful route for like,
• 51:12
  SPEAKER_01: let's make this mistake a minimum number of times.
• 51:14
  SPEAKER_01: Let's codify what we've done.
• 51:16
  SPEAKER_01: Let's give people permission to admit they don't know and to grow because it's not
• 51:23
  SPEAKER_01: a one-off game.
• 51:24
  SPEAKER_01: It's an iterated game.
• 51:26
  SPEAKER_01: And you'll do better over time if your team is better next time.
• 51:30
  SPEAKER_00: I want to go back to psychological safety because back about five,
• 51:34
  SPEAKER_00: 10 years ago,
• 51:34
  SPEAKER_00: we kind of talked so much about it in enduring leadership side circles.
• 51:38
  SPEAKER_00: This was pretty well understood that, you know, A good team feels a safe team.
• 51:43
  SPEAKER_00: If you, for example,
• 51:43
  SPEAKER_00: make a mistake,
• 51:44
  SPEAKER_00: you cause an outage as an engineer,
• 51:46
  SPEAKER_00: like literally you push a code and that code causes outage.
• 51:49
  SPEAKER_00: We should have a blameless postmortem where we don't say that, hey,
• 51:52
  SPEAKER_00: Gergay pushed this code and he caused the outage because that's now,
• 51:55
  SPEAKER_00: I mean, I feel terrible.
• 51:56
  SPEAKER_00: But let's talk about, for example, how could have we avoided this at a system level?
• 52:02
  SPEAKER_00: Why did a test not fire?
• 52:03
  SPEAKER_00: Why was it rolled out immediately when he was a new joiner on the team or she was a new joiner?
• 52:08
  SPEAKER_00: Maybe some onboarding could have helped, yada, yada, yada.
• 52:11
  SPEAKER_00: How do you define psychological safety and why is it important?
• 52:15
  SPEAKER_00: Especially just playing devil's advocate.
• 52:17
  SPEAKER_00: Like when we're talking about smart engineers who know stuff,
• 52:19
  SPEAKER_00: like, is it all that important to us?
• 52:21
  SPEAKER_00: Because we should know our thing, right?
• 52:23
  SPEAKER_00: I'm just like, I'm kind of removing my emotional hat here.
• 52:27
  SPEAKER_01: Yeah, we should know our thing, sure.
• 52:29
  SPEAKER_01: So I would say,
• 52:31
  SPEAKER_01: and it's been a little hard to like, We could give you a 15 question, true, false.
• 52:58
  SPEAKER_01: Or maybe Likert scale,
• 53:00
  SPEAKER_01: but like a 15 question assessment for the team and predict 30% of whether
• 53:05
  SPEAKER_01: or not you will hit whatever your goals are.
• 53:07
  SPEAKER_01: That's insanity.
• 53:09
  SPEAKER_01: That is more powerful than any other signal.
• 53:12
  SPEAKER_01: Like, okay,
• 53:14
  SPEAKER_01: so you think that we're all engineers and this is all just computers and tech
• 53:18
  SPEAKER_01: and emotions have nothing to do with it.
• 53:20
  SPEAKER_01: Find me anything that's even remotely that powerful predictor of success.
• 53:25
  SPEAKER_01: It's a people problem, like period.
• 53:28
  SPEAKER_01: Right.
• 53:29
  SPEAKER_01: And going back to.
• 53:31
  SPEAKER_01: Right.
• 53:31
  SPEAKER_01: It's it's not a single shot game.
• 53:33
  SPEAKER_01: It's an iterated game.
• 53:34
  SPEAKER_01: I regularly cite Fred Brooks and the Mythical Man month essays.
• 53:37
  SPEAKER_01: Right.
• 53:38
  SPEAKER_01: And in the surgical team,
• 53:39
  SPEAKER_01: like the surgical team is my least favorite chapter in that book
• 53:43
  SPEAKER_01: because it's structured as if.
• 53:45
  SPEAKER_01: We're only ever going to do one project.
• 53:47
  SPEAKER_01: Get the best person to do this.
• 53:50
  SPEAKER_01: Have everyone else build the tools and do the documentation.
• 53:53
  SPEAKER_01: It's very hierarchical.
• 53:54
  SPEAKER_01: Right.
• 53:54
  SPEAKER_01: And it's very like power dynamics mad.
• 53:57
  SPEAKER_01: Yeah.
• 53:57
  SPEAKER_01: Right.
• 53:58
  SPEAKER_01: But it might make sense if you're only ever going to do one task.
• 54:03
  SPEAKER_01: Instead, we're playing like in a game theory sense.
• 54:06
  SPEAKER_01: It's not prisoner's dilemma.
• 54:08
  SPEAKER_01: It's iterated prisoner's dilemma.
• 54:09
  SPEAKER_01: There's a different strategy.
• 54:11
  SPEAKER_01: And the strategy is find the most junior person that can do it with supervision so
• 54:16
  SPEAKER_01: that they learn and grow and you still get a decent outcome.
• 54:20
  SPEAKER_01: And that next,
• 54:21
  SPEAKER_01: like the higher level people can solve something that they don't know how to do yet
• 54:25
  SPEAKER_01: or that no one else can tackle.
• 54:28
  SPEAKER_01: This is the only way to structure it so that growth happens.
• 54:32
  SPEAKER_01: But in order for that growth to happen,
• 54:34
  SPEAKER_01: people have to be willing to admit that they don't know a thing.
• 54:39
  SPEAKER_01: And that brings you straight back to psychological safety and learning culture.
• 54:44
  SPEAKER_01: If it is a culture that highlights information sharing, learning, and growth, you will succeed.
• 54:52
  SPEAKER_00: And you mentioned that the AI labs probably take this for granted.
• 54:56
  SPEAKER_00: Because they don't know what they're like.
• 54:57
  SPEAKER_00: Well, I'm going to paraphrase it.
• 54:59
  SPEAKER_00: With the AI labs, there are a few things going.
• 55:00
  SPEAKER_00: First of all, business is booming.
• 55:02
  SPEAKER_00: They are growing.
• 55:02
  SPEAKER_00: Revenue goes up.
• 55:04
  SPEAKER_00: Customers goes up.
• 55:05
  SPEAKER_00: Funding goes up.
• 55:06
  SPEAKER_00: Headcount goes up.
• 55:07
  SPEAKER_00: Their teams are only ever expanding.
• 55:08
  SPEAKER_01: They're in hypergrowth.
• 55:10
  SPEAKER_00: We've seen companies like Netflix and Uber in hypergrowth.
• 55:13
  SPEAKER_00: This is even faster.
• 55:14
  SPEAKER_00: Your equity also goes up.
• 55:15
  SPEAKER_00: So you don't need to worry about things like performance reviews
• 55:18
  SPEAKER_00: because performance reviews are typically important
• 55:20
  SPEAKER_00: where like how much bonus I'm going to But when you're at a place
• 55:24
  SPEAKER_00: that goes like this and your equity goes up,
• 55:26
  SPEAKER_00: your financial part is taken care of just by focusing on helping
• 55:30
  SPEAKER_00: the company succeed regardless of your titles.
• 55:32
  SPEAKER_00: Second of all,
• 55:33
  SPEAKER_00: because they're building new technology and they know no one knows,
• 55:36
  SPEAKER_00: so they just tell Okay, I don't know.
• 55:37
  SPEAKER_00: I'm learning from you.
• 55:38
  SPEAKER_00: Here's what I learned.
• 55:40
  SPEAKER_00: So it's just all natural.
• 55:41
  SPEAKER_00: Whereas compared to a company that might not grow like this,
• 55:44
  SPEAKER_00: they might still be using, let's say, AI.
• 55:47
  SPEAKER_00: But now there's less of an incentive to say,
• 55:48
  SPEAKER_00: I don't know, because there's a performance review coming up.
• 55:51
  SPEAKER_00: You want to look like this.
• 55:52
  SPEAKER_00: You don't want people to say like, well, this person says, I don't know a lot.
• 55:56
  SPEAKER_00: Your bonus will be lower.
• 55:57
  SPEAKER_00: You might be put on a performance improvement plan because you're now at a company
• 56:00
  SPEAKER_00: that every year lets go of 6% of people.
• 56:04
  SPEAKER_00: And your goal is suddenly going from like, I just don't want to be there.
• 56:08
  SPEAKER_00: I want to hang on to this job, but it's now hanging on.
• 56:13
  SPEAKER_00: And you also,
• 56:14
  SPEAKER_00: you're now very of maybe your teammates because now there's an internal competition.
• 56:19
  SPEAKER_00: You want to Hog that project.
• 56:23
  SPEAKER_00: And so this
• 56:23
  SPEAKER_01: is a huge difference.
• 56:25
  SPEAKER_01: Yeah, it's huge.
• 56:26
  SPEAKER_01: Like if you get into the mode of believing that the team outcomes are what matter
• 56:32
  SPEAKER_01: for your own success,
• 56:34
  SPEAKER_01: then it is much easier for you to realize like,
• 56:37
  SPEAKER_01: okay, I don't have to have this one shiny task.
• 56:41
  SPEAKER_01: Like the team will be successful by virtue of someone doing this.
• 56:45
  SPEAKER_01: And it might be my turn next.
• 56:47
  SPEAKER_01: If when it breaks down into that like individualism and like,
• 56:51
  SPEAKER_01: I don't trust the guy across the way from me or,
• 56:54
  SPEAKER_01: you know,
• 56:54
  SPEAKER_01: right shoulder to me,
• 56:56
  SPEAKER_01: like even though we're on the same team,
• 56:57
  SPEAKER_01: we're scraping by and like, you know, fighting for attention and success.
• 57:04
  SPEAKER_01: And that's not a recipe for innovation and growth.
• 57:06
  SPEAKER_01: Like that is a recipe for stagnation and just bad.
• 57:11
  SPEAKER_00: One of the obvious things that I assume does not do good for psychological growth
• 57:17
  SPEAKER_00: or is terrible for it is layoffs.
• 57:20
  SPEAKER_00: Can you tell me, based on your observation, and we've seen, right?
• 57:23
  SPEAKER_00: We've all been part of it in 2022, 2023.
• 57:26
  SPEAKER_00: There has been a series of layoffs.
• 57:28
  SPEAKER_00: Right now, companies are doing it again.
• 57:30
  SPEAKER_00: Back then,
• 57:31
  SPEAKER_00: it was to do with interest rates and overhiring during zero interest rates
• 57:37
  SPEAKER_00: and growth kind of going down.
• 57:38
  SPEAKER_00: Now, some companies are actually openly saying that it's AI.
• 57:42
  SPEAKER_00: Some are not as openly,
• 57:43
  SPEAKER_00: but we find out that Meta,
• 57:44
  SPEAKER_00: for example,
• 57:45
  SPEAKER_00: wanted to reduce scene ties because they thought that AI would be there.
• 57:49
  SPEAKER_00: From your observations,
• 57:50
  SPEAKER_00: at a company where a layoff happens,
• 57:55
  SPEAKER_00: How does it change how software engineers show up,
• 57:58
  SPEAKER_00: work, think, speaking both about your experience, the people you talk with?
• 58:03
  SPEAKER_01: I mean, it's stressful.
• 58:05
  SPEAKER_01: I've had experiences of there being layoffs in the area and just it'll be weeks
• 58:13
  SPEAKER_01: or months of every one-on-one starts with, so how are you doing?
• 58:18
  SPEAKER_01: How are you coping?
• 58:20
  SPEAKER_01: There's this huge amount of Negative performance influence of like, okay,
• 58:27
  SPEAKER_01: we got to like emotionally support each other first before we actually even get back
• 58:32
  SPEAKER_01: to doing like the work.
• 58:35
  SPEAKER_01: And I'm very skeptical that in the executive accounting on all of this,
• 58:40
  SPEAKER_01: that they're taking into consideration just the sheer amount of lost productivity
• 58:46
  SPEAKER_01: and lost time from like, yeah, People are stressed.
• 58:50
  SPEAKER_01: People are spending time brushing up their resume in case they're next on the chopping block.
• 58:55
  SPEAKER_01: People are spending that one-on-one time like in therapy instead of in planning.
• 59:00
  SPEAKER_01: It is a huge hidden cost to all of that and it's really demotivating.
• 59:08
  SPEAKER_01: It drives home like it is a Reminder that, okay, this is just a business obligation.
• 59:15
  SPEAKER_01: This is not a lifestyle choice.
• 59:18
  SPEAKER_01: And if me putting in extra doesn't give me extra safety on this, Then what's the point?
• 59:26
  SPEAKER_01: There's a lot of cultural baggage that happens for that.
• 59:30
  SPEAKER_01: And it's one of the things that I actually really liked about Adobe was Adobe's had
• 59:34
  SPEAKER_01: the same CEO for a long time.
• 59:36
  SPEAKER_01: He just stepped down.
• 59:37
  SPEAKER_01: He has a replacement now.
• 59:39
  SPEAKER_01: But he was there for the Great Recession crash.
• 59:43
  SPEAKER_01: And they did some layoffs at that era.
• 59:45
  SPEAKER_01: And he saw how much it hurt morale and company productivity and all of the rest of that.
• 59:50
  SPEAKER_01: And And he learned like it.
• 59:53
  SPEAKER_01: We thought it made numeric sense at the time, but it was not good for the soul of the company.
• 59:59
  SPEAKER_01: And that's not the company that we want to be.
• 1:00:01
  SPEAKER_01: And so Adobe didn't have the layoffs in the last few years.
• 1:00:04
  SPEAKER_01: Like there's.
• 1:00:05
  SPEAKER_01: Group by group, like local things as things get reprioritized, but it's not this sweeping.
• 1:00:10
  SPEAKER_00: Yeah, we've seen that most of big tech has had it, especially in 2023.
• 1:00:14
  SPEAKER_00: Yeah.
• 1:00:15
  SPEAKER_00: Speaking of Adobe, so you've moved to Adobe a few years ago.
• 1:00:20
  SPEAKER_01: What are you focused on at Adobe?
• 1:00:22
  SPEAKER_01: So there I'm very senior, I see focused on developer experience.
• 1:00:28
  SPEAKER_01: I happen to say this is a polite fiction.
• 1:00:30
  SPEAKER_01: Developer experience is the business of productivity.
• 1:00:33
  SPEAKER_01: We just know enough not to say that we can measure productivity.
• 1:00:36
  SPEAKER_01: It does not give everyone...
• 1:00:37
  SPEAKER_00: Unlike McKinsey, who me and Kent have bonded over.
• 1:00:40
  SPEAKER_01: Yeah, that's horrible.
• 1:00:45
  SPEAKER_01: But we will link it in the show notes below.
• 1:00:47
  SPEAKER_01: Yeah.
• 1:00:48
  SPEAKER_01: So it's what do we need for tools and culture and all of the things to improve
• 1:00:55
  SPEAKER_01: the experience of being a developer and make you more productive and efficient?
• 1:01:00
  SPEAKER_01: Most companies spend more on engineering comp than anything else.
• 1:01:04
  SPEAKER_00: So today you've been focused on thinking about it and working on it for both
• 1:01:07
  SPEAKER_00: at Adobe, but also to some extent at Google.
• 1:01:10
  SPEAKER_00: What do you know today about developer productivity?
• 1:01:14
  SPEAKER_00: How does this relate to AI, coding tool, output, metrics, all of that things?
• 1:01:19
  SPEAKER_00: What is it that we can measure?
• 1:01:21
  SPEAKER_00: What is it worth measuring?
• 1:01:23
  SPEAKER_00: And what are things that are hard to know?
• 1:01:26
  SPEAKER_01: Yeah.
• 1:01:26
  SPEAKER_01: I mean, we've taken it as axiomatic that it is hard,
• 1:01:29
  SPEAKER_01: if not impossible, to measure productivity because what are the units even?
• 1:01:35
  SPEAKER_01: I had a while back a sort of clever idea, and I use that as a loaded term.
• 1:01:42
  SPEAKER_01: I had this idea that if we don't know what the units are,
• 1:01:46
  SPEAKER_01: like economics has prior art for this, right?
• 1:01:49
  SPEAKER_01: You have an index metric.
• 1:01:51
  SPEAKER_01: Where you measure all of the things that might influence this and you come up with
• 1:01:57
  SPEAKER_01: a way to effectively like basket those together.
• 1:02:01
  SPEAKER_01: And then you have a unitless,
• 1:02:03
  SPEAKER_01: but like directionally useful and useful over time,
• 1:02:07
  SPEAKER_01: like number that you can sort of like pin to.
• 1:02:11
  SPEAKER_01: And my theory had been,
• 1:02:13
  SPEAKER_01: so measure all of the things that we think are positive or negative indicators
• 1:02:17
  SPEAKER_01: of something is going well.
• 1:02:19
  SPEAKER_01: Releases, test coverage, number of PRs, everything.
• 1:02:26
  SPEAKER_01: Just cast a wide net.
• 1:02:28
  SPEAKER_01: My initial formulation on this was the wisdom of the crowds idea.
• 1:02:33
  SPEAKER_01: No individual can estimate how many jelly beans are in the jar,
• 1:02:36
  SPEAKER_01: but you get 100 people and you'll get real close.
• 1:02:38
  SPEAKER_01: So we do a like, am I hot or not?
• 1:02:42
  SPEAKER_01: Which would you prefer?
• 1:02:44
  SPEAKER_01: One release per week more or
• 1:02:49
  SPEAKER_00: X tests,
• 1:02:50
  SPEAKER_01: right?
• 1:02:51
  SPEAKER_01: And you run a bunch of iterations of that game
• 1:02:54
  SPEAKER_01: with your relatively experienced engineers to work out the linear trade-offs
• 1:03:00
  SPEAKER_01: between all of those factors.
• 1:03:01
  SPEAKER_01: To figure out what matters more.
• 1:03:02
  SPEAKER_01: Right, to figure out what it is.
• 1:03:04
  SPEAKER_01: You probably want to de-dupe some things because they're just going to be
• 1:03:08
  SPEAKER_01: the same stat in a different package.
• 1:03:10
  SPEAKER_01: But find the ones that have some non-linearity.
• 1:03:14
  SPEAKER_01: And then...
• 1:03:15
  SPEAKER_01: Mush that all together, right?
• 1:03:17
  SPEAKER_01: Now you have something that sort of gives you an aggregate,
• 1:03:22
  SPEAKER_01: like this is kind of our productivity number.
• 1:03:24
  SPEAKER_01: And Adobe is doing something along these lines.
• 1:03:28
  SPEAKER_01: Slightly smarter than my initial formulation.
• 1:03:30
  SPEAKER_01: But like,
• 1:03:31
  SPEAKER_01: then you can watch that track up or down and,
• 1:03:34
  SPEAKER_01: you know, over time for a given population.
• 1:03:37
  SPEAKER_01: What I've realized is that won't capture everything in this post AI boom.
• 1:03:44
  SPEAKER_01: And the reason that I came to that was a paper that I put out,
• 1:03:49
  SPEAKER_01: the Develop, Deploy,
• 1:03:51
  SPEAKER_01: Operate paper recently,
• 1:03:54
  SPEAKER_01: where one of the points in there is there's four forms of impact in tech.
• 1:03:59
  SPEAKER_01: There's product success,
• 1:04:01
  SPEAKER_01: the stuff everyone talks about and everyone else is better at than I am.
• 1:04:04
  SPEAKER_01: Like, that's not my job.
• 1:04:05
  SPEAKER_01: People buy a product or use a product, whatever.
• 1:04:08
  SPEAKER_01: Net promoter score revenue, like all of this stuff, right?
• 1:04:11
  SPEAKER_01: There's...
• 1:04:13
  SPEAKER_01: Compute efficiency, right?
• 1:04:14
  SPEAKER_01: Like, how much are you blowing on AWS?
• 1:04:16
  SPEAKER_01: Oh, yeah.
• 1:04:16
  SPEAKER_01: Right.
• 1:04:17
  SPEAKER_01: Or if you're shipping.
• 1:04:18
  SPEAKER_01: I'll send yours all too well.
• 1:04:19
  SPEAKER_01: Right.
• 1:04:19
  SPEAKER_01: You know,
• 1:04:20
  SPEAKER_01: or like even like the Android team has a tradeoff for if you can reduce
• 1:04:26
  SPEAKER_00: the size of
• 1:04:27
  SPEAKER_00: an application download by a megabyte.
• 1:04:29
  SPEAKER_00: It makes a difference for many metrics, yes.
• 1:04:31
  SPEAKER_00: Right.
• 1:04:32
  SPEAKER_01: You know, and so like all of those things, like that is a form of impact.
• 1:04:35
  SPEAKER_01: Then there's engineering productivity or we phrased it as engineering capacity.
• 1:04:41
  SPEAKER_01: Because if we save you 5% productivity gains,
• 1:04:45
  SPEAKER_01: capacity gains, you don't get to spend that somewhere else.
• 1:04:48
  SPEAKER_01: You just have more capacity on doing more engineering.
• 1:04:52
  SPEAKER_01: Or having slack time for innovation or picking up when something happens, that kind of stuff.
• 1:04:58
  SPEAKER_01: Right.
• 1:04:58
  SPEAKER_01: But they're not like dollar fungible.
• 1:04:59
  SPEAKER_01: Exactly.
• 1:05:00
  SPEAKER_01: You don't get money in a different account.
• 1:05:02
  SPEAKER_01: And then the fourth form of impact, and this ties back to the AI thing.
• 1:05:05
  SPEAKER_01: Fourth form of impact is changing the game, right?
• 1:05:08
  SPEAKER_01: Strategic innovation, like research makes impossible things possible.
• 1:05:13
  SPEAKER_01: Security makes incredibly bad things impossible, right?
• 1:05:17
  SPEAKER_01: But if you change the nature of what we are doing,
• 1:05:21
  SPEAKER_01: the cost structure, etc., then that's a different game.
• 1:05:27
  SPEAKER_01: And the problem that I think that we see with AI,
• 1:05:29
  SPEAKER_01: even with a clever technology,
• 1:05:31
  SPEAKER_01: You know, index metric approach to trying to capture the gestalt of productivity.
• 1:05:36
  SPEAKER_01: If you're not looking for the value of a given change is substantially different,
• 1:05:43
  SPEAKER_01: then there's no way you'll ever be able to capture that in the signal, right?
• 1:05:47
  SPEAKER_01: And we don't have anything that says the value of a change of this size has changed.
• 1:05:54
  SPEAKER_01: So for instance, like it is entirely possible at this point.
• 1:06:00
  SPEAKER_01: I hang out in security spaces and do a lot of low-level C++ library stuff.
• 1:06:06
  SPEAKER_01: You can write a JPEG parser in a couple weeks with whatever additional features you need.
• 1:06:12
  SPEAKER_01: That would be insane to say that
• 1:06:16
  SPEAKER_00: sentence out loud three years ago.
• 1:06:18
  SPEAKER_00: Okay, so what you're saying is...
• 1:06:21
  SPEAKER_00: What none of the measurements really capture is the possibility,
• 1:06:27
  SPEAKER_00: the impact,
• 1:06:28
  SPEAKER_00: the effort it is to do a massive innovation, which right now with AI is happening.
• 1:06:34
  SPEAKER_00: I'll give you also one example and tell me if this is what you mean with it.
• 1:06:41
  SPEAKER_00: There's a friend who works at a CTO at an engineering company,
• 1:06:47
  SPEAKER_00: a mid-sized engineering company, they're a niche industry.
• 1:06:51
  SPEAKER_00: The CTO,
• 1:06:52
  SPEAKER_00: they have been looking at release frequency,
• 1:06:54
  SPEAKER_00: the usual metrics, talk with their engineering team, are they happy, are they not?
• 1:06:58
  SPEAKER_00: And of course, now they're adopting AI as well.
• 1:07:00
  SPEAKER_00: They're seeing that, well,
• 1:07:02
  SPEAKER_00: people are more efficient and just privately see this person was telling me like,
• 1:07:05
  SPEAKER_00: look,
• 1:07:06
  SPEAKER_00: like I kind of know that for a fact that we could probably do the same amount
• 1:07:09
  SPEAKER_00: of work with like half the amount of people and then we could be cashflow positive.
• 1:07:13
  SPEAKER_00: First, there's a morale thing.
• 1:07:15
  SPEAKER_00: So he's like, also like, he likes the people.
• 1:07:17
  SPEAKER_00: So it's like,
• 1:07:18
  SPEAKER_00: I don't,
• 1:07:19
  SPEAKER_00: I'm not sure I want the emotional package on me,
• 1:07:21
  SPEAKER_00: but we can see some people are picking it up faster and some slower.
• 1:07:24
  SPEAKER_00: And we kind of left it at that.
• 1:07:25
  SPEAKER_00: A few weeks later,
• 1:07:26
  SPEAKER_00: I'm talking with this person that says like,
• 1:07:28
  SPEAKER_00: you wouldn't believe, but we're probably going to be cashflow positive.
• 1:07:31
  SPEAKER_00: I'm like, how?
• 1:07:32
  SPEAKER_00: Like, did you go ahead with the cut?
• 1:07:33
  SPEAKER_00: He's like, no.
• 1:07:34
  SPEAKER_00: What happened is one of our engineers came to me and said, like, we license this component.
• 1:07:40
  SPEAKER_00: Think of a library,
• 1:07:41
  SPEAKER_00: like it's a binary,
• 1:07:42
  SPEAKER_00: but that was written by a vendor in the space that has a high performance thing
• 1:07:46
  SPEAKER_00: that they use in their software.
• 1:07:48
  SPEAKER_00: It's a core part of it.
• 1:07:49
  SPEAKER_00: And they have a revenue share where they pay like,
• 1:07:52
  SPEAKER_00: I don't know, like 5, 10, 15% of their revenue for this component.
• 1:07:55
  SPEAKER_00: It's so important.
• 1:07:56
  SPEAKER_00: It's obviously licensed,
• 1:07:57
  SPEAKER_00: closed source,
• 1:07:58
  SPEAKER_00: but it's really important and it would take them Years of a five person
• 1:08:02
  SPEAKER_00: or 10 person team to write it and it might or might not work.
• 1:08:05
  SPEAKER_00: So it's not worth it.
• 1:08:06
  SPEAKER_00: Then you came and said like, what if, what if we just gave the specs?
• 1:08:09
  SPEAKER_00: Because they have all massive amounts of tests around this thing, a test harness.
• 1:08:12
  SPEAKER_00: They gave it to AI.
• 1:08:14
  SPEAKER_00: AI wrote a component to them.
• 1:08:15
  SPEAKER_00: And at first they thought, well, maybe we'll open source it.
• 1:08:23
  SPEAKER_00: But then they told the supplier and the supplier said, like, it's a huge business for them.
• 1:08:27
  SPEAKER_00: They're paying them to not open source it.
• 1:08:30
  SPEAKER_00: And to give them the implementation, which they can re-license.
• 1:08:35
  SPEAKER_00: And suddenly, the game was completely changed.
• 1:08:38
  SPEAKER_00: Like if this person would have gone ahead with,
• 1:08:40
  SPEAKER_00: let's say, the layoffs, this engineer might have just got it.
• 1:08:43
  SPEAKER_00: And no one might have known that the game
• 1:08:49
  SPEAKER_00: Right, exactly.
• 1:08:51
  SPEAKER_01: And so I think that this is the insight that tries to bridge the gap
• 1:08:55
  SPEAKER_01: between I have not seen a research based number
• 1:08:58
  SPEAKER_01: that says productivity post AI is up more than like 20 or 30 percent.
• 1:09:04
  SPEAKER_01: But then everyone talks about like these 10x sort of experiences.
• 1:09:07
  SPEAKER_01: I super don't think it's actually 10x, but yeah.
• 1:09:10
  SPEAKER_01: Yeah.
• 1:09:11
  SPEAKER_01: But like, how do you bridge that gap?
• 1:09:13
  SPEAKER_01: And I think it's exactly this thing of a lot of the value of AI is not in doing
• 1:09:19
  SPEAKER_01: the same thing faster.
• 1:09:22
  SPEAKER_01: It's doing a harder, more valuable thing that you wouldn't have even thought about.
• 1:09:27
  SPEAKER_01: And if your productivity measures are the logical equivalent
• 1:09:32
  SPEAKER_01: of counting how many widgets are coming out of the factory...
• 1:09:35
  SPEAKER_01: And then you start producing super gold-plated deluxe widgets.
• 1:09:38
  SPEAKER_01: There's no way to measure that or track that over time because we just don't have
• 1:09:43
  SPEAKER_01: the tracking of what's the value of this component or feature.
• 1:09:46
  SPEAKER_01: And that also matches with talking with senior engineers that are like,
• 1:09:51
  SPEAKER_01: yes, we're up 70% on amount of code.
• 1:09:55
  SPEAKER_01: We're up 30% on outages.
• 1:09:57
  SPEAKER_01: Number of features is the same.
• 1:09:59
  SPEAKER_01: The thing that we can't know for sure is,
• 1:10:03
  SPEAKER_00: are the value of those features up?
• 1:10:05
  SPEAKER_01: Because how do you do that?
• 1:10:08
  SPEAKER_00: One challenge that AI is giving is entering the market for new graduates.
• 1:10:17
  SPEAKER_00: There seems to be expectation.
• 1:10:19
  SPEAKER_00: Companies are very eager to hire senior engineers who know how to use AI,
• 1:10:22
  SPEAKER_00: and they have taste and judgment.
• 1:10:24
  SPEAKER_00: And we have these things thrown around.
• 1:10:26
  SPEAKER_00: It's not deliberate,
• 1:10:26
  SPEAKER_00: but whenever I talk to the AI labs,
• 1:10:28
  SPEAKER_00: they always tell me increasingly that they look for taste,
• 1:10:30
  SPEAKER_00: judgment, agency, these kind of things.
• 1:10:35
  SPEAKER_00: Let's go back to education, to computer science education.
• 1:10:40
  SPEAKER_00: What are your views of how today's computer science education works,
• 1:10:45
  SPEAKER_00: especially for how we know the industry is changing?
• 1:10:48
  SPEAKER_00: And what are some of your ideas of what could possibly be changed?
• 1:10:51
  SPEAKER_01: I will riff a little on my bona fides there to add some weight to this.
• 1:10:58
  SPEAKER_01: So after the book,
• 1:11:00
  SPEAKER_01: I got recruited into the ACM IEEE,
• 1:11:04
  SPEAKER_01: AAAI work on setting the curriculum requirements for an undergraduate CS degree.
• 1:11:10
  SPEAKER_01: So this is like the professional societies saying these are the things that we need to be doing.
• 1:11:15
  SPEAKER_01: And for the steering committee for that work, I was on the steering committee.
• 1:11:19
  SPEAKER_01: I was the only currently working engineer.
• 1:11:22
  SPEAKER_01: Everyone else is active.
• 1:11:23
  SPEAKER_01: Okay, some sense, because what can we teach?
• 1:11:27
  SPEAKER_01: And I managed mostly the software engineering requirements.
• 1:11:30
  SPEAKER_01: But it does mean that I see exactly what the requirements are and have been living
• 1:11:35
  SPEAKER_01: in that space for years.
• 1:11:37
  SPEAKER_01: And bluntly,
• 1:11:39
  SPEAKER_01: and I gave a talk on this in a pre-AI keynote,
• 1:11:43
  SPEAKER_01: bluntly,
• 1:11:44
  SPEAKER_01: most of the CS curriculum hasn't substantially changed Since like 2000,
• 1:11:50
  SPEAKER_01: maybe earlier.
• 1:11:52
  SPEAKER_01: The average thing in the curriculum is exactly what was needed to build a database
• 1:11:57
  SPEAKER_01: or an operating system in the 90s.
• 1:11:59
  SPEAKER_01: That doesn't sound good, does it?
• 1:12:00
  SPEAKER_01: Not for a field that prides itself on innovation, right?
• 1:12:05
  SPEAKER_01: And like there's all of these things where like I have to genuinely ask like,
• 1:12:11
  SPEAKER_01: are we teaching this because it is relevant and important?
• 1:12:15
  SPEAKER_01: Or are we teaching this because it is tradition and the history of- Or we know how to teach it.
• 1:12:21
  SPEAKER_01: Right.
• 1:12:21
  SPEAKER_01: Or we know how to teach it.
• 1:12:22
  SPEAKER_01: But because like the history of,
• 1:12:24
  SPEAKER_01: you know,
• 1:12:25
  SPEAKER_01: software engineering or CS departments,
• 1:12:27
  SPEAKER_01: like we're only at 50, 60, 70 years for the earliest CS departments, right?
• 1:12:33
  SPEAKER_01: Like this is a field that is way, way too young to To be doing stuff out of tradition.
• 1:12:38
  SPEAKER_01: Even pre-AI,
• 1:12:39
  SPEAKER_01: I was like,
• 1:12:40
  SPEAKER_01: I think a lot of this needs to be burned down because all of
• 1:12:43
  SPEAKER_01: the curriculum requirements, all of the teaching has mostly been through accretion.
• 1:12:49
  SPEAKER_01: We just throw more stuff in there and cut the amount of depth on the existing stuff by epsilon.
• 1:12:57
  SPEAKER_01: And I don't think that works.
• 1:13:01
  SPEAKER_01: Even pre-AI was saying,
• 1:13:02
  SPEAKER_01: the thing that I really want is you spend more time Practice and drill
• 1:13:07
  SPEAKER_01: and actually become like a fluent programmer,
• 1:13:11
  SPEAKER_01: algorithmic thinker,
• 1:13:13
  SPEAKER_01: not you have a trivia understanding of a million different topics.
• 1:13:18
  SPEAKER_01: And now AI, right?
• 1:13:21
  SPEAKER_01: Like we guarantee like you can do all of an undergraduate degree without ever thinking about it.
• 1:13:29
  SPEAKER_01: If you can just feed it into an AI because it will do just fine on all
• 1:13:32
  SPEAKER_01: of those programs and assignments.
• 1:13:34
  SPEAKER_01: And so a lot of what I'm pushing for is we need to get out of the code.
• 1:13:40
  SPEAKER_01: We need to up level substantially.
• 1:13:44
  SPEAKER_01: Like you need to be able to read code and think as a systems thinker
• 1:13:49
  SPEAKER_01: and like algorithmically like understand the structure and execution.
• 1:13:55
  SPEAKER_01: But.
• 1:13:56
  SPEAKER_01: No one needs to implement a linked list and we probably shouldn't even teach linked lists.
• 1:14:01
  SPEAKER_00: It's way too slow, right?
• 1:14:03
  SPEAKER_00: Memory is a slow thing.
• 1:14:04
  SPEAKER_01: The more that we can refocus on design,
• 1:14:10
  SPEAKER_01: specification,
• 1:14:11
  SPEAKER_01: like precision in saying what you want to happen and then testing,
• 1:14:16
  SPEAKER_01: validation,
• 1:14:17
  SPEAKER_01: verification,
• 1:14:17
  SPEAKER_01: like making sure that the AI actually did the stuff that you needed to happen,
• 1:14:21
  SPEAKER_01: right?
• 1:14:23
  SPEAKER_01: If we can teach those things,
• 1:14:25
  SPEAKER_01: Then it's suddenly perfectly relevant, you know, for professionals.
• 1:14:31
  SPEAKER_01: And I think the big question is,
• 1:14:33
  SPEAKER_01: will that be CS or are we looking at CS enrollment is going to plummet?
• 1:14:39
  SPEAKER_01: But software engineering programs are going to kick up because,
• 1:14:43
  SPEAKER_01: let's be honest,
• 1:14:44
  SPEAKER_01: the stuff that's going to matter in an AI era and still get a job is teamwork,
• 1:14:51
  SPEAKER_01: communication, design, specification, testing, validation, right?
• 1:14:56
  SPEAKER_01: Oh, that's a bunch of software engineering topics.
• 1:14:58
  SPEAKER_00: And also building production software, especially with AI.
• 1:15:02
  SPEAKER_00: I was talking with Peter Mattis,
• 1:15:04
  SPEAKER_00: co-founder of CockroachDB,
• 1:15:07
  SPEAKER_00: Cockroach Labs,
• 1:15:09
  SPEAKER_00: and he was telling me that during his college on the side, he built the GIMP.
• 1:15:15
  SPEAKER_00: And he got a lot of opportunities afterwards and then he built GTK+.
• 1:15:19
  SPEAKER_00: And I was thinking, hang on, like he did that on the side next to a CS degree.
• 1:15:24
  SPEAKER_00: Would you not want, just throw aside computer science.
• 1:15:27
  SPEAKER_00: I did computer science as well.
• 1:15:29
  SPEAKER_00: But for a software engineering degree,
• 1:15:31
  SPEAKER_00: would you not want to start on day one, especially with AI?
• 1:15:34
  SPEAKER_00: People come in here and you build on one class,
• 1:15:36
  SPEAKER_00: you build one software and the other will build, you build a software with the team.
• 1:15:40
  SPEAKER_00: On the third one, you extend the software.
• 1:15:42
  SPEAKER_00: On the fourth one,
• 1:15:44
  SPEAKER_00: because My experience being in a computer science thing, we did a budget theory.
• 1:15:49
  SPEAKER_00: My favorite class was the one team-based class where we actually built software.
• 1:15:52
  SPEAKER_00: In the four years, there was one, only one.
• 1:15:55
  SPEAKER_00: And then I took an extra class to actually learn .NET.
• 1:16:00
  SPEAKER_00: And then when I got into a real world, it felt I was starting from scratch.
• 1:16:04
  SPEAKER_00: I was picking up all of these things,
• 1:16:05
  SPEAKER_00: which we're not taught at the time these days are, but The practice was was missing.
• 1:16:10
  SPEAKER_00: I had the theory, but I definitely did not have the reps.
• 1:16:14
  SPEAKER_01: Right.
• 1:16:14
  SPEAKER_01: And I would argue that the LeetCode interview was because of this,
• 1:16:22
  SPEAKER_01: because, OK, how do you find signal that someone has put in the reps?
• 1:16:27
  SPEAKER_01: Right.
• 1:16:27
  SPEAKER_01: It's the students that were playing in the programming contests.
• 1:16:31
  SPEAKER_01: Right.
• 1:16:31
  SPEAKER_01: Those are the ones that had enough time.
• 1:16:33
  SPEAKER_00: I guess that was the only reps that were kind of a little bit common that were open to.
• 1:16:39
  SPEAKER_00: Engineers who didn't have software engineering expertise.
• 1:16:41
  SPEAKER_00: Sorry.
• 1:16:43
  SPEAKER_00: People with affinity for programming who did not have
• 1:16:45
  SPEAKER_01: software engineering expertise.
• 1:16:47
  SPEAKER_01: Exactly.
• 1:16:47
  SPEAKER_01: Right.
• 1:16:48
  SPEAKER_01: And so like it made perfect sense from that perspective for us
• 1:16:51
  SPEAKER_01: to like ask those sort of lead code questions in the programming contest styles,
• 1:16:57
  SPEAKER_01: not because we needed solutions to yet another breadth first search problem
• 1:17:00
  SPEAKER_01: or reversing a linked list or any of that nonsense,
• 1:17:03
  SPEAKER_01: but specifically because it was a proxy for you've been in an environment
• 1:17:08
  SPEAKER_01: where you put in reps.
• 1:17:09
  SPEAKER_01: Whereas the amount of actual like programming time on task for
• 1:17:12
  SPEAKER_01: an average degree is pretty modest.
• 1:17:16
  SPEAKER_01: And, you know, you go back to 10,000 hours to mastery and Malcolm Gladwell.
• 1:17:20
  SPEAKER_00: I think that's a Gladwell.
• 1:17:22
  SPEAKER_00: And like, yeah, like where are you getting the time?
• 1:17:25
  SPEAKER_00: And if I think back of the boot camps that spun up in the from like early 2010s,
• 1:17:31
  SPEAKER_00: they started to get really strong 2014 to 2017.
• 1:17:33
  SPEAKER_00: What they did is a three month, three to six month
• 1:17:38
  SPEAKER_00: Yeah, yeah, yeah.
• 1:18:02
  SPEAKER_00: There was such a huge demand for software engineers that they got hired as
• 1:18:06
  SPEAKER_00: a junior engineer level or almost at an apprentice level.
• 1:18:08
  SPEAKER_00: But what they knew compared to computer science grads,
• 1:18:13
  SPEAKER_00: they knew version control,
• 1:18:14
  SPEAKER_00: they knew Git,
• 1:18:15
  SPEAKER_00: they knew pull requests,
• 1:18:16
  SPEAKER_00: they knew a database, they knew a frontend, they knew Chrome developer tools.
• 1:18:22
  SPEAKER_00: So they actually had a bunch of valuable skills in only three months that
• 1:18:26
  SPEAKER_00: the people coming out from the program in four years might or might not have had.
• 1:18:30
  SPEAKER_00: It was more of a coin toss.
• 1:18:31
  SPEAKER_00: Yeah, exactly.
• 1:18:32
  SPEAKER_01: I've done the math.
• 1:18:34
  SPEAKER_01: The number of hours in a CS degree is roughly equivalent to like 15 months
• 1:18:42
  SPEAKER_01: on average of full-time employment.
• 1:18:45
  SPEAKER_00: And that includes all of the theory stuff,
• 1:18:48
  SPEAKER_00: all of the
• 1:18:48
  SPEAKER_01: things that are questionably relevant and is similarly trying to take you
• 1:18:54
  SPEAKER_01: from no experience to employable in either an industry or academic direction.
• 1:19:00
  SPEAKER_01: And like that is...
• 1:19:02
  SPEAKER_01: Compared to a six month bootcamp, right?
• 1:19:04
  SPEAKER_01: Like that's that's not a huge gap,
• 1:19:07
  SPEAKER_01: especially when one of those is laser focused on these are all things
• 1:19:11
  SPEAKER_01: that are just immediately relevant.
• 1:19:14
  SPEAKER_01: And one of them is here's some stuff that'll hopefully teach you how to think
• 1:19:19
  SPEAKER_01: about these problems, maybe or less charitably.
• 1:19:22
  SPEAKER_01: This was useful when I was a student.
• 1:19:25
  SPEAKER_00: And also,
• 1:19:25
  SPEAKER_00: can we just mention some of the most successful schools were actually outside of
• 1:19:30
  SPEAKER_00: the US.
• 1:19:31
  SPEAKER_00: One of them, well,
• 1:19:32
  SPEAKER_00: a lot of the schools are,
• 1:19:33
  SPEAKER_00: of course,
• 1:19:34
  SPEAKER_00: you know,
• 1:19:35
  SPEAKER_00: the usual suspects are very highly regarded from the MIT's,
• 1:19:39
  SPEAKER_00: the Stanford's and so on.
• 1:19:40
  SPEAKER_00: But Waterloo in Canada,
• 1:19:43
  SPEAKER_00: I was very surprised to see that a bunch of Silicon Valley companies have
• 1:19:46
  SPEAKER_00: a high number of Waterloo companies.
• 1:19:48
  SPEAKER_00: And I asked this on social media, like, what's on with Waterloo?
• 1:19:51
  SPEAKER_00: They're up in Canada.
• 1:19:52
  SPEAKER_00: Why is everyone hiring from there?
• 1:19:53
  SPEAKER_00: So it turns out that Waterloo has this program where,
• 1:19:57
  SPEAKER_00: yeah, they teach you computer science or software engineering.
• 1:20:00
  SPEAKER_00: I'm not sure which one.
• 1:20:01
  SPEAKER_00: But the main part is that every...
• 1:20:03
  SPEAKER_00: They have dedicated placements.
• 1:20:05
  SPEAKER_00: By the time someone graduates,
• 1:20:06
  SPEAKER_00: they will have had four placements slash internships at different companies.
• 1:20:10
  SPEAKER_00: And because they have the relationship, this will often be the Shopify, the meta, this.
• 1:20:14
  SPEAKER_00: So you will get an engineer who,
• 1:20:18
  SPEAKER_00: three months times four, that's like a year working inside of these companies.
• 1:20:22
  SPEAKER_00: And that is the balance.
• 1:20:23
  SPEAKER_00: Yeah, exactly.
• 1:20:46
  SPEAKER_00: I don't think it's just Waterloo.
• 1:20:48
  SPEAKER_01: Like a lot of the Canadian schools are- This was
• 1:20:49
  SPEAKER_00: just one example,
• 1:20:51
  SPEAKER_01: by the way.
• 1:20:51
  SPEAKER_01: Yeah.
• 1:20:51
  SPEAKER_01: But yeah,
• 1:20:52
  SPEAKER_01: a lot of the Canadian schools for CS have substantially stronger programs
• 1:20:57
  SPEAKER_01: on internships and co-ops in a way that a lot of those degrees run five years now.
• 1:21:02
  SPEAKER_01: But I think it is a much stronger educational experience because there are some
• 1:21:06
  SPEAKER_01: of those lessons,
• 1:21:07
  SPEAKER_01: like I will fully admit,
• 1:21:09
  SPEAKER_01: like there's stuff we did not put into the software engineering requirements.
• 1:21:13
  SPEAKER_01: Because there's just no way to do that in the academic setting, right?
• 1:21:17
  SPEAKER_01: There's some lessons you're going to have to learn the real way,
• 1:21:20
  SPEAKER_01: like in the field, even though you gave the example of like doing a project.
• 1:21:25
  SPEAKER_01: I want us to do more projects,
• 1:21:27
  SPEAKER_01: but I'm really nervous about there being one project course in most degrees
• 1:21:33
  SPEAKER_01: because you know what no team has ever looked like?
• 1:21:37
  SPEAKER_01: A bunch of people with exactly the same level of background, right?
• 1:21:40
  SPEAKER_01: Like it also gives the wrong taste about this is what teamwork is like.
• 1:21:47
  SPEAKER_01: I'll take it.
• 1:21:47
  SPEAKER_01: It's better than not doing it.
• 1:21:49
  SPEAKER_00: But.
• 1:21:51
  SPEAKER_00: Well, we have what we have.
• 1:21:53
  SPEAKER_00: And this system will hopefully with people like yourself
• 1:21:56
  SPEAKER_00: and others will hopefully start to also change or just adapt to the times.
• 1:22:01
  SPEAKER_00: But AI is right now here.
• 1:22:03
  SPEAKER_00: And one.
• 1:22:05
  SPEAKER_00: Big question, more experienced engineers are thinking about.
• 1:22:10
  SPEAKER_00: We've learned to build software pre-AI.
• 1:22:13
  SPEAKER_00: We know the fundamentals.
• 1:22:15
  SPEAKER_00: Many of us went through computer science,
• 1:22:16
  SPEAKER_00: so we actually have a very strong theoretical understanding which we might
• 1:22:20
  SPEAKER_00: or might not use some parts of it.
• 1:22:21
  SPEAKER_00: But what about for new grads?
• 1:22:24
  SPEAKER_00: Now that they'll enter a workforce of just today with just the capability that we have, what...
• 1:22:30
  SPEAKER_00: Do you think the fundamentals are that they should focus on and what are the things
• 1:22:34
  SPEAKER_00: that they should just rely on AI?
• 1:22:36
  SPEAKER_00: There's always a question of like,
• 1:22:37
  SPEAKER_00: should you just ask AI for this or should you learn the basics, right?
• 1:22:40
  SPEAKER_01: Yeah.
• 1:22:41
  SPEAKER_01: So, I mean,
• 1:22:41
  SPEAKER_01: I think learn the basics is an entirely loaded question because nobody agrees
• 1:22:45
  SPEAKER_01: on what the basics are.
• 1:22:48
  SPEAKER_01: And that's fair.
• 1:22:49
  SPEAKER_01: Kent Beck said a couple months ago
• 1:22:51
  SPEAKER_01: when we were last together something like everyone that's currently working has
• 1:22:58
  SPEAKER_01: grown up in an era where there was a playbook.
• 1:23:00
  SPEAKER_01: Like we know at least in theory the right way to do this and there's no playbook right now.
• 1:23:06
  SPEAKER_01: Right.
• 1:23:06
  SPEAKER_01: So we all have to get a little comfortable of not knowing the answer.
• 1:23:09
  SPEAKER_01: And we certainly don't know the answer on what is still the basics.
• 1:23:13
  SPEAKER_01: The things that I think that the juniors should be focusing on one.
• 1:23:19
  SPEAKER_01: The resilience to avoid just copying your ticket into a prompt and then trying
• 1:23:26
  SPEAKER_01: to check in that result.
• 1:23:28
  SPEAKER_01: Right.
• 1:23:28
  SPEAKER_01: Like if you're if that is all that you are doing.
• 1:23:31
  SPEAKER_01: You can be replaced with a very small shell script.
• 1:23:34
  SPEAKER_01: Like that is not a defensible advantage for you being on staff.
• 1:23:39
  SPEAKER_01: I think it's really important to use AI to better yourself and to widen your inputs,
• 1:23:47
  SPEAKER_01: but not to immediately produce all of your outputs.
• 1:23:52
  SPEAKER_01: There's a metaphor that I really liked from about a year ago at a think tank.
• 1:23:57
  SPEAKER_01: We called it Andrew's Axe.
• 1:24:00
  SPEAKER_01: And it's If I go chop down a tree with an axe,
• 1:24:06
  SPEAKER_01: I do not say that it was the axe that chopped down the tree.
• 1:24:09
  SPEAKER_01: It's me that chopped down the tree.
• 1:24:11
  SPEAKER_01: The AI, as much as we occasionally anthropomorphize it,
• 1:24:15
  SPEAKER_01: is still a tool and 1979 IBM training manual.
• 1:24:20
  SPEAKER_01: A computer will never be held accountable.
• 1:24:22
  SPEAKER_01: Therefore, a computer should not be making management decisions.
• 1:24:25
  SPEAKER_01: The AI is still not accountable.
• 1:24:27
  SPEAKER_01: The AI labs are not going to indemnify us for the mistakes that these things produce.
• 1:24:34
  SPEAKER_01: So you have to be able to stand up for whatever your name is on for those pull requests.
• 1:24:39
  SPEAKER_01: And that requires a lot more rigor and not cutting those corners.
• 1:24:44
  SPEAKER_01: And that attention to detail, I think, is really hard to train.
• 1:24:49
  SPEAKER_01: Going back to sort of the undergraduate version of this,
• 1:24:51
  SPEAKER_01: the scary story that I've been hearing from a lot of profs is,
• 1:24:56
  SPEAKER_01: The best students,
• 1:24:57
  SPEAKER_01: like the B,
• 1:24:58
  SPEAKER_01: B plus and above students,
• 1:25:01
  SPEAKER_01: they are better than they ever have been because they have.
• 1:25:04
  SPEAKER_01: Yeah, because instead of relying on a TA that is available,
• 1:25:09
  SPEAKER_01: you know,
• 1:25:09
  SPEAKER_01: a couple hours a week and is one or two grades above them,
• 1:25:12
  SPEAKER_01: they have an infinitely patient,
• 1:25:14
  SPEAKER_01: always available, incredibly well read, sometimes hallucinatory tutor.
• 1:25:20
  SPEAKER_01: Right.
• 1:25:20
  SPEAKER_01: At the drop of a hat.
• 1:25:21
  SPEAKER_01: Right.
• 1:25:21
  SPEAKER_01: They can always learn and explain and figure out.
• 1:25:24
  SPEAKER_00: And they actually have the autonomy
• 1:25:26
  SPEAKER_01: and the drive to use this.
• 1:25:28
  SPEAKER_01: Right.
• 1:25:28
  SPEAKER_01: And the and the metacognitive resources to avoid that.
• 1:25:32
  SPEAKER_01: Like, I'm just going to copy and paste.
• 1:25:34
  SPEAKER_01: Right.
• 1:25:35
  SPEAKER_01: The very worst students are somewhat better.
• 1:25:38
  SPEAKER_01: Because they used to not be able to do anything, but now they can produce slop.
• 1:25:41
  SPEAKER_00: Yeah, they just throw in slop, yeah.
• 1:25:43
  SPEAKER_00: Which is a bit hard to detect, but I get what you're calling.
• 1:25:46
  SPEAKER_01: But everyone in the middle has kind of collapsed.
• 1:25:48
  SPEAKER_01: They don't have that resilience.
• 1:25:50
  SPEAKER_01: They would rather go to the party or play video games than do the homework.
• 1:25:54
  SPEAKER_01: And so they're like, well, just this one time, I'll just get in the habit of doing this thing.
• 1:25:59
  SPEAKER_01: And that slippery slope just gets worse.
• 1:26:02
  SPEAKER_01: And so there are a lot of profs from decent schools that are like, Yeah, I don't know, man.
• 1:26:07
  SPEAKER_01: Like, I'm not sure that anyone, like, below, like,
• 1:26:11
  SPEAKER_01: the vast middle, everyone below a B, might not understand how code works at all.
• 1:26:16
  SPEAKER_01: Like, the system is just not set up to be tracking that sort of thing.
• 1:26:22
  SPEAKER_00: Or at least giving them feedback, for example, alerting,
• 1:26:25
  SPEAKER_01: those kind of things.
• 1:26:25
  SPEAKER_01: Right.
• 1:26:26
  SPEAKER_01: And so a lot of the discourse initially was,
• 1:26:29
  SPEAKER_01: how do we prevent students from using these AI tools?
• 1:26:32
  SPEAKER_01: And I think that that's nonsense because it's echoes of,
• 1:26:36
  SPEAKER_01: you know, we got high level languages and there's compilers.
• 1:26:40
  SPEAKER_01: But now you're not going to learn how to program in assembly, right?
• 1:26:43
  SPEAKER_01: The industry is using it anyway.
• 1:26:45
  SPEAKER_01: Right.
• 1:26:45
  SPEAKER_01: They're going to be asked to use,
• 1:26:47
  SPEAKER_01: they're going to be forced,
• 1:26:48
  SPEAKER_01: pressured to use it harder in some places when they get a job.
• 1:26:52
  SPEAKER_00: Well, in fact,
• 1:26:53
  SPEAKER_00: it might be,
• 1:26:54
  SPEAKER_00: it will for sure be a red flag if a new grad says, like, I've not used these tools.
• 1:26:58
  SPEAKER_00: Like, come on.
• 1:26:58
  SPEAKER_00: Right.
• 1:26:59
  SPEAKER_00: Have you not?
• 1:27:00
  SPEAKER_00: Do you not want to work in this industry?
• 1:27:02
  SPEAKER_00: It's an open secret what we're doing.
• 1:27:03
  SPEAKER_00: We're publishing it on blogs and everywhere.
• 1:27:05
  SPEAKER_01: Yeah, exactly.
• 1:27:06
  SPEAKER_01: Right.
• 1:27:06
  SPEAKER_01: And so the line of, well, we're going to never let you use these tools cannot possibly hold.
• 1:27:14
  SPEAKER_01: I co-published a piece in Communications of ACM with a bunch of academics this month
• 1:27:22
  SPEAKER_01: that I think the opening line there is imagine forward to 2040 and
• 1:27:28
  SPEAKER_01: And we hold the line treating all use of AI as cheating,
• 1:27:32
  SPEAKER_01: but a generation of students, AI,
• 1:27:35
  SPEAKER_01: you know,
• 1:27:36
  SPEAKER_01: they've grown up with AI in the same way that the internet natives have grown up
• 1:27:39
  SPEAKER_01: with the network.
• 1:27:42
  SPEAKER_01: If we continue asking them these traditional questions about, like,
• 1:27:46
  SPEAKER_01: find a palindrome in this string or measure the volume of rainfall in this graph,
• 1:27:51
  SPEAKER_01: right?
• 1:27:51
  SPEAKER_01: Like, are they going to learn anything from a traditional CS degree?
• 1:27:56
  SPEAKER_01: The answer is obviously no.
• 1:27:58
  SPEAKER_01: So, like,
• 1:28:00
  SPEAKER_01: if we know that we cannot stay here and there's plenty of things that need fixing,
• 1:28:06
  SPEAKER_01: like...
• 1:28:07
  SPEAKER_01: Let's burn it down and do something that matters and figure out how to use this appropriately.
• 1:28:13
  SPEAKER_01: And I think finding ways to assume that they're using AI and what are
• 1:28:19
  SPEAKER_01: the new learning objectives and how do you make that still frictionful in the ways
• 1:28:26
  SPEAKER_01: that cause them to think and learn about different things,
• 1:28:31
  SPEAKER_01: like almost certainly at a higher level.
• 1:28:37
  SPEAKER_01: Undergraduate education and curriculum design and assessment.
• 1:28:41
  SPEAKER_00: Well, it sounds like whoever's,
• 1:28:42
  SPEAKER_00: so the professors,
• 1:28:44
  SPEAKER_00: the The TAs,
• 1:28:46
  SPEAKER_00: the people who are designing this course,
• 1:28:47
  SPEAKER_00: they also need to put in a bunch of work because no one has this figured out.
• 1:28:50
  SPEAKER_00: You need to experiment.
• 1:28:52
  SPEAKER_00: You need to first use the tools yourself.
• 1:28:54
  SPEAKER_00: You need to come up with novel ways of creating an assignment which you need
• 1:28:58
  SPEAKER_00: to use AI to solve, but AI itself is not enough.
• 1:29:02
  SPEAKER_00: How do you get the feedback?
• 1:29:04
  SPEAKER_00: You might need to put in more time,
• 1:29:06
  SPEAKER_00: let's say, sit down with someone afterwards and have them experiment.
• 1:29:09
  SPEAKER_00: And one of the interviews that seem to be working well is when you're pairing
• 1:29:17
  SPEAKER_00: with someone,
• 1:29:17
  SPEAKER_00: like video or in person,
• 1:29:19
  SPEAKER_00: you're using AI, but you're explaining what you're doing and how, right?
• 1:29:24
  SPEAKER_00: You cannot just fully fake it and go away, give it to a friend or just have Claude do it.
• 1:29:29
  SPEAKER_00: You're now showing your cognitive ability.
• 1:29:32
  SPEAKER_00: The interviewer can push you into uncomfortable,
• 1:29:36
  SPEAKER_00: you know,
• 1:29:36
  SPEAKER_00: they'll get a gauge of like how expert you are and push you further and further and
• 1:29:40
  SPEAKER_01: so on.
• 1:29:40
  SPEAKER_01: Yeah, exactly.
• 1:29:42
  SPEAKER_01: And at the undergraduate level,
• 1:29:43
  SPEAKER_01: like there is some move towards more oral exams and those sort of things.
• 1:29:48
  SPEAKER_01: But also I think a really simple solution is stop grading just the outcome.
• 1:29:54
  SPEAKER_01: Focus on the process, right?
• 1:29:56
  SPEAKER_01: Like even if you don't come up with wildly harder assignments for assessment,
• 1:30:04
  SPEAKER_01: stop grading just the code that they turned in.
• 1:30:07
  SPEAKER_01: Grade instead, what are the prompts that you used to get here?
• 1:30:11
  SPEAKER_01: Also, what are the tests that you provided?
• 1:30:13
  SPEAKER_01: Like how did you make sure that this thing is actually doing what you need it to do?
• 1:30:18
  SPEAKER_01: And like when you start focusing on like the process that got you there instead
• 1:30:24
  SPEAKER_01: of just the outcome, like I think not only CS, but most disciplines, right?
• 1:30:30
  SPEAKER_01: Like we've been using go write a new essay on capture in the rye.
• 1:30:34
  SPEAKER_01: Like we don't need another essay on capture, right?
• 1:30:37
  SPEAKER_01: Go write a new linked list.
• 1:30:38
  SPEAKER_01: We don't need another linked list.
• 1:30:39
  SPEAKER_01: We've been using the artifact that's produced from that as a proxy for you put in
• 1:30:45
  SPEAKER_01: the time and the struggle.
• 1:30:48
  SPEAKER_01: And like faced the friction that we assume that you now have learned something.
• 1:30:53
  SPEAKER_01: It might be more appropriate to just focus on do we have evidence
• 1:30:57
  SPEAKER_01: that you've struggled and learned?
• 1:31:00
  SPEAKER_01: Turn in the whole package, right?
• 1:31:02
  SPEAKER_00: Curriculums are designed by committees, which is for a good reason.
• 1:31:05
  SPEAKER_00: But if you had Titus' software engineering school,
• 1:31:08
  SPEAKER_00: where you could yourself put together the curriculum and study course for
• 1:31:12
  SPEAKER_00: an undergrad course of,
• 1:31:13
  SPEAKER_00: let's say, three to four years, how would it look like right now?
• 1:31:17
  SPEAKER_00: And we're talking about getting engineers,
• 1:31:21
  SPEAKER_00: ideally having engineers who can hit the ground running at software
• 1:31:25
  SPEAKER_00: and engineering companies,
• 1:31:26
  SPEAKER_00: obviously using AI tools, building software, that kind of stuff.
• 1:31:29
  SPEAKER_01: Yeah, so...
• 1:31:30
  SPEAKER_01: I think the first course, like the first semester starts unaided by AI.
• 1:31:37
  SPEAKER_01: Like we need to get you through at least the level that you can read code and have
• 1:31:41
  SPEAKER_01: a vague idea of what's going on.
• 1:31:43
  SPEAKER_01: Like you can understand control flow and those sorts of things.
• 1:31:46
  SPEAKER_01: I think there's some really interesting things on well scaffolded and harnessed AI
• 1:31:51
  SPEAKER_01: for educational purposes.
• 1:31:53
  SPEAKER_01: Of we're intentionally not just going to give you the answer.
• 1:31:57
  SPEAKER_01: And in conjunction with if you're assessing the whole package and process
• 1:32:01
  SPEAKER_01: of creation,
• 1:32:02
  SPEAKER_01: then you don't need to worry about them going to chat GPT and just getting
• 1:32:05
  SPEAKER_01: the whole answer.
• 1:32:06
  SPEAKER_01: Some of that's covered.
• 1:32:07
  SPEAKER_01: I mean, I'll be honest.
• 1:32:09
  SPEAKER_01: I don't know.
• 1:32:10
  SPEAKER_01: Nobody knows.
• 1:32:11
  SPEAKER_01: There is no playbook.
• 1:32:12
  SPEAKER_01: But one of the ideas.
• 1:32:14
  SPEAKER_00: I want to get opinionated.
• 1:32:16
  SPEAKER_01: So one of the ideas that I've had is let's make CS0 like non-majors and majors alike.
• 1:32:24
  SPEAKER_01: Intro to programs,
• 1:32:25
  SPEAKER_01: like specifically,
• 1:32:26
  SPEAKER_01: how do you describe the inputs and outputs and structure and design of
• 1:32:30
  SPEAKER_01: the interfaces, right?
• 1:32:32
  SPEAKER_01: So we could talk about what's the difference between a checkbox and a radio button.
• 1:32:37
  SPEAKER_01: When do you use a button versus some other UI widget
• 1:32:42
  SPEAKER_01: for non-UI things like what command line flags, you know, positional parameters?
• 1:32:46
  SPEAKER_01: How do you like interface these components specifically so that if that's
• 1:32:51
  SPEAKER_01: the only class that you ever took,
• 1:32:53
  SPEAKER_01: At least the AI vibe-coded slop that you produce for your own use cases is going
• 1:33:00
  SPEAKER_01: to have a slightly more coherent,
• 1:33:02
  SPEAKER_01: sensible design language and be more fit for purpose.
• 1:33:05
  SPEAKER_01: I live in New York.
• 1:33:08
  SPEAKER_01: The condo building I'm in, there's an app for the laundry room.
• 1:33:13
  SPEAKER_01: And drives me insane because, like, they use, like, menu UI for button, right?
• 1:33:20
  SPEAKER_01: And, like, just the design language is completely wrong.
• 1:33:23
  SPEAKER_01: Start with that.
• 1:33:24
  SPEAKER_01: And then graduate kind of along the lines of what you were saying of,
• 1:33:27
  SPEAKER_01: like, we want to move up into, like, projects and teamwork and things like that.
• 1:33:32
  SPEAKER_01: Try to motivate them into themselves wanting to solve a bigger problem and
• 1:33:39
  SPEAKER_01: then needing some Background on, oh, this is how the web works.
• 1:33:44
  SPEAKER_01: This is how a remote API RPC call works.
• 1:33:47
  SPEAKER_01: You can't give people a solution to a problem they don't realize they have, right?
• 1:33:51
  SPEAKER_01: And until they're trying to solve these problems,
• 1:33:54
  SPEAKER_01: those solutions that we present in theory are all just trivia.
• 1:33:59
  SPEAKER_01: They're just isolated, academic, not connected to anything.
• 1:34:03
  SPEAKER_01: Like motivate that by like getting them into a situation where they need that as
• 1:34:08
  SPEAKER_01: the answer first.
• 1:34:09
  SPEAKER_01: And then you can provide that background.
• 1:34:11
  SPEAKER_01: I also desperately want more projects for two reasons.
• 1:34:17
  SPEAKER_01: I don't want the assignments to be,
• 1:34:20
  SPEAKER_01: there is a correct answer because that's not how the industry works.
• 1:34:24
  SPEAKER_01: Mostly not how research works.
• 1:34:26
  SPEAKER_01: And I want them to grade, part of your grade is critique of others.
• 1:34:33
  SPEAKER_01: Like look at their outcomes or their code or both, right?
• 1:34:38
  SPEAKER_01: And be able to say like, that was great.
• 1:34:42
  SPEAKER_01: Did you think about doing a different thing here?
• 1:34:44
  SPEAKER_01: And then also grade them on can you give feedback without being a jerk?
• 1:34:50
  SPEAKER_01: Because psychological safety and teamwork, right?
• 1:34:53
  SPEAKER_01: And all of this sounds like a lot of work,
• 1:34:55
  SPEAKER_01: except you can use AI to like do some of that analysis.
• 1:34:59
  SPEAKER_01: Like let's do the sentiment analysis.
• 1:35:03
  SPEAKER_01: Was that rude?
• 1:35:05
  SPEAKER_01: Things like that.
• 1:35:06
  SPEAKER_01: And I think we could have a much more practical,
• 1:35:11
  SPEAKER_01: hands-on, and much more effective educational structure.
• 1:35:16
  SPEAKER_01: And I worry that that's going to be software engineering departments and not CS departments.
• 1:35:22
  SPEAKER_00: And as closing, for...
• 1:35:23
  SPEAKER_00: For grads right now,
• 1:35:26
  SPEAKER_00: college students sitting inside of CS programs,
• 1:35:28
  SPEAKER_00: software engineering programs who are seeing,
• 1:35:31
  SPEAKER_00: okay, it's more competitive to get into the industry, tools are changing.
• 1:35:35
  SPEAKER_00: What advice would you get in terms of tactics,
• 1:35:37
  SPEAKER_00: strategies to activities to do to maximize your chances of getting into the industry
• 1:35:43
  SPEAKER_00: and then doing well there?
• 1:35:45
  SPEAKER_01: This might be risky, but most of my success in life has come from not
• 1:35:51
  SPEAKER_00: doing the thing
• 1:35:51
  SPEAKER_00: that everyone else is doing.
• 1:35:53
  SPEAKER_01: Like it is really incredibly hard to stand out if you are just listening to what is
• 1:35:59
  SPEAKER_01: a hot topic and chasing the same shiny thing that
• 1:36:02
  SPEAKER_00: literally everyone else is doing.
• 1:36:04
  SPEAKER_01: Right.
• 1:36:05
  SPEAKER_01: And so I think it is to thine own self be true.
• 1:36:08
  SPEAKER_01: Right.
• 1:36:09
  SPEAKER_01: Like find something that you are actually interested in and put in
• 1:36:12
  SPEAKER_01: the time doing something that you care about,
• 1:36:15
  SPEAKER_01: especially if it's weirder than what everyone else around you is doing,
• 1:36:19
  SPEAKER_01: because then you will have a novel story to tell your interviewer.
• 1:36:24
  SPEAKER_01: You will be worth talking to at a cocktail party so that
• 1:36:27
  SPEAKER_01: the startup founder is like, yeah, I want to take a risk on this kid, right?
• 1:36:35
  SPEAKER_01: And the more time that you put into these things,
• 1:36:37
  SPEAKER_01: the more expertise that you can bring to it, the better off you're going to be.
• 1:36:42
  SPEAKER_01: I have a worry that with AI-assisted engineering,
• 1:36:47
  SPEAKER_01: Our emphasis on raw engineering skill is dropping.
• 1:36:51
  SPEAKER_01: Like you can get by with a lot less of that depth.
• 1:36:55
  SPEAKER_01: But the only way to replace that is you need to understand the problem domain
• 1:36:59
  SPEAKER_01: that you're operating in to a much deeper level.
• 1:37:02
  SPEAKER_01: And I think that that might actually lead to less mobility in the industry.
• 1:37:08
  SPEAKER_01: Like it's not, I'm a C++ engineer or I'm a backend engineer.
• 1:37:12
  SPEAKER_01: It's a, I'm an engineer in embedded audio.
• 1:37:16
  SPEAKER_01: Right.
• 1:37:17
  SPEAKER_01: Like you understand those problem domains to a much deeper level
• 1:37:21
  SPEAKER_01: and that's your actual advantage.
• 1:37:23
  SPEAKER_00: As closing,
• 1:37:24
  SPEAKER_00: what's a book that you would recommend and why outside of your excellent book,
• 1:37:28
  SPEAKER_00: of course?
• 1:37:28
  SPEAKER_01: I mentioned it earlier, but I'll stick with it.
• 1:37:31
  SPEAKER_01: Psychology of Software Teams by Cat Hicks,
• 1:37:34
  SPEAKER_01: especially like there are a couple of chapters in that that I could have
• 1:37:38
  SPEAKER_01: that downloaded into the brains of every leader around.
• 1:37:41
  SPEAKER_01: I would be so happy.
• 1:37:42
  SPEAKER_01: Like the brains in jars analogy is.
• 1:37:47
  SPEAKER_01: I'm glad I got to finally
• 1:37:54
  SPEAKER_00: sit down with Titus, and this conversation did not disappoint.
• 1:37:58
  SPEAKER_00: To start with,
• 1:37:59
  SPEAKER_00: I knew that Google's codebase was large,
• 1:38:00
  SPEAKER_00: but I had no idea we're talking about 250 million lines of just C++ code.
• 1:38:06
  SPEAKER_00: And this was all pre-AI,
• 1:38:07
  SPEAKER_00: so we're not talking about over They were both code bases generated by AI.
• 1:38:11
  SPEAKER_00: As comparison,
• 1:38:12
  SPEAKER_00: Uber's mobile apps had code bases of about one to two million lines of iOS
• 1:38:16
  SPEAKER_00: and Android code back in 2016,
• 1:38:19
  SPEAKER_00: and Uber crammed a lot of business logic into those apps.
• 1:38:21
  SPEAKER_00: How Titus and the team cleaned up these foundations
• 1:38:24
  SPEAKER_00: Without using AI was also pretty educational.
• 1:38:26
  SPEAKER_00: They built deterministic code generation tools to parse the code,
• 1:38:30
  SPEAKER_00: match patterns, then generate edits.
• 1:38:32
  SPEAKER_00: This is an approach that still could work more reliably even today
• 1:38:35
  SPEAKER_00: than just asking non-deterministic AI to refactor a codebase,
• 1:38:38
  SPEAKER_00: even though it is arguably more difficult to write advanced parsing logic on
• 1:38:42
  SPEAKER_00: a codebase like Titus and team did.
• 1:38:44
  SPEAKER_00: I also appreciated Titus' view on where AI is helpful today.
• 1:38:48
  SPEAKER_00: He mentioned this quote he heard from a senior engineer that said,
• 1:38:51
  SPEAKER_00: we are up 70% on the amount of code, but we're also up 30% on outages.
• 1:38:56
  SPEAKER_00: The number of features is the same.
• 1:38:58
  SPEAKER_00: Well, that doesn't sound all that good.
• 1:39:00
  SPEAKER_00: Where's the value from AI then?
• 1:39:02
  SPEAKER_00: Titus says that we won't get all that much value from doing the same stuff we did
• 1:39:06
  SPEAKER_00: before AI, but faster.
• 1:39:08
  SPEAKER_00: The value is to be more ambitious and do harder things
• 1:39:11
  SPEAKER_00: that we would have never attempted before AI.
• 1:39:13
  SPEAKER_00: Finally,
• 1:39:14
  SPEAKER_00: I hear a lot of worry for new grads in tech about how they'll get a job or not,
• 1:39:18
  SPEAKER_00: especially when AI agents are so commonly used,
• 1:39:21
  SPEAKER_00: but CS programs have not incorporated them into their program.
• 1:39:25
  SPEAKER_00: Titus is one of the few engineers who has been part of the steering committee for
• 1:39:28
  SPEAKER_00: the CS curriculum and says that this program needs to either change substantially
• 1:39:32
  SPEAKER_00: and be more practice oriented or CS enrollment will plummet.
• 1:39:37
  SPEAKER_00: Titus also says that it might make sense to finally have
• 1:39:40
  SPEAKER_00: a software engineering curriculum that is focused
• 1:39:42
  SPEAKER_00: on students getting far more experience building software with these modern tools.
• 1:39:46
  SPEAKER_00: I don't think any of us has an answer to what will work,
• 1:39:49
  SPEAKER_00: but having more people from the industry get more involved in refreshing
• 1:39:52
  SPEAKER_00: the CS software engineer curriculum sounds like a step in the right direction to me.
• 1:39:56
  SPEAKER_00: Check out the show notes for related to Pragmatic Engineer deep dives
• 1:39:59
  SPEAKER_00: on Google's engineer culture,
• 1:40:00
  SPEAKER_00: measuring developer productivity, and the impact of AI and software engineering.
• 1:40:04
  SPEAKER_00: If you enjoyed this podcast,
• 1:40:06
  SPEAKER_00: I'd greatly appreciate it if you left a rating on it and perhaps a review.
• 1:40:09
  SPEAKER_00: Thanks a lot and see you in the next one.