Latest / The Pragmatic Engineer / Why performant code matters (but gets widely ignored), with Casey Muratori

Transcript

Why performant code matters (but gets widely ignored), with Casey Muratori · The Pragmatic Engineer

0:00
SPEAKER_01: Why do most devs not care about writing performance software?
0:03
SPEAKER_01: And should we?
0:04
SPEAKER_01: Today's guest, Casey Moratori, spent the last decade arguing that we should.
0:08
SPEAKER_01: He also says that most software out there runs 10 to 100 times slower than it needs to.
0:12
SPEAKER_01: Today we discuss why the focus on performance took a backseat across the industry
0:16
SPEAKER_01: and why Casey thinks the tide is finally turning.
0:19
SPEAKER_01: Why you'll want to learn reading assembly if you're serious
0:21
SPEAKER_01: about high performance code and why it's less scary than it sounds.
0:24
SPEAKER_01: The saying, premature optimization is the root of all evil.
0:27
SPEAKER_01: Why Casey says that the majority of people use it to avoid thinking
0:30
SPEAKER_01: about performance when they really should.
0:32
SPEAKER_01: And many more.
0:33
SPEAKER_01: If you want to get better at writing faster software and become a better engineer
0:37
SPEAKER_01: while doing so, then this episode is for you.
0:39
SPEAKER_01: And if you're one of the people who hit a like on this post by Ryan asking
0:42
SPEAKER_01: for this episode to be more about programming than about AI,
0:45
SPEAKER_01: this episode is also for you.
0:46
SPEAKER_01: This episode is presented by Antistesis.
0:49
SPEAKER_01: If you work with agents,
0:49
SPEAKER_01: your job is no longer just writing code, it's also specifying and testing it.
0:53
SPEAKER_01: Antistesis is the most effective method of verifying agentic code today.
0:57
SPEAKER_01: This episode is brought to you by Sentry.
0:59
SPEAKER_01: You probably already know what Sentry is because you're a developer.
1:01
SPEAKER_01: If not, just ask a dev and they'll tell you.
1:04
SPEAKER_01: I use Sentry to monitor the backend of The Pragmatic Engineer for any and all events and errors.
1:08
SPEAKER_01: Of course, Sentry doesn't only do errors.
1:09
SPEAKER_01: They also have logs,
1:10
SPEAKER_01: replays,
1:11
SPEAKER_01: spans, profiles, metrics, and more because they're all connected to the same trace.
1:15
SPEAKER_01: One new capability Sentry has I'm really liking is its ability to fix errors.
1:19
SPEAKER_01: Let me show you.
1:20
SPEAKER_01: Here's a list of errors on my admin backend.
1:22
SPEAKER_01: There's a recent error on auth that I want to check out.
1:24
SPEAKER_01: Let's have Sear run an autofix for us.
1:27
SPEAKER_01: Sear is Sentry's AI debugging tool.
1:29
SPEAKER_01: First, it generates a root cause analysis.
1:32
SPEAKER_01: It's finding some problem with HTTP versus HTTPS URLs.
1:35
SPEAKER_01: Cool.
1:36
SPEAKER_01: Now that we know what's going wrong, Sear can create a plan on how to go about fixing it.
1:40
SPEAKER_01: I could go and edit this plan, but I'm happy with it, so let's create the actual code fix.
1:45
SPEAKER_01: Here's the code fix that Sear generated.
1:47
SPEAKER_01: Assuming it looks good, and in my case it does, let's draft the pull request.
1:52
SPEAKER_01: And boom, the PR is created, ready to merge.
1:54
SPEAKER_01: What I love about autofix is how Senshi went from showing a list
1:57
SPEAKER_01: of errors inside my application to offering me a fast way to fix it and close
2:01
SPEAKER_01: the loop while I stay in charge of this bug fix the whole time.
2:05
SPEAKER_01: Debugging got a whole lot faster and a whole lot easier.
2:07
SPEAKER_01: Check out Sentry at sentry.io slash pragmatic and start detecting errors,
2:11
SPEAKER_01: diagnosing your root causes and fixing issues and regressions today.
2:14
SPEAKER_01: All right, Casey, welcome to the podcast.
2:17
SPEAKER_01: It's so nice to have you here.
2:19
SPEAKER_00: Thank you so much.
2:20
SPEAKER_00: It's great to be here.
2:20
SPEAKER_00: Thank you for the invitation.
2:22
SPEAKER_01: Now, I want to go back when we start to the beginning.
2:25
SPEAKER_01: How did you get into tech, programming, computers?
2:28
SPEAKER_00: Well, I guess computers, it's like very, very early on.
2:33
SPEAKER_00: My dad was a programmer at Digital Equipment Corporation,
2:36
SPEAKER_00: which is a company that people will know if they studied computer history,
2:39
SPEAKER_00: but would not know if you just looked at the landscape today.
2:42
SPEAKER_00: They're completely gone, right?
2:44
SPEAKER_00: They got absorbed partly by Intel, partly by Compaq, I think.
2:48
SPEAKER_00: They kind of got broken up.
2:51
SPEAKER_00: At that time, it was kind of a really big computer manufacturer.
2:55
SPEAKER_00: Computers like the PDP-11, that's a Digital Equipment Corporation computer.
3:00
SPEAKER_00: The VAX, like things that you may have heard of in computer history.
3:02
SPEAKER_01: Oh, these were these massive mainframes.
3:05
SPEAKER_00: Yeah, minicomputers as well,
3:07
SPEAKER_00: so like smaller also sometimes than mainframes,
3:09
SPEAKER_00: like the kind of next step down, right?
3:11
SPEAKER_00: And so in general, that era, my dad was a programmer there.
3:16
SPEAKER_00: He would later end up at Intel because, you know, like I said, parts got acquired.
3:20
SPEAKER_00: He left his job.
3:20
SPEAKER_00: He just ended up at Intel through kind of digital's eventual demise.
3:24
SPEAKER_00: But as a result,
3:26
SPEAKER_00: we always had computers at home,
3:28
SPEAKER_00: even though at that time, you know, that might have been a little bit odd.
3:34
SPEAKER_00: You know,
3:34
SPEAKER_00: I learned a program when I was seven,
3:37
SPEAKER_00: which would have been in like, you know, 1982 or something like that.
3:42
SPEAKER_00: And so at that time,
3:43
SPEAKER_00: you know,
3:43
SPEAKER_00: maybe you might have an Apple or Commodore kind of computer at home,
3:47
SPEAKER_00: like some kind of early computer.
3:50
SPEAKER_00: I don't remember the exact dates of those computers, but most people didn't.
3:54
SPEAKER_00: And it was only until a little bit later that you would.
3:56
SPEAKER_00: And you probably wouldn't have had a programmer in your household to teach you,
4:00
SPEAKER_00: more importantly, right?
4:03
SPEAKER_00: So I learned really early on, and that's when I got into computers.
4:06
SPEAKER_00: How I got into games was I ended up randomly interning at Microsoft,
4:10
SPEAKER_00: and I met people there, and I went off into games through them.
4:15
SPEAKER_00: That was how that happened, if that makes sense.
4:17
SPEAKER_01: But wait, how is the Microsoft slash games relationship?
4:20
SPEAKER_01: That's not kind of a given, right?
4:22
SPEAKER_01: Microsoft is not.
4:23
SPEAKER_01: They have one game, right?
4:25
SPEAKER_01: It's Flight Simulator.
4:27
SPEAKER_00: Yes, at that time, it would have been very weird.
4:30
SPEAKER_00: The reason that happened was a guy called Chris Hecker,
4:33
SPEAKER_00: who a lot of people don't really know his history because there was sort
4:39
SPEAKER_00: of two waves of bringing games to Microsoft Windows.
4:43
SPEAKER_00: Because, I mean,
4:43
SPEAKER_00: now it's funny to think about now because people think of like,
4:46
SPEAKER_00: what platform are you going to play games on if you have a PC?
4:49
SPEAKER_00: Windows is like the default.
4:50
SPEAKER_00: Linux is now an insurgent to that, but Windows is the default.
4:54
SPEAKER_00: And so you think about how did that happen?
4:57
SPEAKER_00: You know, because that wasn't the case if you were back in the early days.
5:02
SPEAKER_00: Microsoft Windows, not a gaming platform.
5:04
SPEAKER_00: It's almost nothing on it.
5:05
SPEAKER_00: It's like Solitaire and Minesweeper and a few other sort of games.
5:08
SPEAKER_00: How did it happen?
5:10
SPEAKER_00: So what happened, the reason for this, this kind of gets into a technological reason.
5:16
SPEAKER_00: The reason for this is that it was very hard to actually produce images
5:20
SPEAKER_00: that could be displayed on the screen quickly.
5:23
SPEAKER_00: And to understand why this is,
5:24
SPEAKER_00: is like,
5:26
SPEAKER_00: But in general,
5:27
SPEAKER_00: you can just imagine you have this operating system running,
5:31
SPEAKER_00: which is Microsoft Windows.
5:33
SPEAKER_00: It's controlling the graphics card.
5:34
SPEAKER_00: It's often running at a fairly high resolution compared to what a game might want to run at.
5:40
SPEAKER_00: You have to negotiate with it in order to display your bitmap in some way
5:43
SPEAKER_00: that won't destroy all the things that it's trying to display, so on and so forth.
5:48
SPEAKER_00: Early versions of Windows up through Windows for Workgroups,
5:53
SPEAKER_00: let's say, if anyone remembers that name.
5:55
SPEAKER_01: Was it after 3.1?
5:57
SPEAKER_00: It's 3.51, I think is what it's called.
5:59
SPEAKER_00: Or maybe just 3.1.
6:00
SPEAKER_00: Yeah, there's NT351, no.
6:02
SPEAKER_00: So it's like, I think you're right, 3.1, 3.1, I don't know, something like this.
6:05
SPEAKER_00: Windows for Workgroups was, you know, around that time.
6:09
SPEAKER_00: That version of Windows,
6:11
SPEAKER_00: which is in the 3 series,
6:12
SPEAKER_00: didn't really have a way to quickly use the CPU to fill pixels,
6:17
SPEAKER_00: which is what games need to do, right?
6:19
SPEAKER_00: There's no GPU acceleration really at this point.
6:21
SPEAKER_01: There's a little bit we could talk about, but it's not mainstream.
6:24
SPEAKER_01: It's not in consumer.
6:25
SPEAKER_00: So they need to be able to do this sort of thing.
6:27
SPEAKER_00: They need to do it in a double buffered way so they can draw to a back buffer and
6:30
SPEAKER_00: then show it to the screen.
6:31
SPEAKER_00: And that has to happen very quickly.
6:33
SPEAKER_00: And there just wasn't a way to do this in Windows.
6:35
SPEAKER_00: In Windows,
6:35
SPEAKER_00: you had to kind of go through this API where you would produce sort of a bitmap
6:40
SPEAKER_00: that wasn't necessarily in the right format for the display you were using.
6:44
SPEAKER_00: And then it had to do a translation from that bitmap to the other one
6:46
SPEAKER_00: when it displayed it, all this sort of stuff.
6:49
SPEAKER_00: So games on Windows like things like Doom, they're not coming to Windows, right?
6:53
SPEAKER_00: That kind of future was not in the works for Windows.
6:57
SPEAKER_00: And that was kind of what some people in Microsoft wanted to change.
7:00
SPEAKER_00: And one of the people who brought this change about was a guy named Chris Hecker.
7:04
SPEAKER_00: And he was like, okay, we could actually just make a library.
7:10
SPEAKER_00: That did the fast blitz to the screen so that we could have a way
7:14
SPEAKER_00: that people could do these draws and get them on the screen quick enough
7:18
SPEAKER_00: to make gaming viable.
7:20
SPEAKER_00: Would it be 100% as fast as DOS?
7:22
SPEAKER_00: Probably not.
7:23
SPEAKER_00: But could you actually run, you know, some of these new games?
7:27
SPEAKER_00: You know, Wolfenstein 3D, I think, would have been out at this time.
7:30
SPEAKER_00: Doom was kind of on the horizon and that sort of thing, right?
7:33
SPEAKER_00: And so he started this project that he was not supposed to do.
7:35
SPEAKER_00: He did not have the authority to do this called WinG.
7:40
SPEAKER_00: Which I did at least stand for like Win Graphics or something like that.
7:43
SPEAKER_00: Total Skunk Works Project.
7:45
SPEAKER_00: He had cover from his manager.
7:46
SPEAKER_00: I'm assuming I can say all this stuff now because it's ancient history.
7:49
SPEAKER_00: He had cover from his manager.
7:50
SPEAKER_00: The guy's name was Michael Edwards.
7:52
SPEAKER_00: And Michael Edwards basically just kind of ran cover for this,
7:55
SPEAKER_00: which is a thing that probably wasn't going to fly because They were in a division
7:59
SPEAKER_00: at that time, which would have been Microsoft Research today.
8:02
SPEAKER_00: It was called AT, Advanced Technology.
8:04
SPEAKER_00: It was the early version of Microsoft Research.
8:06
SPEAKER_00: You were not supposed to be shipping core libraries for Windows.
8:10
SPEAKER_00: It had nothing to do with that.
8:11
SPEAKER_00: Long story long, what ended up happening is that product did ship.
8:16
SPEAKER_00: WinG, I guess you wouldn't call it a product.
8:18
SPEAKER_00: It's an add-on for Windows.
8:19
SPEAKER_00: It did ship, and it was the first step towards DirectX.
8:24
SPEAKER_00: People forget that.
8:25
SPEAKER_00: WinG was the first way you did this.
8:27
SPEAKER_00: And then eventually we had DirectX and DibSections and Win95 and all that sort of stuff.
8:32
SPEAKER_00: When I went to Microsoft,
8:33
SPEAKER_00: amusingly,
8:34
SPEAKER_00: the person I was supposed to be reporting to as an intern was Michael Edwards.
8:40
SPEAKER_00: When I showed up first day,
8:42
SPEAKER_00: along with the other interns,
8:44
SPEAKER_00: a guy named Rudy, a guy named Rajiv, we were all supposed to report to him, right?
8:48
SPEAKER_00: Because you get a couple interns in under one, you know, sort of manager.
8:52
SPEAKER_00: We show up and we're just taken aside,
8:55
SPEAKER_00: you know,
8:55
SPEAKER_00: after some kind of,
8:56
SPEAKER_00: you know, stupid HR orientation thing that was, you know, lame as it always is.
9:01
SPEAKER_00: And we get taken aside by somebody.
9:03
SPEAKER_00: I don't remember the lyrics.
9:04
SPEAKER_00: They're like, look, bit of a problem.
9:07
SPEAKER_00: The person you interviewed with and were supposed to report to, they aren't here.
9:14
SPEAKER_00: I don't remember exactly how they put it.
9:15
SPEAKER_00: So you're going to go talk to this other guy and they'll find something for you to do.
9:19
SPEAKER_00: Turns out the WinG thing had blown up the previous week.
9:23
SPEAKER_00: And Mike Edwards had stormed out of the building and had not been seen since.
9:28
SPEAKER_00: That is what actually happened.
9:30
SPEAKER_00: So that's,
9:31
SPEAKER_00: I arrive at Microsoft,
9:32
SPEAKER_00: the person I'm supposed to intern with, he just flamed out and left.
9:36
SPEAKER_00: No one knows when he'll be back.
9:38
SPEAKER_00: He did end up.
9:38
SPEAKER_00: Oh, yeah.
10:02
SPEAKER_00: Yeah.
10:03
SPEAKER_00: He knew all these guys because working on WinG,
10:04
SPEAKER_00: he had gone out to see a bunch of game developers and like work with them
10:08
SPEAKER_00: and this sort of thing.
10:09
SPEAKER_00: So I got to go to Humongous Entertainment, which was Ron's new company.
10:13
SPEAKER_00: He gave me a secret of Monkey Island mouse.
10:17
SPEAKER_00: Mousepad, I remember.
10:18
SPEAKER_00: So anyway, it was really cool.
10:20
SPEAKER_00: And that's how I ended up getting into the game industry was through Chris Hecker.
10:24
SPEAKER_00: And he's kind of an unsung hero of getting games on Windows because,
10:27
SPEAKER_00: you know, he just he wasn't out there making his story known.
10:30
SPEAKER_00: But, you know, I am now, I guess.
10:32
SPEAKER_01: But it's so interesting to hear these stories.
10:34
SPEAKER_01: Obviously, now you can share it, I'm sure.
10:36
SPEAKER_01: You know, like for a while, this would have been like only with Yeah.
11:00
SPEAKER_00: It's more than you think because there were three people who really were
11:04
SPEAKER_00: the core people who pushed DirectX, meaning institutionally pushed it.
11:09
SPEAKER_00: There are tons of programmers like Todd Laney who did, you know, really important core work.
11:13
SPEAKER_00: It never would have shipped without people like him.
11:16
SPEAKER_00: So not on the programmer side.
11:16
SPEAKER_00: I'm talking about institutional side.
11:18
SPEAKER_00: It's Eisler, Engstrom, and Alex St. John.
11:22
SPEAKER_00: I don't remember which one.
11:22
SPEAKER_00: It's either Eisler or Engstrom.
11:25
SPEAKER_00: Was the tester on WinG.
11:28
SPEAKER_00: So he came from that team.
11:30
SPEAKER_00: So the start of DirectX, one of the core members of DirectX was on the WinG team.
11:36
SPEAKER_00: So it's a direct lineage.
11:38
SPEAKER_00: It's not even like an unrelated push.
11:40
SPEAKER_00: So WinG really was the start of it.
11:42
SPEAKER_00: And then DirectX was kind of the actual full blossoming into an org
11:47
SPEAKER_00: with Microsoft's blessing at that point,
11:49
SPEAKER_00: you know, that actually became powerful internally.
11:52
SPEAKER_01: And then after this Microsoft internship and getting exposed to all these folks,
11:56
SPEAKER_01: you actually went and built games tooling.
11:59
SPEAKER_01: You then started your own studio as well, right?
12:02
SPEAKER_01: So like how did that sequence?
12:04
SPEAKER_00: I guess there's a couple of steps in there.
12:06
SPEAKER_00: I worked at a startup with Chris Hecker that didn't end up doing anything interesting.
12:10
SPEAKER_00: Then I went to a company called Gas Powered Games,
12:13
SPEAKER_00: which was actually a Microsoft,
12:14
SPEAKER_00: I guess they're not a Microsoft studio, but their publishing deal was Microsoft.
12:18
SPEAKER_00: And they did a game called Dungeon Siege,
12:22
SPEAKER_00: which is kind of a,
12:23
SPEAKER_00: you know, I don't know, it's not a particularly well-known title.
12:26
SPEAKER_00: From there, I went to Rad Game Tools, and that's where I stayed for quite some time.
12:30
SPEAKER_00: I did their character animation system.
12:32
SPEAKER_00: That was a very popular product that ended up getting used in lots and lots of games.
12:36
SPEAKER_00: It's still used to this day,
12:37
SPEAKER_00: much to my surprise, because that's a very, very, I haven't worked on it since 2004.
12:43
SPEAKER_00: But I guess other people had maintained it and some studios just kind of integrated
12:47
SPEAKER_00: into their you could get source licenses.
12:48
SPEAKER_00: So I guess some studios just integrated it into their pipelines that have never removed it.
12:53
SPEAKER_00: And they just maybe keep updating it to to.
12:58
SPEAKER_00: So anyway, that's what I did there.
13:00
SPEAKER_00: And then afterwards, I've been independent since then.
13:02
SPEAKER_00: I just have a company called Melly Rocket where we do various stuff.
13:05
SPEAKER_00: I've done contract work for people through that.
13:09
SPEAKER_00: We now do like the Substack through that where we do stuff.
13:11
SPEAKER_00: I do educational materials.
13:12
SPEAKER_00: So I've kind of just done random stuff since then.
13:16
SPEAKER_00: Although I have done some work on games.
13:18
SPEAKER_00: I noticed your checklist.
13:18
SPEAKER_00: You're talking about The Witness.
13:20
SPEAKER_00: Obviously, that one was an actual specific title that I worked on.
13:24
SPEAKER_00: But that was mostly just because it was a very big project.
13:27
SPEAKER_00: And I'm friends with John.
13:29
SPEAKER_00: So I was just trying to do some helpful programming on the side.
13:32
SPEAKER_00: I did some stuff on how the movement system worked.
13:34
SPEAKER_00: I thought there were some interesting problems that we could solve there for games.
13:37
SPEAKER_00: And so that was a really fun project to work on.
13:40
SPEAKER_01: Yeah.
13:40
SPEAKER_01: And then today you're doing educational stuff on a bunch
13:44
SPEAKER_01: of subprogramming performance on your sub stack.
13:46
SPEAKER_01: And what else are you busy with?
13:48
SPEAKER_00: So we do actually have an unannounced project that we've been working
13:51
SPEAKER_00: on uses up sort of the rest of the time that I have,
13:55
SPEAKER_00: if that makes sense, which is not always so much.
13:58
SPEAKER_00: And that we will we're hoping to announce it sometime soon,
14:02
SPEAKER_00: but it's not We're trying to keep it fairly tight-lipped
14:04
SPEAKER_00: until we actually know we're mostly done
14:05
SPEAKER_00: because we don't know how much time we can always devote to it, if that makes sense.
14:26
SPEAKER_01: Yeah, no, it's pretty typical games related things, right?
14:31
SPEAKER_01: Like tight-lipped until you have something for good reasons.
14:34
SPEAKER_00: Well, sometimes people play the other game.
14:37
SPEAKER_00: They go like, look, we're going to, day one, we're going to be very loud about this.
14:41
SPEAKER_00: Yeah.
15:01
SPEAKER_01: And on Substack,
15:02
SPEAKER_01: it's called Computer Enhance, and you started it with performance-related topics.
15:08
SPEAKER_01: And that's how we started to talk,
15:09
SPEAKER_01: I think, about three years ago, when you already had a Substack.
15:12
SPEAKER_01: And we had a direct message conversation.
15:14
SPEAKER_01: I remember you messaged me saying, hey, Gergay, why do you think in the industry...
15:20
SPEAKER_01: People, software developers just don't really focus on performance.
15:24
SPEAKER_01: I think you specifically wrote...
15:26
SPEAKER_01: I did.
15:28
SPEAKER_01: You were saying how there's little emphasis on performance.
15:31
SPEAKER_01: Even though there seems to be overwhelming evidence that performance is critical to
15:35
SPEAKER_01: the bottom line, And I wanted to ask you, we've had a good back and forth on this.
15:40
SPEAKER_01: And actually,
15:40
SPEAKER_01: I think initially I told you, like, oh,
15:42
SPEAKER_01: here's why you don't need to care about performance when you're building,
15:44
SPEAKER_01: I don't know, distributed systems or like.
15:46
SPEAKER_01: But since then, what have you learned?
15:48
SPEAKER_01: Why do most developers don't know?
15:54
SPEAKER_00: It's a really good question.
15:56
SPEAKER_00: And I do think your answer at the time,
15:59
SPEAKER_00: if I remember it correctly,
16:00
SPEAKER_00: is certainly an accurate one for some subset of industries, which,
16:06
SPEAKER_00: you know,
16:06
SPEAKER_00: you said something along the lines of, look,
16:08
SPEAKER_00: a lot of these pieces of software that you're seeing, the user isn't the purchaser.
16:13
SPEAKER_00: Right.
16:13
SPEAKER_00: Like you were,
16:14
SPEAKER_00: you were like,
16:15
SPEAKER_00: this is some kind of thing where,
16:16
SPEAKER_00: you know,
16:17
SPEAKER_00: somebody very high up is going to look and say, we need software for managing HR.
16:21
SPEAKER_00: They're going to look at the cost of the software.
16:23
SPEAKER_00: They're going to look at the compliance terms of the software, the legal liability, whatever.
16:26
SPEAKER_00: Right.
16:28
SPEAKER_00: And I think that's very true.
16:29
SPEAKER_00: Unfortunately, the situation for a lot of enterprise
16:43
SPEAKER_00: I think that's one thing.
16:43
SPEAKER_00: There's thing two.
16:58
SPEAKER_00: Two, which is that in a lot of cases, you simply have monopoly effects.
17:04
SPEAKER_00: You know,
17:05
SPEAKER_00: people aren't right now realistically going to challenge the social networks
17:10
SPEAKER_00: that currently exist, for example.
17:13
SPEAKER_00: It's very hard,
17:14
SPEAKER_00: you know,
17:14
SPEAKER_00: Blue Sky and Threads have tried to assail X and,
17:18
SPEAKER_00: you know,
17:19
SPEAKER_00: you've got Facebook and Instagram and TikTok and they kind of just own those spaces,
17:25
SPEAKER_00: right?
17:25
SPEAKER_00: And it's very hard to push into those
17:29
SPEAKER_00: And maybe performance could be part of a package where you try to take on one
17:36
SPEAKER_00: of those players like, hey,
17:37
SPEAKER_00: look at how much more responsive our thing is than theirs might be a nice plus,
17:42
SPEAKER_00: but that's not going to be sufficient.
17:43
SPEAKER_00: If you just show up with no plan for how you get adoption,
17:46
SPEAKER_00: no plan how you get big influences over there, all that sort of stuff.
17:50
SPEAKER_00: It can't sell a product on its own into a monopoly space, right?
17:54
SPEAKER_00: If you're just talking about apps that someone can choose to download,
17:56
SPEAKER_00: maybe you've got a shot there.
17:58
SPEAKER_00: But those are just they're forming a smaller and smaller subset of what software is, right?
18:02
SPEAKER_00: And this bigger and bigger subset is like these monopoly platforms you go on to sort
18:06
SPEAKER_00: of work with.
18:07
SPEAKER_00: So I'd say that's another thing.
18:09
SPEAKER_00: Thing number three is I think now people sort of are caring about performance more.
18:12
SPEAKER_00: I think over the past decade,
18:14
SPEAKER_00: the people,
18:15
SPEAKER_00: including myself,
18:16
SPEAKER_00: but many,
18:17
SPEAKER_00: many other people who have been saying that this is
18:19
SPEAKER_00: a problem have actually had some effect.
18:22
SPEAKER_00: Like, I don't think that it was a waste of time.
18:24
SPEAKER_00: I'm seeing a lot of new emphasis on performance,
18:27
SPEAKER_00: people talking about performance, people posting benchmarks on things.
18:31
SPEAKER_00: And so I actually think that the third thing is, well,
18:35
SPEAKER_00: actually,
18:35
SPEAKER_00: it kind of does seem that Pointing at this issue
18:38
SPEAKER_00: and saying this is something we should be doing better has not been completely
18:42
SPEAKER_00: a waste of time.
18:43
SPEAKER_00: I do see things as sort of starting to turn around a little bit.
18:46
SPEAKER_00: I also see people attacking major product categories now
18:50
SPEAKER_00: with performance-based pitches,
18:51
SPEAKER_00: things like FilePilot or the Blick Video Editor,
18:54
SPEAKER_00: like things like this that have been coming out lately,
18:56
SPEAKER_00: where it's like, oh,
18:57
SPEAKER_00: really performance software to try to take on incumbents in a space,
19:01
SPEAKER_00: and they've been getting traction.
19:02
SPEAKER_00: So I think that's also a really good sign.
19:04
SPEAKER_01: Yeah, and I guess on this last category,
19:06
SPEAKER_01: a really good example in the developer community is BUN versus NPM,
19:10
SPEAKER_01: where BUN just said like, okay, we're like 10x or 20x or 50x faster.
19:16
SPEAKER_01: And devs are like, what?
19:18
SPEAKER_01: Is this possible?
19:19
SPEAKER_01: And it was so they there was this outrageous claim.
19:23
SPEAKER_01: I almost I wonder if like you need to have these outrageous claims
19:26
SPEAKER_01: because dev started to pay attention because it was 10x faster in many categories.
19:32
SPEAKER_01: You know,
19:32
SPEAKER_01: linear versus was gyra is also a good example where linear have this benchmark of,
19:38
SPEAKER_01: OK, they have 300 milliseconds for any action.
19:40
SPEAKER_01: And gyra, of course, we know is is just slow because they have a bunch of complexity.
19:46
SPEAKER_01: We can explain why, but it's slow.
19:47
SPEAKER_01: It was never built for that.
19:49
SPEAKER_00: Yeah.
19:49
SPEAKER_00: And, you know,
19:50
SPEAKER_00: if you think about something like a 300 millisecond budget for an operation,
19:53
SPEAKER_00: 300 milliseconds is like an eternity in computing.
19:56
SPEAKER_00: Right.
19:57
SPEAKER_00: And so if you're talking about like our pitches that were not more
20:00
SPEAKER_00: than 300 milliseconds,
20:02
SPEAKER_00: that just shows you how the bar was so far past where it probably should have been
20:08
SPEAKER_00: for something.
20:08
SPEAKER_00: And you see it everywhere.
20:09
SPEAKER_00: You know, you go on to programs and you're waiting sometimes.
20:12
SPEAKER_00: And I don't think people realize just what an eternity a second is
20:18
SPEAKER_00: in modern computing,
20:19
SPEAKER_00: especially when you're sitting on networking like that has,
20:22
SPEAKER_00: you know, sub 10 millisecond ping times.
20:27
SPEAKER_00: This, you know, the actual packets had to travel physical distance to get to this data center.
20:32
SPEAKER_00: And that was being done far faster than this very simple operation
20:36
SPEAKER_00: that you were failing to do in a reasonable amount of time.
20:37
SPEAKER_00: It's just like we are massively underperforming.
20:40
SPEAKER_00: And people don't believe it when you say 10x, 100x, but it's actually true.
20:43
SPEAKER_00: And we've seen a lot of proof of it, as you point out.
20:46
SPEAKER_01: I do wonder if one part of the not really much focus on performance is that a lot
20:51
SPEAKER_01: of developers don't know the baseline thing.
20:53
SPEAKER_01: And I'm reminded by Simon Eriksen,
20:56
SPEAKER_01: the founder of TurboBuffer,
20:57
SPEAKER_01: has this project called Napkin Math,
20:59
SPEAKER_01: where he did a And so he had these numbers and he said that what he found is...
21:22
SPEAKER_01: Whenever inside Shopify, they were deciding, do we choose vendor A or vendor B as a database?
21:27
SPEAKER_01: They would just run a benchmark that they would write themselves
21:32
SPEAKER_01: and they would get like, okay, like,
21:33
SPEAKER_01: I don't know,
21:33
SPEAKER_01: storing this and this,
21:35
SPEAKER_01: it takes two seconds on this one,
21:36
SPEAKER_01: 10 seconds on that one, we will choose a two second one.
21:38
SPEAKER_01: And he looked at it and said, like, hey, like, this doesn't make sense.
21:42
SPEAKER_01: Like, the amount to store in a file system,
21:45
SPEAKER_01: like, here's a theoretical limit, which is, I don't know, 100 milliseconds.
21:48
SPEAKER_01: Like, there's no way that's going to be 10 seconds.
21:50
SPEAKER_01: And it often turns out that he found that the benchmark was just wrong.
21:54
SPEAKER_01: They were benchmarking the wrong thing and they were making decisions.
21:56
SPEAKER_01: So I wonder if there's a thing where many engineers,
22:00
SPEAKER_01: developers are maybe just not aware
22:02
SPEAKER_01: of how truly devastatingly slow this thing is versus the resources you have.
22:08
SPEAKER_00: That is the entire point of my Substack, right?
22:11
SPEAKER_00: So what you just said is exactly true.
22:14
SPEAKER_00: And it is the thing that I hammer home on the Substack through all the parts of
22:18
SPEAKER_00: the courses on there,
22:19
SPEAKER_00: which is that in general,
22:21
SPEAKER_00: there's a misconception about the way that you approach optimization
22:25
SPEAKER_00: in computer science or in software
22:28
SPEAKER_01: engineering, let's say.
22:29
SPEAKER_00: And that misconception is that what you do is you run a profile,
22:34
SPEAKER_00: you identify where the big parts of the profile are,
22:38
SPEAKER_00: you make some changes to those, and you measure statistics.
22:42
SPEAKER_00: The more statistics you can get, the better.
22:47
SPEAKER_00: And you look to see if those statistics improved.
22:49
SPEAKER_00: If they have, that was a good change.
22:50
SPEAKER_00: And you proceed as such.
22:52
SPEAKER_00: And this is completely not correct.
22:55
SPEAKER_00: That is not how anyone has ever,
22:57
SPEAKER_00: you know,
22:57
SPEAKER_00: I've worked with many extremely good optimization people,
23:00
SPEAKER_00: and that is not how it is done.
23:03
SPEAKER_00: The correct way to do optimization is very much like what you just said.
23:07
SPEAKER_00: You first go, what are the operations that this system has to perform?
23:11
SPEAKER_00: What is the underlying hardware capable of doing at its theoretical peak?
23:15
SPEAKER_00: And then you measure the delta between that theoretical maximum and what you have achieved.
23:22
SPEAKER_00: And then your goal during optimization is to shrink that gap to something
23:26
SPEAKER_00: that you think could be plausibly explained and hopefully come up with explanations
23:30
SPEAKER_00: of why you aren't at theoretical.
23:32
SPEAKER_00: Because oftentimes you can't hit theoretical.
23:33
SPEAKER_00: That's why we call it theoretical, right?
23:35
SPEAKER_00: And it's crucial that you do this because otherwise all you're doing with
23:39
SPEAKER_00: that other method is finding a,
23:41
SPEAKER_00: you know,
23:42
SPEAKER_00: with the,
23:42
SPEAKER_00: I'm just going to,
23:43
SPEAKER_00: you know,
23:43
SPEAKER_00: make something I think might be an optimization and look if my statistics improved.
23:46
SPEAKER_00: All you're doing there is finding a local minima.
23:49
SPEAKER_00: That's all you're doing.
23:51
SPEAKER_00: You've got this shape of your performance and you're finding some little spot
23:55
SPEAKER_00: and you're sitting in it.
23:56
SPEAKER_00: That's not optimization.
23:57
SPEAKER_00: That's improvement.
23:59
SPEAKER_00: But optimization means to make optimal, right?
24:01
SPEAKER_00: It means we're going to find what we actually should be able to get this machine to do.
24:06
SPEAKER_00: And so, you know,
24:08
SPEAKER_00: that's why I emphasize that approach because it's the one
24:10
SPEAKER_00: that I've always seen great optimizers take.
24:13
SPEAKER_00: That is how they get good performances by knowing what the maximum could be.
24:17
SPEAKER_00: In addition to that, it also is what lets you become better Yeah.
24:36
SPEAKER_00: New things that people have not figured out about modern CPUs,
24:39
SPEAKER_00: new things that are different about the network backplane,
24:42
SPEAKER_00: new things that are different about the GPU drivers, who knows, right?
24:45
SPEAKER_00: And if you don't have some theoretical maximum to look at and to measure your delta
24:50
SPEAKER_00: from, you don't know We know if there's some serious anomaly there.
24:53
SPEAKER_00: And you would be surprised at how many times we find anomalies like this.
24:59
SPEAKER_00: Things in CPUs that no one knew about,
25:01
SPEAKER_00: and I've literally had them in the course of making the Substack.
25:05
SPEAKER_00: I've been like, what is this?
25:06
SPEAKER_00: And I look into it,
25:07
SPEAKER_00: it's like, oh,
25:07
SPEAKER_00: there's this new renaming,
25:09
SPEAKER_00: this new rat table thing that Intel chips seem to be able to do.
25:12
SPEAKER_00: We didn't know about that.
25:14
SPEAKER_00: And that's like a new thing we have to model when we talk about how to do performance.
25:18
SPEAKER_00: And so that's the other crucial part of, I guess, what you were calling napkin math.
25:22
SPEAKER_00: I also call it back of the envelope.
25:24
SPEAKER_00: That's the term I've heard used for it.
25:25
SPEAKER_00: Oftentimes they're kind of interchangeable, right?
25:27
SPEAKER_00: Knowing what the theoretical is,
25:29
SPEAKER_00: is how you learn as well,
25:31
SPEAKER_00: how you learn about new hardware and new performance options.
25:34
SPEAKER_01: Interesting.
25:35
SPEAKER_01: Plus, by doing this, you're just learning.
25:37
SPEAKER_01: You're becoming a better professional.
25:38
SPEAKER_01: You understand more about given hardware or the inner workings of your computer
25:45
SPEAKER_01: or software stack or kernel,
25:47
SPEAKER_01: all the stuff that I guess goes way beyond The vanilla programming language,
25:52
SPEAKER_01: because you can say I'm an engineer,
25:54
SPEAKER_01: I'm a software engineer because I know how to use this programming language,
25:56
SPEAKER_01: but I'd argue you're probably an engineer if you can go down the stack and you have
26:00
SPEAKER_01: that ability and you have a good understanding of some of it at least, right?
26:04
SPEAKER_01: And you can learn the rest.
26:06
SPEAKER_00: And I would also say that one of the other things that we do in
26:08
SPEAKER_00: the class is teach how to read assembly language.
26:10
SPEAKER_00: And people often ask, like, you know, assembly language, what would I ever need that for?
26:15
SPEAKER_00: There's a very good reason for it.
26:17
SPEAKER_00: And that is that everything else that you might use doesn't tell you anything
26:21
SPEAKER_00: about what the CPU is actually receiving.
26:24
SPEAKER_00: You know,
26:24
SPEAKER_00: if I look at a Java program,
26:26
SPEAKER_00: if I look at a C program,
26:27
SPEAKER_00: if I look at Haskell,
26:29
SPEAKER_00: OCaml, whatever, right, Rust, all I'm seeing is input to a compiler.
26:34
SPEAKER_00: I have no idea what the CPU is actually going to be asked to do.
26:38
SPEAKER_00: If I look at the assembly language output from that compiler,
26:41
SPEAKER_00: I know exactly what the CPU is being asked to do.
26:44
SPEAKER_00: And it's not that hard to be able to learn to read assembly language so
26:48
SPEAKER_00: that you can see very quickly,
26:51
SPEAKER_00: is the CPU being asked to do the things that I think it should be asked to do them
26:55
SPEAKER_00: and in that way, right?
26:57
SPEAKER_00: You don't have to write it hardly ever.
26:59
SPEAKER_00: It's very rare that you have to write assembly language to do anything other
27:03
SPEAKER_00: than sometimes for test purposes, it's easier to do that.
27:06
SPEAKER_00: So you don't have to try and convince the compiler to output something.
27:08
SPEAKER_00: So if you're just testing something,
27:10
SPEAKER_00: sometimes it helps to be able to write some assembly language.
27:12
SPEAKER_00: But if you're just talking about the vast majority of tasks you might do
27:16
SPEAKER_00: in optimization, writing it, no.
27:18
SPEAKER_00: Reading it, essential.
27:20
SPEAKER_00: And it also unlocks this sort of huge world of possibilities to you
27:24
SPEAKER_00: because once you know assembly language,
27:27
SPEAKER_00: you can now do things like read those CPU diagrams.
27:29
SPEAKER_00: Like, you know, when they announce a new processor, they put up a little diagram.
27:33
SPEAKER_00: That diagram tells you stuff like the fastest this thing could do multiplication
27:36
SPEAKER_00: and stuff like that.
27:36
SPEAKER_00: It tells you that if you know assembly language, you can read it right off the chart, right?
27:40
SPEAKER_00: If you don't know assembly language,
27:41
SPEAKER_00: you look at that chart and like I have no idea to do what I'm looking at, right?
27:44
SPEAKER_00: Like it's just this weird flowchart that doesn't really tell me anything, right?
27:47
SPEAKER_00: And so one of the really great things about assembly language is it unlocks all
27:49
SPEAKER_00: of this knowledge for you because it's the actual input language to the machine
27:55
SPEAKER_00: and it allows you to figure out how it's operating.
27:57
SPEAKER_01: Plus, I guess we should add that assembly language is not all that complicated, right?
28:02
SPEAKER_01: Just by nature.
28:04
SPEAKER_01: It's a far simpler language.
28:05
SPEAKER_01: OK, it's harder to read if you've never seen it,
28:08
SPEAKER_01: but it's in terms of the number of operations,
28:11
SPEAKER_01: it's so bare bones because that's what assembly is like.
28:15
SPEAKER_01: Every single higher level language will have way more keywords,
28:18
SPEAKER_01: structure, whatever you name it, right?
28:19
SPEAKER_01: Than assembly.
28:20
SPEAKER_00: Massively more, and especially when you consider the subset that are actually used.
28:25
SPEAKER_00: If you look at the subset of constructs that you would need to understand to be able
28:29
SPEAKER_00: to understand, say,
28:31
SPEAKER_00: just a website from today,
28:33
SPEAKER_00: all of the JavaScript libraries,
28:34
SPEAKER_00: all of the JavaScript syntax,
28:36
SPEAKER_00: all of the DOM, you know, all of the behavior that's going to go on the CSS.
28:40
SPEAKER_00: All of the React.
28:41
SPEAKER_00: React, CSS, right?
28:43
SPEAKER_00: All of that.
28:44
SPEAKER_00: Assembly language,
28:45
SPEAKER_00: you know,
28:46
SPEAKER_00: maybe there's 20,
28:47
SPEAKER_00: 30 instructions you might have to learn total because most assembly,
28:51
SPEAKER_00: most things in legacy assembly like x64,
28:54
SPEAKER_00: most of them are hardly ever output by the compiler.
28:56
SPEAKER_00: So you only need to learn a very small subset.
28:58
SPEAKER_00: That's the ones that are actually going to be,
29:00
SPEAKER_00: that you're going to be seeing in 90% of the cases.
29:02
SPEAKER_00: It's so much simpler.
29:03
SPEAKER_00: And also when you're looking at performance, you're typically only looking at a very small part.
29:09
SPEAKER_00: You've kind of understood roughly what's going on.
29:11
SPEAKER_00: You've seen the basic layouts of your program.
29:13
SPEAKER_00: You've identified what's supposed to be happening.
29:15
SPEAKER_00: And you're just looking to see like, wait,
29:16
SPEAKER_00: why is this part,
29:17
SPEAKER_00: which I don't think should be running this slowly, why is it running this slowly?
29:20
SPEAKER_00: It's just a very small piece you typically end up having to look at as well.
29:24
SPEAKER_00: So it's really much easier if you can understand how to center a div, as they say.
29:29
SPEAKER_00: If you can vertically center a div in HTML,
29:31
SPEAKER_00: then you can probably learn assembly language, I would say.
29:34
SPEAKER_01: Okay.
29:35
SPEAKER_01: You're super passionate about performance optimization.
29:39
SPEAKER_01: You also have really good educational materials,
29:41
SPEAKER_01: both free videos, your paid Substack, the free parts of it, etc.
29:44
SPEAKER_01: But let me just play devil's advocate.
29:46
SPEAKER_01: There's a saying that premature optimization is the root of all evil.
29:50
SPEAKER_01: And we typically use it or I typically use it so many times.
29:53
SPEAKER_01: We're like, oh, should we make this performance?
29:55
SPEAKER_01: Should we optimize the thing?
29:56
SPEAKER_01: And like, nah, let's not do that.
29:57
SPEAKER_01: Let's first build it.
29:59
SPEAKER_01: Let's see if it's good enough for our customers, for ourselves.
30:01
SPEAKER_01: And if we need to, we can always optimize it.
30:03
SPEAKER_01: I mean, you know, like it's not the hardest thing in the world.
30:06
SPEAKER_01: OK, maybe not as good as how you mentioned, because maybe I don't really assembly.
30:09
SPEAKER_01: But and that's that's kind of thinking of building,
30:12
SPEAKER_01: you know, like a SaaS software, building software at Big Tech.
30:15
SPEAKER_01: What is your reply to that?
30:17
SPEAKER_01: Because I feel really good that I made a really good argument here.
30:20
SPEAKER_00: So I guess what I would say is the important part about that,
30:25
SPEAKER_00: and I guess I'll divorce it a little bit from the saying.
30:28
SPEAKER_00: I have an entire lecture on that saying, by the way.
30:30
SPEAKER_00: It's like two hours long, and I gave it at Better Software Conference this year.
30:35
SPEAKER_00: I believe the VOD will be coming out.
30:36
SPEAKER_00: We're linking that in the show
30:37
SPEAKER_01: notes below.
30:38
SPEAKER_00: Okay.
30:38
SPEAKER_00: It'll be like a week or two, I think, till it's up.
30:40
SPEAKER_00: So it may be right at the same time as this.
30:43
SPEAKER_00: But so if you want to find out the history of that phrase, you can go look at that.
30:47
SPEAKER_00: But I wanted to talk about the idea behind it because I think there's,
30:52
SPEAKER_00: I don't So here's the positive side of that first.
30:54
SPEAKER_00: The positive side of that first is for some types of code that will work.
31:16
SPEAKER_00: If you happen to have written some operation poorly,
31:20
SPEAKER_00: where the optimized version of that operation just looks like someone taking a loop
31:26
SPEAKER_00: and changing the loop from your really like,
31:28
SPEAKER_00: you know, naive version to a really well optimized version.
31:31
SPEAKER_01: The typical of like, I wrote a bubble sort, we can later optimize that.
31:36
SPEAKER_00: Who knows, right?
31:37
SPEAKER_00: Anything of that form.
31:38
SPEAKER_00: Okay, maybe we can just do that.
31:40
SPEAKER_00: So there are certain times where you do in your head want to be doing this,
31:45
SPEAKER_00: where you want to say, okay,
31:47
SPEAKER_00: I could go spend a week researching the fastest hash table implementation here.
31:52
SPEAKER_00: But part of software engineering is being smart enough to know it won't matter
31:57
SPEAKER_00: if I do that now or later.
31:58
SPEAKER_00: The architecture around this piece won't have to change.
32:02
SPEAKER_00: I'm quite certain because I understand the problem well enough.
32:05
SPEAKER_00: So it's OK.
32:06
SPEAKER_00: I can defer that to later.
32:08
SPEAKER_00: Maybe it's never too slow with the naive one I put in there and then we don't have
32:11
SPEAKER_00: to do any work.
32:12
SPEAKER_00: Maybe it's too slow later.
32:13
SPEAKER_00: That's OK.
32:14
SPEAKER_00: I just target this one hash table implementation and we'll get as fast as we need.
32:18
SPEAKER_00: Right.
32:19
SPEAKER_00: If you're doing that,
32:20
SPEAKER_00: if you're applying that true engineering mentality to it, you don't have a problem.
32:25
SPEAKER_00: The problem comes when you don't know if the choice that you're making produces
32:31
SPEAKER_00: that kind of optimizable hotspot.
32:34
SPEAKER_00: And I'll give you a very simple example that usually people have had experience with.
32:39
SPEAKER_00: A very simple example would be we write our entire software thing,
32:43
SPEAKER_00: like we just,
32:43
SPEAKER_00: whatever this massive thing that we're imagining doing where we're going
32:46
SPEAKER_00: to ignore optimization.
32:47
SPEAKER_00: We sit down and we write it and we use a paradigm where we ask the server for something.
32:53
SPEAKER_00: We have like some API You know, that we've built for asking servers for things.
32:57
SPEAKER_00: We ask the server and it returns to us what the server's response was.
33:01
SPEAKER_00: And that's like kind of how we architect this thing.
33:04
SPEAKER_00: So everyone writes,
33:05
SPEAKER_00: you know,
33:06
SPEAKER_00: hundreds of thousands or millions of lines of code and they all look like,
33:08
SPEAKER_00: ask the server something,
33:09
SPEAKER_00: do some calculations,
33:10
SPEAKER_00: ask the server for the next thing, do some calculations, right?
33:12
SPEAKER_00: Then at the end, you find this is way too slow, but that's okay.
33:16
SPEAKER_00: You weren't worried about that because you're like, we're baptized at the end.
33:18
SPEAKER_00: You call in some performance experts.
33:19
SPEAKER_00: They look at it and they go, there's nothing we can do for you, sorry.
33:22
SPEAKER_00: Why?
33:22
SPEAKER_00: Well, the reason is because you created a serial dependency chain.
33:27
SPEAKER_00: All of your code looks like wait for a network request to come back, do something.
33:32
SPEAKER_00: Wait for a network request to come back, do something.
33:35
SPEAKER_00: And that serial dependency chain can't really be shortened without just rewriting it.
33:41
SPEAKER_00: If instead you had made the paradigm and told your programmers,
33:45
SPEAKER_00: look, here's what you need to do.
33:46
SPEAKER_00: At the top of every operation,
33:47
SPEAKER_00: you need to figure out all the things you might want to ask the server for.
33:50
SPEAKER_00: You ask them for all of those things, right?
33:54
SPEAKER_00: And then you do all of your processing there and you only Create a chain
33:57
SPEAKER_00: of dependencies if you absolutely couldn't have determined what it was you needed
34:02
SPEAKER_00: to ask the server for.
34:03
SPEAKER_00: Now you're just in this situation because you didn't tell them to do that.
34:05
SPEAKER_00: You have to rewrite all your code.
34:07
SPEAKER_00: Everyone is now going out,
34:08
SPEAKER_00: rewriting all the code,
34:09
SPEAKER_00: if they even can,
34:10
SPEAKER_00: if it's even possible to really do that in a way that's not slower
34:15
SPEAKER_00: than just rewriting the thing, right?
34:16
SPEAKER_00: So what happened there?
34:17
SPEAKER_00: Well, again,
34:17
SPEAKER_00: we talk about this a lot on the Substack,
34:19
SPEAKER_00: but there's this idea of a serial dependency chain.
34:21
SPEAKER_00: It's when you stack things in order, right?
34:24
SPEAKER_00: And the performance of your software is generally determined by
34:27
SPEAKER_00: the longest serial dependency chain because it's something
34:30
SPEAKER_00: that cannot be parallelized.
34:31
SPEAKER_00: If I have thing A that then B depends on,
34:34
SPEAKER_00: that then C depends on that, we cannot shorten that because it has to go in order.
34:39
SPEAKER_00: Everything waits for it.
34:40
SPEAKER_00: We can't multi-thread it because it's dependent.
34:42
SPEAKER_00: We can't make it run wide.
34:44
SPEAKER_00: We can't amortize the network request, whatever.
34:48
SPEAKER_00: That kind of thing can be pervasive in the programming
34:51
SPEAKER_00: and we can't cheaply remove it because it's not a hotspot.
34:54
SPEAKER_00: It's a way that you did things.
34:55
SPEAKER_00: That's the part where that kind of thinking breaks down.
34:58
SPEAKER_00: If every software engineer knew to watch out for false serial dependency chains,
35:05
SPEAKER_00: things where they were creating series of dependent operations
35:08
SPEAKER_00: that could not be optimized the way or other sorts of architectural problems like
35:12
SPEAKER_00: that that cannot be easily fixed,
35:14
SPEAKER_00: Then the world would look more like just wait and optimize the hotspot, right?
35:19
SPEAKER_01: Yeah.
35:20
SPEAKER_01: So this is the architecture, the planning, right?
35:22
SPEAKER_01: Like if in that phase,
35:23
SPEAKER_01: you're like,
35:23
SPEAKER_01: OK, like as this thing grows, like what would get in the way of performance?
35:28
SPEAKER_01: What would slow it down?
35:29
SPEAKER_01: Or you can ask all these questions or like different flavors of these questions,
35:32
SPEAKER_01: you know, or from the other side and so on.
35:34
SPEAKER_00: Another way to think of it is because hotspots is the way that people talk about that.
35:38
SPEAKER_00: Like, oh, it's going to be hotspot optimization.
35:40
SPEAKER_00: We just got a few spikes.
35:41
SPEAKER_00: Someone will come in and clean up those spikes and we're done, right?
35:44
SPEAKER_00: The way to think about it is your codebase will not end up that way by accident
35:47
SPEAKER_00: in most cases anymore.
35:49
SPEAKER_00: You have to engineer up front for a hotspot code base that people can then optimize, right?
35:56
SPEAKER_00: And so that's the crucial takeaway is everybody on your team
36:01
SPEAKER_00: who is making architectural decisions,
36:04
SPEAKER_00: those people must know performance and they must make decisions that will allow
36:08
SPEAKER_00: the other people
36:13
SPEAKER_00: Casey just
36:19
SPEAKER_01: talked about how engineers making architectural decisions should know
36:21
SPEAKER_01: about performance.
36:22
SPEAKER_01: This is also true when choosing your dependencies like which database to use.
36:26
SPEAKER_01: And this is where I want to mention our season sponsor, TurboBuffer.
36:29
SPEAKER_01: You already know how TurboBuffer is a vector and full-text search engine.
36:32
SPEAKER_01: But here's an interesting story from Linear on what happens when you stop thinking
36:36
SPEAKER_01: of TurboBuffer as a search engine and start using it as a primitive
36:39
SPEAKER_01: to reduce latency.
36:40
SPEAKER_01: As context, linear is a local first app.
36:43
SPEAKER_01: So each client keeps a local database,
36:45
SPEAKER_01: and when that client goes back online,
36:47
SPEAKER_01: it needs to catch up with what happened and do it fast.
36:50
SPEAKER_01: Their biggest workspaces generate around a million sync actions per day.
36:54
SPEAKER_01: Doing catch-ups by reading from Postgres was getting slow for large reads,
36:58
SPEAKER_01: so the tail latency got too large.
37:01
SPEAKER_01: And adding more replicas did not help either.
37:03
SPEAKER_01: Linear solved the problem cleverly.
37:05
SPEAKER_01: They started using TurboPuffer as a serving index for each client.
37:08
SPEAKER_01: This is because TurboPuffer Itself is built on top of inverted indexes.
37:12
SPEAKER_01: So for every index value, it stores the documents that that value can be found in.
37:16
SPEAKER_01: The lookup cost for such an index is constant.
37:19
SPEAKER_01: So linear took this structure and had each client's index point to the changes
37:23
SPEAKER_01: that they needed to sync.
37:24
SPEAKER_01: As a result,
37:25
SPEAKER_01: not only did they reduce latency,
37:27
SPEAKER_01: but they kept it constantly low no matter how long the change list is synced to
37:30
SPEAKER_01: the client is.
37:31
SPEAKER_01: Linear published a blog post
37:33
SPEAKER_01: about this refactor titled Rebuilding Linear's Delta Sync Read Path.
37:36
SPEAKER_01: Check it out.
37:37
SPEAKER_01: I love this story because it shows how important it is to choose
37:40
SPEAKER_01: the right primitives and how good primitives can improve your system.
37:43
SPEAKER_01: If you're building systems where you store a lot of data or serve a lot of data,
37:47
SPEAKER_01: TurboPuffer can probably speed things up or save on your costs.
37:50
SPEAKER_01: Learn more at turbopuffer.com slash pragmatic.
37:53
SPEAKER_01: I'd also like to talk about our presenting sponsor and this is this.
37:56
SPEAKER_01: While Casey deliberately does not use AI coding agents for his work, most of us do.
38:00
SPEAKER_01: And when you work with coding agents,
38:02
SPEAKER_01: your job is no longer writing code, it's specifying and testing it.
38:06
SPEAKER_01: Antisysys is the most effective method for verifying a genetic code today.
38:09
SPEAKER_01: Let me explain how it works.
38:10
SPEAKER_01: Antisysys runs your whole system in a hostile simulation.
38:13
SPEAKER_01: By doing so, it finds every bug before your users do.
38:16
SPEAKER_01: And because the simulation is fully deterministic,
38:19
SPEAKER_01: Antisysys doesn't only find bugs,
38:20
SPEAKER_01: it gives you a perfect reproduction of every issue.
38:23
SPEAKER_01: To create such a tool, the Antisysys team needed to invent new kinds of debugging tools as well.
38:27
SPEAKER_01: For example, here's what's called a bug probability graph.
38:30
SPEAKER_01: The x-axis is virtual time, and the y-axis is probability.
38:34
SPEAKER_01: As Antithesis runs the hostile simulations,
38:36
SPEAKER_01: it plots timeframes when the bug probability increases,
38:39
SPEAKER_01: which greatly helps with finding the root cause of the bugs.
38:42
SPEAKER_01: And Antithesis also has a log visualizer.
38:44
SPEAKER_01: Vertical lines going down represent events branching off from the same state.
38:48
SPEAKER_01: And the purple dots are where the debug happens.
38:51
SPEAKER_01: Antisysys is as good as it gets in being able to ship agent-written code.
38:54
SPEAKER_01: It's what teams at Janestreet, Fly.io, and the etcd community use to ship with full confidence.
38:59
SPEAKER_01: Head to antisysys.com slash pragmatic to learn more.
39:02
SPEAKER_01: And with this,
39:02
SPEAKER_01: let's get back to Casey and how,
39:04
SPEAKER_01: if you don't design an architecture that can be performance optimized later,
39:07
SPEAKER_01: you're just rolling the dice.
39:09
SPEAKER_00: And we've seen so many projects.
39:11
SPEAKER_00: I have an entire video where I go through, like,
39:14
SPEAKER_00: look at all these blog posts of people who,
39:15
SPEAKER_00: like, say, you know, it's Facebook, it's Uber, it's everybody.
39:19
SPEAKER_00: They've got blog posts of we had to rewrite this whole thing because the performance was bad.
39:23
SPEAKER_00: If it was always hotspots that made your performance bad,
39:25
SPEAKER_00: you'd never have to rewrite the whole thing.
39:27
SPEAKER_00: So we know that that doesn't work anymore.
39:29
SPEAKER_00: Why?
39:30
SPEAKER_00: Because of the things I just said.
39:31
SPEAKER_01: I was at Uber where I was not making the decision,
39:34
SPEAKER_01: but the teams next to me were and I saw or I kind
39:37
SPEAKER_01: of understood why they were making.
39:38
SPEAKER_01: But typically,
39:39
SPEAKER_01: and right now it's happening with AI companies,
39:41
SPEAKER_01: oftentimes it's like we chose this technology,
39:44
SPEAKER_01: which is Python, and it's single threaded.
39:46
SPEAKER_01: And it made sense at the time on the web server.
39:48
SPEAKER_01: But now we're big.
39:49
SPEAKER_01: And this happened at Uber.
39:50
SPEAKER_01: It was Python and Node.js.
39:52
SPEAKER_01: And then they went to Go and Java on the back end.
39:55
SPEAKER_01: And now with AI companies,
39:56
SPEAKER_01: it was Python, OpenAI, and Entropic are both going through this right now.
40:01
SPEAKER_01: They're both either public about it or everything with Entropic.
40:04
SPEAKER_01: They share with me, but I put it out there.
40:07
SPEAKER_01: They used Python because data scientists or AI or machine learning engineers knew Python.
40:12
SPEAKER_01: They put in a bunch of web servers.
40:13
SPEAKER_01: They had their API run on it.
40:14
SPEAKER_01: Initially, they just scaled horizontally.
40:16
SPEAKER_01: But now they're like, well,
40:18
SPEAKER_01: if we move over to Rust,
40:20
SPEAKER_01: right now they're choosing Rust or Go, but I think it's Rust.
40:23
SPEAKER_01: Well, we can actually have multithreaded and
40:25
SPEAKER_01: The same machine can actually handle more connections.
40:28
SPEAKER_01: So cool.
40:29
SPEAKER_01: I came across a lot of that because I think that's easy and safe to communicate
40:34
SPEAKER_01: because it doesn't look bad on you.
40:35
SPEAKER_01: But you're right.
40:36
SPEAKER_01: A lot of times,
40:38
SPEAKER_01: I don't think on entering blog posts,
40:39
SPEAKER_01: you'll get the real reason that these companies put out there.
40:44
SPEAKER_01: When it's kind of a very kind of,
40:45
SPEAKER_01: you know,
40:46
SPEAKER_01: easy to own mistake or not mistake,
40:48
SPEAKER_01: but just the decision, which made sense, they'll tell you.
40:51
SPEAKER_01: But if it's something that was an oversight,
40:53
SPEAKER_01: you're not really going to get that on a public facing engineering blog post,
40:56
SPEAKER_01: except for maybe some startups who are really But don't forget,
41:00
SPEAKER_01: a lot of those blog posts are going to help someone get promoted or get recognized.
41:03
SPEAKER_01: And they will always be way more positive,
41:06
SPEAKER_01: especially when there's a content writer team, which large companies do have.
41:10
SPEAKER_01: So it's not quite PR, but it's somewhere midway in between.
41:15
SPEAKER_00: And I mean, yes.
41:16
SPEAKER_00: And also I would just point out the fact that like the fact
41:20
SPEAKER_00: that these things are happening,
41:22
SPEAKER_00: though, is all we really need to know for the signal.
41:26
SPEAKER_00: Right.
41:26
SPEAKER_00: Because in general, this should not be happening.
41:29
SPEAKER_00: If the the ideas about optimization were true,
41:32
SPEAKER_00: you'd never have to rewrite something in a language in a different language
41:35
SPEAKER_00: unless you just preferred that language.
41:38
SPEAKER_00: It would just the story would just be.
41:40
SPEAKER_00: We rewrote it in this language because we wanted to use this new language.
41:43
SPEAKER_00: It would never be, or for Rust, it might be just memory safety.
41:46
SPEAKER_00: We see those blog posts, right?
41:47
SPEAKER_00: It's like, why did we write into Rust?
41:48
SPEAKER_00: It wasn't performance.
41:49
SPEAKER_00: It was just we wanted the memory safety or something like that.
41:51
SPEAKER_01: If I'm a...
41:53
SPEAKER_01: Software Engineer Programmer.
41:55
SPEAKER_01: And I'd like to just get better at writing performance code.
42:00
SPEAKER_01: I'm interested,
42:00
SPEAKER_01: you know,
42:01
SPEAKER_01: maybe after this podcast or looking to some of the things that you did,
42:05
SPEAKER_01: what is a learning path you would follow outside of your Substack where you cover
42:09
SPEAKER_01: a lot of these things?
42:09
SPEAKER_01: But what are areas that you think are kind of like you need
42:12
SPEAKER_01: to understand these things to like get better at writing performance code?
42:16
SPEAKER_00: I think it's actually very simple and perhaps a little bit counterintuitive.
42:23
SPEAKER_00: So I'll start with the very good news about learning to write performance software.
42:30
SPEAKER_00: The good news is that optimization of the kind that we sort of talked about,
42:36
SPEAKER_00: the like hotspot kind,
42:38
SPEAKER_00: where it's like somebody is going to go in here,
42:41
SPEAKER_00: maybe they're going to even rewrite this routine in hand coded assembly
42:44
SPEAKER_00: or something crazy like this, right?
42:46
SPEAKER_00: That's very rarely necessary these days.
42:50
SPEAKER_00: One of the reasons that you don't see hotspot optimization as a thing
42:54
SPEAKER_00: that really matters that much anymore,
42:56
SPEAKER_00: and one of the reasons I advise that architecture
42:59
SPEAKER_00: and not making bad decisions is much more important.
43:02
SPEAKER_00: Is because a lot of libraries already have been optimized for you that you might use.
43:07
SPEAKER_00: CPUs are incredibly good at taking bad code and running it quickly and so on.
43:11
SPEAKER_00: So typically when we're talking about the causes of performance,
43:15
SPEAKER_00: negative performance that aren't I'm not squeezing every last little thing out of
43:18
SPEAKER_00: the hardware,
43:19
SPEAKER_00: but rather just making sure this thing isn't running like a hundred times slower
43:22
SPEAKER_00: than it should be.
43:24
SPEAKER_00: Usually it's more just about having an awareness again of what
43:28
SPEAKER_00: the computer should be able to do and making sure you're making
43:35
SPEAKER_00: And if you do those things,
43:37
SPEAKER_00: you will generally be within,
43:39
SPEAKER_00: you know,
43:39
SPEAKER_00: 2x or something, which is 50x better than the people who are 100x away, right?
43:45
SPEAKER_00: So the good news is,
43:46
SPEAKER_00: in order to write software that's much better than a lot of
43:48
SPEAKER_00: the software you use today, you don't have to know that much.
43:52
SPEAKER_00: So what do you have to know?
43:53
SPEAKER_00: What I argue and what we focus on the Substack is I think you just have to go
43:58
SPEAKER_00: through the experience once of learning,
44:01
SPEAKER_00: reading the assembly language,
44:02
SPEAKER_00: seeing how the CPU works,
44:04
SPEAKER_00: seeing the difference, seeing why Python is slow, which we show on the Substack.
44:08
SPEAKER_00: One of the first things I show is I walk you through the assembly language necessary
44:13
SPEAKER_00: to execute A plus B in Python.
44:15
SPEAKER_00: Okay.
44:16
SPEAKER_00: And it's so vast that, you know, I have to skip most of it.
44:19
SPEAKER_00: It's massive, right?
44:20
SPEAKER_00: It's like this huge.
44:20
SPEAKER_00: And whereas,
44:22
SPEAKER_00: you know,
44:22
SPEAKER_00: if you have the equivalent function in C, it's one instruction, add, right?
44:27
SPEAKER_00: So, you know, understanding basic things like that.
44:30
SPEAKER_00: Go through,
44:31
SPEAKER_00: learn to read a semi-language,
44:33
SPEAKER_00: learn to look at some code,
44:34
SPEAKER_00: learn to do some CPU timings, and you just have that experience.
44:38
SPEAKER_00: Just spend,
44:39
SPEAKER_00: you know,
44:39
SPEAKER_00: a month or two of nights
44:42
SPEAKER_00: or whatever you want just understanding some performance stuff and going through
44:46
SPEAKER_00: a few examples where you play with it and you see the difference.
44:50
SPEAKER_01: And just so I understand,
44:52
SPEAKER_01: you're saying,
44:53
SPEAKER_01: do this not because,
44:55
SPEAKER_01: let's say you're doing iOS development or web development with React.
45:00
SPEAKER_01: You will not look at the assembly that the React does.
45:05
SPEAKER_01: If you do this on a project,
45:06
SPEAKER_01: you will be able to conceptualize what is likely happening,
45:10
SPEAKER_01: what the layers are,
45:12
SPEAKER_01: and you might be able to decide, like,
45:13
SPEAKER_01: do I want this layer or do I want to use,
45:16
SPEAKER_01: let's say, WebGL, which if you're a React engineer, you probably haven't touched it.
45:20
SPEAKER_01: But again,
45:21
SPEAKER_01: you can skip a bunch of those things and it comes to tradeoffs with maintainability,
45:24
SPEAKER_01: yada, yada.
45:25
SPEAKER_01: But that now you will know,
45:28
SPEAKER_01: like, kind of what you say by keeping this layer or not keeping it and so on.
45:32
SPEAKER_01: Do I get that right?
45:33
SPEAKER_00: Essentially, yes.
45:34
SPEAKER_00: And like, you know, the simplest example is the Python example.
45:38
SPEAKER_00: Most people have never internalized the fact that it takes,
45:41
SPEAKER_00: you know,
45:42
SPEAKER_00: maybe on the order of 100 more CPU instructions to do an ad in Python than it does
45:47
SPEAKER_00: to do it in an equivalent language like C for the same piece of text,
45:51
SPEAKER_00: just A plus B compiled in two different,
45:53
SPEAKER_00: you know, in two different languages, right?
45:54
SPEAKER_00: And so just understanding even just that is enough for you to kind of go like, oh,
46:00
SPEAKER_00: OK, A,
46:01
SPEAKER_00: now I kind of understand why if I'm using Python,
46:05
SPEAKER_00: I kind of have to use libraries to do things.
46:07
SPEAKER_00: And those libraries were written in C,
46:09
SPEAKER_00: because it's like if I'm ever going to do any operations on a large number
46:13
SPEAKER_00: of things,
46:13
SPEAKER_00: I can't do it in this language because the amplification factor is so high
46:18
SPEAKER_00: on each operation that it just kills the performance immediately,
46:22
SPEAKER_00: whereas these other languages don't have that.
46:24
SPEAKER_00: Right.
46:25
SPEAKER_00: And so understanding those orders of magnitude and what's actually going on,
46:30
SPEAKER_00: I think that allows the programmer to know, okay,
46:33
SPEAKER_00: if I think through what I'm doing right now,
46:35
SPEAKER_00: can I afford the super slowdown that I'm going to take?
46:40
SPEAKER_00: And usually, I don't think you have to be a performance expert to make that decision.
46:44
SPEAKER_00: You could usually know, like, okay,
46:46
SPEAKER_00: is this a part of the code that can afford to be 100 times slower than it should be
46:50
SPEAKER_00: or not, right?
46:51
SPEAKER_00: And, you know, most people can, I think, make that decision fairly logically.
46:55
SPEAKER_00: And if it's not,
46:56
SPEAKER_00: then now you know, like, oh, okay,
46:57
SPEAKER_00: if I'm in Python,
46:58
SPEAKER_00: then what I gotta do is either I gotta go find a library call
47:02
SPEAKER_00: that will do these sorts of things and structure around how that library works,
47:06
SPEAKER_00: Or I should maybe get something like Cython or something
47:09
SPEAKER_00: where I can do compiled stuff inside my Python
47:13
SPEAKER_00: and make my code work around calls out to that kind of code.
47:17
SPEAKER_00: You know,
47:18
SPEAKER_00: you now have the tools you need upfront to make Make sure that when you write
47:22
SPEAKER_00: the program,
47:23
SPEAKER_00: you've put the parts that needed this and you've structured the code in such a way
47:28
SPEAKER_00: that you are only paying the 100x on things that you know are very infrequent
47:32
SPEAKER_00: or happen like only, you know, once per every so often.
47:37
SPEAKER_00: That's, I think, the biggest thing is just the knowledge.
47:40
SPEAKER_00: And once you know,
47:41
SPEAKER_00: you can start to make much better decisions in any language
47:44
SPEAKER_00: because it doesn't take you very long.
47:47
SPEAKER_00: A simple search or asking an AI or whatever is the common practice that you're going to do.
47:53
SPEAKER_00: A simple bit of that,
47:55
SPEAKER_00: once you know what you're asking for,
47:57
SPEAKER_00: will get you this information back very quickly.
48:00
SPEAKER_00: You just have to know that you should have been thinking about it.
48:03
SPEAKER_01: Now, as a software, you mentioned it's good to understand how the CPU works.
48:07
SPEAKER_01: As a software engineer who is not a games developer, I'm not doing low-level stuff.
48:11
SPEAKER_01: What does that give me?
48:13
SPEAKER_01: Because for the most part,
48:16
SPEAKER_01: even in academia or in computer science,
48:19
SPEAKER_01: there is some level of some basic CPU theory taught,
48:23
SPEAKER_01: but usually we just kind of, we stop at the code.
48:26
SPEAKER_01: Okay, maybe you look at the assembly, but you rarely go further than that.
48:30
SPEAKER_01: The folks So you're talking about
48:41
SPEAKER_00: specifically the knowing the CPU part?
48:43
SPEAKER_01: Knowing about the CPU,
48:44
SPEAKER_01: knowing about the details about the CPU,
48:46
SPEAKER_01: because you mentioned that that's also part of it, right?
48:48
SPEAKER_01: It's not necessarily just stopping at assembly.
48:50
SPEAKER_00: So the reason for that is more the other way around.
48:54
SPEAKER_00: The reason to learn the assembly language is so that you know what the CPU is doing.
48:58
SPEAKER_00: So it's the CPU part that's actually important.
49:01
SPEAKER_00: And the part that's important about it is that the CPU is basically,
49:05
SPEAKER_00: you can think of it as a little machine Whose internal gearings we are not privy to
49:11
SPEAKER_00: because for the CPUs that we care about.
49:14
SPEAKER_00: So, you know,
49:15
SPEAKER_00: an M series CPU in a Mac,
49:17
SPEAKER_00: a Zen core CPU in a server or in a laptop or an Intel, you know, core series.
49:23
SPEAKER_00: These CPUs are not documented at the level where you're going to be thinking
49:29
SPEAKER_00: about how each little individual part works.
49:31
SPEAKER_00: And to that end,
49:33
SPEAKER_00: it's unclear that you would have time to do so anyway,
49:34
SPEAKER_00: because these are massive,
49:36
SPEAKER_00: very complicated machines that we're talking about, right?
49:39
SPEAKER_00: But from a high level,
49:40
SPEAKER_00: from a more black box perspective,
49:42
SPEAKER_00: they are machines that we can think of
49:44
SPEAKER_00: in relatively straightforward ways once we know kind
49:48
SPEAKER_00: of what their core instructions are that they tend to execute.
49:53
SPEAKER_00: And they break down into a couple of different categories.
49:56
SPEAKER_00: There's how does data move into and out of a core?
50:00
SPEAKER_00: And this is basically how like load store units work, how the cache levels work, L1, L2, L3.
50:07
SPEAKER_00: So we have like, L0 now sometimes.
50:09
SPEAKER_00: Things like this.
50:10
SPEAKER_00: How does that work and why?
50:12
SPEAKER_00: What is the granularity of it?
50:14
SPEAKER_00: What is the policy?
50:15
SPEAKER_00: How does the CPU go about actually working with those things?
50:18
SPEAKER_00: Understanding that part of the machine is crucial because when you're working with
50:22
SPEAKER_00: a lot of data,
50:23
SPEAKER_00: the difference can be massive if you structure it in one way versus structure
50:27
SPEAKER_00: in another way, right?
50:28
SPEAKER_00: Again, architectural decisions that have nothing to do with hotspots.
50:31
SPEAKER_00: They're how all the data is laid out and what the access pattern is, right?
50:34
SPEAKER_00: Things that are very hard to change sometimes.
50:36
SPEAKER_00: So that's one part of the machine you want to understand.
50:39
SPEAKER_00: Another part of the machine you want to understand is how the instructions flow through it.
50:43
SPEAKER_00: And a lot of people have heard about like branch misprediction or things like this,
50:47
SPEAKER_00: iCache misses.
50:48
SPEAKER_00: There's words that you might hear, but you're not really sure what they mean.
50:51
SPEAKER_00: They're all actually pretty simple to conceptualize.
50:53
SPEAKER_00: Sometimes they're harder to pin down exactly how they work
50:56
SPEAKER_00: because branch predictors are,
50:57
SPEAKER_00: you know, getting more and more complicated and so on.
50:59
SPEAKER_00: But you can still categorize the behavior of them and understand how code flows
51:03
SPEAKER_00: through it and when you might care and when you won't.
51:05
SPEAKER_00: And then finally,
51:06
SPEAKER_00: there's the execution unit scheduling part,
51:08
SPEAKER_00: which is about knowing what's the raw sort of throughput for any particular type
51:12
SPEAKER_00: of operation,
51:13
SPEAKER_00: floating point multiplies,
51:14
SPEAKER_00: integer additions, division, whatever it is that you might want to know, right?
51:18
SPEAKER_00: Once you learn a little bit of assembly language, you understand what it's reading.
51:22
SPEAKER_00: You understand how it turns those assembly language instructions
51:24
SPEAKER_00: into micro operations,
51:26
SPEAKER_00: which it actually does, and how they get distributed through that machine.
51:29
SPEAKER_00: That flow chart that they put, like,
51:32
SPEAKER_00: basically up on the we've announced the new Zencore,
51:35
SPEAKER_00: that flow chart,
51:36
SPEAKER_00: you can look at it and go, I know the performance of this machine roughly.
51:40
SPEAKER_00: Right.
51:41
SPEAKER_00: Not exactly.
51:42
SPEAKER_00: Because like I said,
51:42
SPEAKER_00: there's all these little edge cases that if you really want to be a crazy optimizer,
51:46
SPEAKER_00: which again, I don't really advocate people do.
51:48
SPEAKER_00: I don't think it's important that they be like crazy hyper optimizers.
51:51
SPEAKER_00: If you like to do it, great.
51:52
SPEAKER_00: It's a lot of fun, but it's not the important part.
51:55
SPEAKER_00: Just look at that CPU, like, okay,
51:57
SPEAKER_00: I see what the CPU should be getting in terms of,
51:59
SPEAKER_00: like, what I could do with this size data load, that size data load.
52:02
SPEAKER_00: This is what I could probably get out of it for if I was doing a bunch of,
52:05
SPEAKER_00: like, how to do a bunch of, like, math ops on it, you know.
52:08
SPEAKER_00: And I think that's just something that should be kind of par for the car software engineering.
52:12
SPEAKER_00: You go to school for four years to learn this,
52:14
SPEAKER_00: there's no reason you can't learn this in a few months.
52:17
SPEAKER_00: It's not that hard.
52:18
SPEAKER_01: Yeah, plus I guess just from a craftsmanship perspective, like we should know our tools.
52:22
SPEAKER_01: We should know the machines that we're programming.
52:24
SPEAKER_01: Obviously,
52:24
SPEAKER_01: we know that our code will be,
52:26
SPEAKER_01: if you're doing web,
52:26
SPEAKER_01: it'll be running on all these different things or if it's mobile
52:30
SPEAKER_01: on all these different phones.
52:32
SPEAKER_01: But from a conceptual point of view, like we should be able to know what's going on.
52:36
SPEAKER_01: So I feel there's a bit of a pride as well.
52:37
SPEAKER_01: Like if nothing else, you would learn a bunch of stuff.
52:40
SPEAKER_01: Like I know some of it, but I'm now getting motivation to learn more about it.
52:44
SPEAKER_00: I do think there's a craftsmanship angle.
52:46
SPEAKER_00: I think there's a large number of people who maybe don't feel fulfilled
52:51
SPEAKER_00: when they I've certainly heard from lots of people who when they write something
52:54
SPEAKER_00: and it's just kind of this amorphous high level thing,
52:57
SPEAKER_00: they don't get as much satisfaction out of it.
52:59
SPEAKER_00: And then when they They learn how they can look more deeply at what's going on.
53:03
SPEAKER_00: They feel much more satisfied.
53:05
SPEAKER_00: Even if they didn't change what level they were programming at,
53:08
SPEAKER_00: they feel much more satisfied that now they know what they're doing.
53:10
SPEAKER_00: And it's like, oh, I see.
53:11
SPEAKER_00: And I understand why this thing was happening this way and this thing was happening that way.
53:15
SPEAKER_00: That's very satisfying.
53:16
SPEAKER_00: So there's an aspect of that.
53:18
SPEAKER_00: I also want to emphasize another part, which is that it's a percentages game.
53:23
SPEAKER_00: If we convinced enough library maintainers that this stuff was important and
53:28
SPEAKER_00: the libraries all get a lot faster,
53:30
SPEAKER_00: all of a sudden all the people using the libraries code gets a lot faster and so on
53:34
SPEAKER_00: and so forth.
53:35
SPEAKER_00: If the API is start changing to make it easier to Well,
53:38
SPEAKER_00: plus, I do think right now there's still an edge in just being
53:53
SPEAKER_01: And so on and so forth.
53:54
SPEAKER_01: So like I feel there's like, and worst case, you just learn it.
54:14
SPEAKER_01: Like net new knowledge,
54:16
SPEAKER_01: which will probably not be as outdated with AI, which we'll get into later.
54:20
SPEAKER_01: But this stuff, it just feels very interesting, right?
54:23
SPEAKER_01: It moves your brain.
54:24
SPEAKER_00: It does.
54:25
SPEAKER_00: And thankfully, like,
54:27
SPEAKER_00: it's also not that hard to update your knowledge because, again,
54:31
SPEAKER_00: you get these presentations that the CPU companies give and they're like,
54:35
SPEAKER_00: here's the changes we made.
54:36
SPEAKER_00: So you kind of are aware every time a new thing gets,
54:39
SPEAKER_00: you know, there's little tiny things that creep in that you don't that, you know.
54:43
SPEAKER_00: But again,
54:43
SPEAKER_00: there's people out there who are running lots of micro benchmarks
54:46
SPEAKER_00: that you will find out about.
54:47
SPEAKER_00: And they often uncover these for you as well.
54:50
SPEAKER_00: Right.
54:50
SPEAKER_00: So.
54:51
SPEAKER_01: So I want to talk about games.
54:52
SPEAKER_01: You've built games for decades at this point.
54:57
SPEAKER_01: Can you give an overview for those of us who are not in the games industry?
55:02
SPEAKER_01: How is a game typically built from the games that you know of,
55:07
SPEAKER_01: that you've observed or you worked on,
55:09
SPEAKER_01: especially trying to compare for, you know, like a typical situation?
55:12
SPEAKER_01: SaaS or a distributed system or something, we're building a website or services.
55:17
SPEAKER_01: You plan this stuff.
55:18
SPEAKER_01: We'll do an estimate.
55:20
SPEAKER_01: We'll build it in a few months or a few weeks.
55:22
SPEAKER_01: We deploy it.
55:23
SPEAKER_01: And then we monitor it.
55:24
SPEAKER_01: And then we keep tweaking it.
55:25
SPEAKER_01: And then fast forward five years later, it's now this gigantic thing with microservices.
55:30
SPEAKER_01: But it keeps evolving, right?
55:32
SPEAKER_01: We do this a lot of prototyping thing.
55:34
SPEAKER_01: For games, it's pretty obvious, right?
55:36
SPEAKER_01: From the get-go events we're talking, there will be a launch.
55:39
SPEAKER_01: But can you,
55:41
SPEAKER_01: when you're inside or when you join a game studio,
55:44
SPEAKER_01: like what would you observe there in terms of what the process is like
55:47
SPEAKER_01: and how it's different or how it feels weird compared to like this,
55:51
SPEAKER_01: I guess,
55:51
SPEAKER_01: I don't know, traditional software, SaaS software, whatever development that is.
55:54
SPEAKER_00: So I guess what I would say is unfortunately I'm probably the wrong one to ask
55:58
SPEAKER_00: because my knowledge is outdated at this point because one of the things
56:04
SPEAKER_00: that has happened to games recently is I feel like they've moved closer.
56:07
SPEAKER_00: I don't want to necessarily say entirely in development practices,
56:11
SPEAKER_00: but at least in terms of the nature of the product.
56:16
SPEAKER_00: Has changed somewhat dramatically to be more like something like SaaS, where,
56:23
SPEAKER_00: you know,
56:23
SPEAKER_00: if you take some of the most popular things that are being in terms of dollars,
56:28
SPEAKER_00: let's say.
56:28
SPEAKER_00: So I guess maybe popular is kind of might be hard.
56:31
SPEAKER_00: Yeah.
56:31
SPEAKER_00: Yeah.
56:49
SPEAKER_00: These things are starting to look a lot more like an always on live service kind
56:56
SPEAKER_00: of we ship incremental features to our customers kinds of things.
57:01
SPEAKER_00: And so I would actually say that,
57:03
SPEAKER_00: you know,
57:04
SPEAKER_00: I'm the wrong one to ask about what that actually looks like from the inside
57:08
SPEAKER_00: because I haven't actually gone and worked at one of those companies.
57:11
SPEAKER_00: I have friends there,
57:12
SPEAKER_00: so I hear things,
57:13
SPEAKER_00: but I'm not the right one to like give it an accurate picture of it.
57:16
SPEAKER_00: But I just would point out From my perspective,
57:19
SPEAKER_00: the game industry practices look different today than they did when I have
57:23
SPEAKER_00: a more intimate sort of experience with what we were actually building.
57:27
SPEAKER_01: But can we talk about it when you were building games,
57:30
SPEAKER_01: which was,
57:30
SPEAKER_01: you know,
57:30
SPEAKER_01: 10 plus years ago when I understand these were the games where they were built,
57:35
SPEAKER_01: they were launched,
57:36
SPEAKER_01: you know,
57:37
SPEAKER_01: maybe they got a patch or two and then they were kind of,
57:39
SPEAKER_01: you know, the team moved on, they were disbanded.
57:41
SPEAKER_01: It was a time box thing that was a lot of development,
57:44
SPEAKER_01: a big launch and either it It went big, huge hit, or a huge failure, right?
57:50
SPEAKER_01: And the studio goes bankrupt.
57:52
SPEAKER_01: So how did that work?
57:53
SPEAKER_01: Because I feel that's a world where, okay,
57:55
SPEAKER_01: today a bunch of games don't have those constraints,
57:57
SPEAKER_01: but it has a bunch of constraints that I'm interested in what worked
58:01
SPEAKER_01: in those constraints.
58:03
SPEAKER_00: So...
58:04
SPEAKER_00: In the early days, you didn't have licensable engines.
58:08
SPEAKER_00: So up until sort of the point,
58:10
SPEAKER_00: like nowadays, like this is why I say it's a lot different now than it used to be.
58:14
SPEAKER_00: You know,
58:15
SPEAKER_00: nowadays you think of web development,
58:16
SPEAKER_00: like I'm gonna go grab like a thing like React and I'm gonna make this thing
58:19
SPEAKER_00: or whatever.
58:20
SPEAKER_00: I'm gonna go grab an off the shelf database thing,
58:22
SPEAKER_00: Postgres or Oracle, or I don't know, like what would be the thing of choice, right?
58:26
SPEAKER_01: But by the way, just definitely Postgres and not Oracle for most people.
58:29
SPEAKER_01: Okay, sorry.
58:30
SPEAKER_00: Postgres, I didn't want to slight anybody there, so I apologize.
58:33
SPEAKER_00: So, okay, definitely Postgres, sorry, Oracle.
58:36
SPEAKER_00: So you're going to go use some kind of a variant of one of these off-shelf databases and so on.
58:41
SPEAKER_00: That's more, like I said, what people might be doing nowadays, too.
58:43
SPEAKER_00: They'll grab the Unreal Engine.
58:45
SPEAKER_00: They're not going to develop an engine on their own.
58:46
SPEAKER_00: They'll grab backend server stuff from people.
58:49
SPEAKER_00: It might even be some Postgres in there, right?
58:51
SPEAKER_00: Like, who knows?
58:52
SPEAKER_00: In the earlier days, none of this stuff existed for games.
58:55
SPEAKER_00: I actually worked, like I said, in middleware at the time.
58:58
SPEAKER_00: So I was actually sort of one of the people who was working at the time
59:01
SPEAKER_00: on maybe changing that a little,
59:03
SPEAKER_00: like producing code that would get reused throughout games,
59:05
SPEAKER_00: which was actually fairly rare.
59:07
SPEAKER_01: And so back in the day, every game built their own rendering engine, for example.
59:11
SPEAKER_00: Correct.
59:12
SPEAKER_00: No way.
59:13
SPEAKER_00: Yeah, yeah, yeah.
59:14
SPEAKER_00: And so really early on, right,
59:17
SPEAKER_00: if you rewind the clock far enough,
59:20
SPEAKER_00: then yeah,
59:21
SPEAKER_00: the degree to which people were reusing code for their rendering engine,
59:27
SPEAKER_00: it'd be like,
59:27
SPEAKER_00: because I got some code from Dave or whatever,
59:31
SPEAKER_00: or we were both at Atari and somebody wrote this good routine, so we used it, right?
59:36
SPEAKER_00: There was that kind of thing, but there wasn't like this sort of set engine.
59:40
SPEAKER_00: And the time when that sort of maybe you could say first started happening
59:44
SPEAKER_00: a little more widespread was with things like what id Software did,
59:49
SPEAKER_00: where they sort of started having like, oh,
59:51
SPEAKER_00: you know,
59:51
SPEAKER_00: like someone's going to build something with the Doom engine or someone's going
59:54
SPEAKER_00: to build something with the Quake engine.
59:56
SPEAKER_00: There was also the build engine at the time made by Ken Silverman and some things like that.
1:00:00
SPEAKER_00: So there were There were some early cases where a few people would make a game,
1:00:05
SPEAKER_00: but they were making a game very much like that.
1:00:07
SPEAKER_00: Like if we did the Doom engine, we're going to make a game very much like Doom.
1:00:10
SPEAKER_00: So it really was the case that for most games,
1:00:12
SPEAKER_00: people were rebuilding most of the things from scratch, at least for their studio.
1:00:17
SPEAKER_00: And studios often, their existing code base was kind of part of the value of the studio too.
1:00:23
SPEAKER_00: Like if you are...
1:00:24
SPEAKER_00: Think Blizzard, and we just built Warcraft 1.
1:00:27
SPEAKER_00: Well,
1:00:27
SPEAKER_00: rolling all of that knowledge and code into Warcraft 2 is a huge advantage for us
1:00:33
SPEAKER_00: because everybody else who wants to build a competitor to Warcraft 1 has to do all
1:00:38
SPEAKER_00: of that from scratch.
1:00:39
SPEAKER_00: They have to make the pathfinder,
1:00:40
SPEAKER_00: they have to make the level editing tools,
1:00:41
SPEAKER_00: they have to make the rendering, they have to make whatever.
1:00:44
SPEAKER_00: And so, you know, that was how things were traditionally done.
1:00:48
SPEAKER_01: No wonder the games industry is so secretive compared to the rest of
1:00:51
SPEAKER_01: the software engineering industry.
1:00:52
SPEAKER_01: Like, seriously.
1:00:54
SPEAKER_00: Used to be.
1:00:54
SPEAKER_00: Used to be, at least.
1:00:55
SPEAKER_00: Yeah, maybe now it's changing.
1:00:57
SPEAKER_00: And so there were two really big risks,
1:00:59
SPEAKER_00: typically, when you started a game project in those days.
1:01:03
SPEAKER_00: One was the engine risk.
1:01:05
SPEAKER_00: Would we be able to make something that would be technically able to do what we need
1:01:10
SPEAKER_00: to do for this game?
1:01:13
SPEAKER_00: And that risk comes in a lot of flavors.
1:01:15
SPEAKER_00: One, will it happen at all?
1:01:17
SPEAKER_00: Two, will it happen fast enough for us to actually reliably build the game on it, right?
1:01:24
SPEAKER_00: One of the things, I mean,
1:01:25
SPEAKER_00: I don't know how detailed an answer you're looking for for this question,
1:01:27
SPEAKER_00: so stop me if I'm going down too many tangents,
1:01:29
SPEAKER_00: but one of the things you also have to remember is that at that time,
1:01:34
SPEAKER_00: And this is sort of still true today, but at that time, it was very important.
1:01:37
SPEAKER_00: There was no way to really buy something all that much faster than what you had.
1:01:42
SPEAKER_00: There was not a huge strata of PCs that you could buy or anything like that.
1:01:46
SPEAKER_00: So the rendering engine,
1:01:48
SPEAKER_00: there wasn't like a way you're leveling So there are people
1:01:49
SPEAKER_00: who started doing things like buying SGI workstations.
1:02:04
SPEAKER_00: Because those were actually faster and up, right?
1:02:07
SPEAKER_00: And things like that is what, you know, you kind of had to do and so on.
1:02:11
SPEAKER_00: So anyway,
1:02:12
SPEAKER_00: so there was a huge engine risk and some games just failed
1:02:15
SPEAKER_00: because they couldn't produce a thing
1:02:16
SPEAKER_00: that could technologically do what they needed.
1:02:22
SPEAKER_00: You had id Software that was kind of unrivaled at making those kind of first person engines.
1:02:26
SPEAKER_00: You had Bullfrog who had this engine,
1:02:28
SPEAKER_00: those pseudo three engine that they did for like the racing games,
1:02:31
SPEAKER_00: Magic Carpet, Dungeon Keeper.
1:02:32
SPEAKER_00: Like they were all based on this, you know, one core tech and all the sorts of things.
1:02:37
SPEAKER_00: There's that engine risk.
1:02:38
SPEAKER_00: That was huge.
1:02:39
SPEAKER_00: And how do you mitigate it?
1:02:40
SPEAKER_00: You didn't.
1:02:41
SPEAKER_00: You were just, right?
1:02:42
SPEAKER_00: Because there wasn't a way to buy one off the shelf.
1:02:44
SPEAKER_00: So you were just kind of gritting your teeth.
1:02:46
SPEAKER_00: The other big risk,
1:02:47
SPEAKER_00: and this one is still somewhat true today,
1:02:49
SPEAKER_00: but it's just much less because you can start, you can do prototyping early.
1:02:53
SPEAKER_00: The risk is, is the game any good?
1:02:55
SPEAKER_00: Like, what are we building?
1:02:57
SPEAKER_00: Is it interesting?
1:02:57
SPEAKER_00: Is it fun?
1:02:58
SPEAKER_00: And when you think about this problem of we can't even really run the game
1:03:02
SPEAKER_00: as it will be because we're just building this engine and we don't even really have
1:03:05
SPEAKER_00: a way to test the game super well and we can't really build it.
1:03:08
SPEAKER_00: I want to say like Thief the Dark Project, a very famous game.
1:03:23
SPEAKER_00: Lookingglass game was formative in the stealth genre, launched a franchise, which was Thief.
1:03:28
SPEAKER_00: You know,
1:03:29
SPEAKER_00: I want to say everything I heard from people on that team was that like
1:03:32
SPEAKER_00: the final core gameplay only sort of came together like right at the end.
1:03:36
SPEAKER_00: Right.
1:03:37
SPEAKER_00: And so the game just could have been a lot more, could have been polished a lot more.
1:03:41
SPEAKER_00: But it's just the timing of these things coming together was so hard.
1:03:44
SPEAKER_00: And so it really was an incredibly different thing.
1:03:47
SPEAKER_00: And Nobody really had a way around it.
1:03:50
SPEAKER_00: Eventually,
1:03:51
SPEAKER_00: there was sort of this push towards something, well,
1:03:54
SPEAKER_00: it was basically early vertical slice prototyping,
1:03:57
SPEAKER_00: where as games started getting bigger and people were like,
1:03:59
SPEAKER_00: we can't keep doing this,
1:04:00
SPEAKER_00: especially if we're going to be putting millions of dollars on,
1:04:02
SPEAKER_00: this is not an option, right?
1:04:04
SPEAKER_00: They started to move towards this thing about, look,
1:04:06
SPEAKER_00: what we're going to do is we're going to focus the entire studio
1:04:09
SPEAKER_00: on building one vertical gameplay slice as fast as we can, as hacky as we can.
1:04:14
SPEAKER_00: Whatever we have to do, do that.
1:04:17
SPEAKER_00: Prove that that is engaging to play.
1:04:20
SPEAKER_00: And then start building out everything else.
1:04:23
SPEAKER_00: Because we simply can't afford to not know what that thing is.
1:04:27
SPEAKER_00: And then we can start building like spreadsheets that'll schedule what are the assets we need.
1:04:32
SPEAKER_00: We can actually believe in the thing.
1:04:34
SPEAKER_00: We can see it running and it fills in all those details.
1:04:36
SPEAKER_00: Right.
1:04:37
SPEAKER_00: And that I believe, you know, I'm not a game historian.
1:04:40
SPEAKER_00: So take what I'm saying with a huge grain of salt that I believe was
1:04:43
SPEAKER_00: a pretty big paradigm shift for the industry when they started going,
1:04:46
SPEAKER_00: OK, we got to actually know.
1:04:48
SPEAKER_00: And it became much less seat of the pants after that, if you will.
1:04:52
SPEAKER_01: Now, I'm interested in your observation.
1:04:53
SPEAKER_01: I know you're not a game historian,
1:04:55
SPEAKER_01: but you were in the industry and you still remain connected to it.
1:04:59
SPEAKER_01: What happened when game engines became widespread?
1:05:03
SPEAKER_01: They became not only licensable, like,
1:05:06
SPEAKER_01: you know,
1:05:07
SPEAKER_01: We're talking about like Unreal Engine for larger studios,
1:05:09
SPEAKER_01: but ones like Unity or Godot, which now amateurs could also afford.
1:05:14
SPEAKER_01: I mean, amateurs in the sense that you're a college kid or you do some side project,
1:05:19
SPEAKER_01: you can already afford the license and you can build stuff.
1:05:22
SPEAKER_01: Because now that risk has gone for Clear the Studio.
1:05:24
SPEAKER_01: So that risk is eliminated.
1:05:26
SPEAKER_01: And it also,
1:05:26
SPEAKER_01: I guess it now opened up so much more people who can now have a shot at creating
1:05:32
SPEAKER_01: a game because you no longer have to either have this massive amount to license,
1:05:35
SPEAKER_01: this super expensive game engine.
1:05:38
SPEAKER_01: You no longer have that risk.
1:05:39
SPEAKER_01: The only risk is, is it fun?
1:05:42
SPEAKER_01: What have you observed happen in terms of both for the industry,
1:05:46
SPEAKER_01: for development, pace, those kind of things?
1:05:49
SPEAKER_01: And the reason I'm asking,
1:05:50
SPEAKER_01: because I wonder if there's going to be a parallel with AI where,
1:05:53
SPEAKER_01: okay, you know, like you needed to have an engineer who is- You nailed it.
1:05:57
SPEAKER_01: I feel games might give us a bit of a hint of what we might expect at the broader industry.
1:06:03
SPEAKER_00: So that is actually, I would say that's a brilliant analysis of the situation.
1:06:07
SPEAKER_00: For not having lived through games and for noticing that, that's impressive.
1:06:11
SPEAKER_00: I'll say that first.
1:06:13
SPEAKER_00: And I totally agree with that.
1:06:15
SPEAKER_00: I've said to people in the past who have asked about sort of AI impact on games in that sense.
1:06:22
SPEAKER_00: And I've sort of said as much,
1:06:24
SPEAKER_00: I've said like,
1:06:25
SPEAKER_00: the licensable engine thing kind of was our AI transition already, unfortunately.
1:06:31
SPEAKER_00: And I regret to inform you that the news is not probably that positive.
1:06:37
SPEAKER_00: There are some definite positive things that happen early on because,
1:06:42
SPEAKER_00: as you say,
1:06:43
SPEAKER_00: it opens up the ability to make games to people who could not have marshaled
1:06:50
SPEAKER_00: the technical staff necessary to produce a competitive engine.
1:06:55
SPEAKER_00: And so giving them the ability to make games is a pretty important thing,
1:06:59
SPEAKER_00: and it allows a bunch of people to make sort of some artistic expressions
1:07:03
SPEAKER_00: that they just wouldn't have been able to do.
1:07:07
SPEAKER_00: Early on, that tends to be a net positive because you just have some more games coming out.
1:07:12
SPEAKER_00: Maybe some of them aren't that good, but some of the existing games aren't that good.
1:07:16
SPEAKER_00: That's not that different.
1:07:17
SPEAKER_00: But then you get some really cool games coming from some sources
1:07:19
SPEAKER_00: that just simply wouldn't have been able to do it.
1:07:21
SPEAKER_00: Thumbs up.
1:07:22
SPEAKER_00: Problem is,
1:07:24
SPEAKER_00: it rapidly kind of accelerates into this kind of a nasty scenario
1:07:29
SPEAKER_00: where you just have massive numbers of releases.
1:07:33
SPEAKER_00: And I think at this point, we're at the point where I want to say Steam.
1:07:37
SPEAKER_00: Steam games are in the like tens of thousands or a hundred thousand per year
1:07:42
SPEAKER_00: or something like that.
1:07:44
SPEAKER_00: It's so massive that there is no way
1:07:48
SPEAKER_00: that your game will be organically noticed anymore, pretty much period.
1:07:51
SPEAKER_00: So essentially it's this really nasty problem where you just have the market flooded
1:07:57
SPEAKER_00: with products.
1:07:58
SPEAKER_00: And there,
1:08:00
SPEAKER_00: you know,
1:08:00
SPEAKER_00: it used to be that if you made a quality game,
1:08:02
SPEAKER_00: if it was fun,
1:08:03
SPEAKER_00: people would find it because there were so few games that someone would play
1:08:08
SPEAKER_00: the fun one and tell people about it and it would get purchased, right?
1:08:11
SPEAKER_00: Like word of mouth or just exposure on a storefront.
1:08:14
SPEAKER_00: Fast forward to today, where we have this sort of massive influx of games, again, pre-AI.
1:08:19
SPEAKER_00: It's just because now the barrier to entry is very low.
1:08:31
SPEAKER_00: And you really need a strategy to make sure your game gets found.
1:08:36
SPEAKER_00: Is it possible that sometimes,
1:08:39
SPEAKER_00: you know,
1:08:40
SPEAKER_00: a small indie game with no marketing plan or nothing will get discovered and become
1:08:45
SPEAKER_00: a huge hit?
1:08:46
SPEAKER_00: Absolutely.
1:08:47
SPEAKER_00: It does still happen once in a while.
1:08:49
SPEAKER_00: The chances that you will be that game are like zero.
1:08:52
SPEAKER_00: So you kind of now need a marketing strategy,
1:08:57
SPEAKER_00: a real marketing strategy,
1:08:59
SPEAKER_00: and going into the market for games without one is And expecting
1:09:05
SPEAKER_00: to sell any significant number of copies above,
1:09:08
SPEAKER_00: you know, maybe a few thousand at best is really unwise.
1:09:12
SPEAKER_00: If you want to hit reasonable numbers of sales of a game,
1:09:14
SPEAKER_00: you have to have an idea of how people will find out about this about this game.
1:09:20
SPEAKER_01: So if I'm getting this right,
1:09:22
SPEAKER_01: it sounds like the game itself being good as table stakes,
1:09:25
SPEAKER_01: but not enough on its own, right?
1:09:27
SPEAKER_01: That distribution,
1:09:28
SPEAKER_01: marketing,
1:09:28
SPEAKER_01: getting people to hear about the game is much more of a differentiator
1:09:32
SPEAKER_01: because there's just too many good games out there and now they're easier to create.
1:09:37
SPEAKER_00: I think that's exactly right.
1:09:39
SPEAKER_00: And that's just the unfortunate reality of it now.
1:09:42
SPEAKER_00: Was that a good trade?
1:09:44
SPEAKER_00: I don't know, but that's what happened.
1:09:46
SPEAKER_00: And so that's where we are in the industry now.
1:09:48
SPEAKER_01: There's this other thing that I heard about,
1:09:50
SPEAKER_01: which is how new games not only compete with other new games,
1:09:53
SPEAKER_01: but with old games as well, right?
1:09:55
SPEAKER_01: Like the other day, I spent a few hours playing Death Rally, which is a game from the 90s.
1:10:00
SPEAKER_01: And every year, there's more and more good games to play.
1:10:03
SPEAKER_01: They all take away from the time that the new games have.
1:10:06
SPEAKER_00: Yes, and that problem will only get worse because one of the things that
1:10:10
SPEAKER_00: the game industry could rely on in the past that is much harder to rely on now is
1:10:16
SPEAKER_00: that older games would look dated technologically in ways that consumers cared
1:10:22
SPEAKER_00: about.
1:10:23
SPEAKER_00: And we have now kind of also crossed the threshold where there is a segment of
1:10:29
SPEAKER_00: the market where people really do care about the latest ray-traced lighting
1:10:34
SPEAKER_00: and all these sorts of things and more photorealistic rendering or whatever it is.
1:10:39
SPEAKER_00: But a large portion of the gaming market by revenue...
1:10:43
SPEAKER_00: Doesn't really care what the game looked like all that much in a sense
1:10:48
SPEAKER_00: that whatever we're doing today is good enough.
1:10:50
SPEAKER_00: So 10 years from now,
1:10:52
SPEAKER_00: if the games look much better for some reason,
1:10:55
SPEAKER_00: no one will really think of that as a huge differentiator in terms of sales.
1:11:00
SPEAKER_00: You go back to 1995 and technological advances were a huge differentiator in terms of sales.
1:11:06
SPEAKER_00: You come out with something that looks good,
1:11:08
SPEAKER_00: that takes advantage of the hardware of that day,
1:11:10
SPEAKER_00: and boy,
1:11:11
SPEAKER_00: did it look cooler and feel more responsive and all these other things as compared
1:11:14
SPEAKER_00: to earlier titles, right?
1:11:16
SPEAKER_00: And so that's also going to increase the degree to which the thing
1:11:20
SPEAKER_00: that you're talking about will happen.
1:11:22
SPEAKER_00: I can go play an older game because it doesn't feel obviously dated in an audiovisual way.
1:11:27
SPEAKER_00: I don't have to be an appreciator of retro gaming to go play something from 2017.
1:11:33
SPEAKER_00: It just looks fine, probably, right?
1:11:35
SPEAKER_00: So there's that.
1:11:37
SPEAKER_00: The other thing that I'll just mention,
1:11:38
SPEAKER_00: which we kind of already touched on,
1:11:39
SPEAKER_00: but that ties directly into your point,
1:11:42
SPEAKER_00: is that also live service is such a prominent thing now.
1:11:46
SPEAKER_00: People are just logging on and playing Fortnite for several hours or something.
1:11:51
SPEAKER_00: That's also taking away from the possible revenue that might be spent
1:11:55
SPEAKER_00: on buying some indie game or some new AAA game even.
1:11:58
SPEAKER_00: Okay, so I need to ask you this.
1:12:00
SPEAKER_00: GTA 6.
1:12:22
SPEAKER_01: How is that in 2026 at a time when we have better tools than we have ever before
1:12:27
SPEAKER_01: and we can build software and games faster than before?
1:12:30
SPEAKER_01: Like, how do games take 10 plus years to develop?
1:12:34
SPEAKER_01: Is this some kind of outlier or has AAA game development taking many,
1:12:38
SPEAKER_01: many years just not changed at all?
1:12:41
SPEAKER_01: What do you think is going on here?
1:12:42
SPEAKER_00: So from a player's perspective,
1:12:44
SPEAKER_00: I can understand why someone would look at it and go,
1:12:46
SPEAKER_00: wow, Grand Theft Auto 6 has been in development a long time.
1:12:49
SPEAKER_00: How does that make sense?
1:12:50
SPEAKER_00: Or, you know, something like this.
1:12:53
SPEAKER_00: From a business perspective,
1:12:54
SPEAKER_00: you have to understand that Grand Theft Auto 6 is not a game that they are selling
1:12:58
SPEAKER_00: to players who are going to play the game.
1:13:00
SPEAKER_00: That's not what it is from a product standpoint, right?
1:13:03
SPEAKER_00: What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5.
1:13:09
SPEAKER_00: Grand Theft Auto V at the time was,
1:13:12
SPEAKER_00: if I'm not mistaken,
1:13:14
SPEAKER_00: by far the most revenue generating entertainment product in existence.
1:13:20
SPEAKER_00: The online part of that game was generating like billions of dollars.
1:13:25
SPEAKER_00: And like I said, not a game industry historian.
1:13:28
SPEAKER_00: So, you know, take what I have to say with a huge grain of salt.
1:13:31
SPEAKER_00: But Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will.
1:13:35
SPEAKER_00: They were a huge, huge live service revenue generating product.
1:13:42
SPEAKER_00: So from Rockstar or Take-Two's perspective, right?
1:13:46
SPEAKER_00: Grand Theft Auto 6 is not just let's try to get out the next Grand Theft Auto
1:13:51
SPEAKER_00: as soon as we can because we'll make money selling that title.
1:13:54
SPEAKER_00: It's a we are going to replace the most profitable thing we have ever built,
1:13:59
SPEAKER_00: which is still generating a ton of money for us with a new thing.
1:14:03
SPEAKER_00: And you can better believe that they want to make sure that they are going to do that right.
1:14:09
SPEAKER_00: Because the last thing you want to do is ship a new product
1:14:14
SPEAKER_00: that cannibalizes something from your old product and
1:14:17
SPEAKER_00: then is less revenue generating, right?
1:14:20
SPEAKER_00: So I'm sure that their planning around Grand Theft Auto 6 is not just about trying
1:14:24
SPEAKER_00: to produce a Grand Theft Auto that their fans will love and will buy as
1:14:28
SPEAKER_00: the original single player gaming experience that it was.
1:14:31
SPEAKER_00: I'm sure they care very deeply about that just from a reputation perspective.
1:14:35
SPEAKER_00: And from an artistic standpoint,
1:14:36
SPEAKER_00: I'm sure there's a lot of people on those teams who care about that.
1:14:39
SPEAKER_00: But from a business standpoint,
1:14:41
SPEAKER_00: I am sure there's also been a tremendous amount of thought and work put
1:14:45
SPEAKER_00: into what does the live part look like?
1:14:49
SPEAKER_00: And that's a huge undertaking, you know, that.
1:14:53
SPEAKER_00: I'm sure that they've been planning for quite some time as well.
1:14:55
SPEAKER_00: So it's a massive, massive thing that they're doing here.
1:14:58
SPEAKER_00: How well will it succeed?
1:14:59
SPEAKER_00: I have no idea,
1:15:00
SPEAKER_00: but it is not just a new Grand Theft Auto,
1:15:02
SPEAKER_00: is I guess the way that I would look at it.
1:15:04
SPEAKER_00: Grand Theft Auto V, I think, was somewhat of a surprise to them.
1:15:07
SPEAKER_00: I don't think they knew it was going to generate that kind of online revenue.
1:15:12
SPEAKER_00: I mean, maybe they had hopes,
1:15:13
SPEAKER_00: but I don't think they knew that it would be that kind of a massive moneymaker
1:15:17
SPEAKER_00: that it was.
1:15:18
SPEAKER_00: And so this is the first product really where they know they will have the audience
1:15:23
SPEAKER_00: for Red Dead Redemption.
1:15:24
SPEAKER_00: They kind of tried to do Red Dead Redemption 2.
1:15:27
SPEAKER_00: They did a similar thing where they tried to have the online thing.
1:15:29
SPEAKER_00: It didn't.
1:15:29
SPEAKER_00: I don't think it hit nearly as big as Grand Theft Auto.
1:15:32
SPEAKER_00: Grand Theft Auto 6 is the first time they're shipping.
1:15:35
SPEAKER_00: A true update to what is their flagship.
1:15:39
SPEAKER_00: And so, you know,
1:15:40
SPEAKER_00: it's equivalent to like a relaunch of Google search or something like
1:15:44
SPEAKER_00: that is what they are doing here.
1:15:47
SPEAKER_00: And so I, you know, I'm sure if I was in charge of that project, I would be sweating bullets.
1:15:52
SPEAKER_00: So I'm sure that they are putting a lot of thought into it.
1:15:55
SPEAKER_00: And it's a very massive undertaking, I'm sure.
1:15:57
SPEAKER_01: I'd like to switch gears to Softwarecraft.
1:15:59
SPEAKER_01: You made this video titled,
1:16:01
SPEAKER_01: Clean Code,
1:16:01
SPEAKER_01: Horrible Performance,
1:16:03
SPEAKER_01: an essay slash video,
1:16:05
SPEAKER_01: showing how Uncle Bob Martin's polymorphism-based refactoring pattern runs about 1.5
1:16:11
SPEAKER_01: to 15 times lower than a plain table switch version.
1:16:15
SPEAKER_01: Can we talk about the responses to this piece?
1:16:17
SPEAKER_00: Well, I guess I can put that in context.
1:16:19
SPEAKER_00: So that is sort of from that course on the Substack.
1:16:22
SPEAKER_00: So it kind of goes with a bunch of other videos that are part of like
1:16:26
SPEAKER_00: the Substack thing.
1:16:27
SPEAKER_00: I guess the first thing I'd say is, I feel like the response to it was very positive.
1:16:32
SPEAKER_00: I was kind of surprised.
1:16:34
SPEAKER_00: There are plenty of people who didn't like it.
1:16:35
SPEAKER_00: Don't get me wrong.
1:16:36
SPEAKER_00: It's controversial to be sure.
1:16:38
SPEAKER_00: But I was surprised at just how many people were enthusiastic about it as well.
1:16:42
SPEAKER_00: But what I would say is it's really I I don't really think it should be
1:16:47
SPEAKER_00: so controversial because There's one thing where people want to just use
1:16:53
SPEAKER_00: the term clean code to mean code that they like or think is written properly.
1:16:58
SPEAKER_00: And that's not something you can argue against, right?
1:17:01
SPEAKER_00: Because that's just, you know, I probably have a version of what I think is clean code.
1:17:05
SPEAKER_00: And obviously, I don't think that's bad, right?
1:17:08
SPEAKER_00: Like, it's my idea of what good code looks like.
1:17:11
SPEAKER_00: So if your idea of clean code is just whatever you want it,
1:17:14
SPEAKER_00: you know,
1:17:15
SPEAKER_00: whatever you happen to think are good programming practices,
1:17:17
SPEAKER_00: I might agree with those programming practices.
1:17:18
SPEAKER_00: I don't know.
1:17:19
SPEAKER_00: So in this particular video,
1:17:20
SPEAKER_00: I was talking specifically about the things that were advocated
1:17:23
SPEAKER_00: that are like very specific things that are said,
1:17:26
SPEAKER_00: like don't have functions over a particular length or these sorts of things, right?
1:17:31
SPEAKER_00: Things should not know the type at runtime or whatever, right?
1:17:34
SPEAKER_00: They don't mesh well when you put them together.
1:17:36
SPEAKER_00: In isolation, some of them might be fine.
1:17:49
SPEAKER_00: So for example,
1:17:50
SPEAKER_00: if you really prefer lots of small functions,
1:17:53
SPEAKER_00: that's actually fine if the compiler can see all those functions and know
1:17:58
SPEAKER_00: that it can safely inline them and collapse them as necessary.
1:18:01
SPEAKER_00: This is a part a lot of people missed about the video,
1:18:04
SPEAKER_00: I guess,
1:18:04
SPEAKER_00: It's a pretty short video,
1:18:05
SPEAKER_00: so I didn't explain anything in detail,
1:18:06
SPEAKER_00: but a lot of redundant code happens when you have lots of tiny,
1:18:11
SPEAKER_00: you know, these little tiny functions.
1:18:13
SPEAKER_00: And if they're all virtual functions in C++,
1:18:15
SPEAKER_00: let's say the compiler can't know for sure which ones of them are being called and
1:18:19
SPEAKER_00: so on.
1:18:19
SPEAKER_00: Even if you put things like final in them,
1:18:22
SPEAKER_00: people have a lot of weird beliefs about how the code works.
1:18:26
SPEAKER_00: You can just go do this testing.
1:18:28
SPEAKER_00: When you have lots of these little functions,
1:18:30
SPEAKER_00: if they're all like statically defined and aren't virtual calls,
1:18:33
SPEAKER_00: if they're just known calls,
1:18:35
SPEAKER_00: like or just member functions,
1:18:36
SPEAKER_00: when I say static, I kind of mean just known to the translation unit, not external.
1:18:40
SPEAKER_00: The compiler can put those together,
1:18:41
SPEAKER_00: collapse all of the redundant code It can actually produce something reasonable
1:18:44
SPEAKER_00: that will run pretty fast out of that.
1:18:46
SPEAKER_00: It can also do things like widen the code path if it needs to vectorize to run
1:18:51
SPEAKER_00: in SIMD and stuff.
1:18:52
SPEAKER_00: The compiler has all these options
1:18:54
SPEAKER_00: to take what is fundamentally not particularly great code in terms
1:18:58
SPEAKER_00: of how you would want it to run at runtime,
1:19:00
SPEAKER_00: but it might be able to turn it into that
1:19:02
SPEAKER_00: because optimizing compilers are pretty heroic these days and the sorts
1:19:07
SPEAKER_00: of things they can do.
1:19:08
SPEAKER_00: If instead you use all of these things that were recommended,
1:19:12
SPEAKER_00: you completely block out the compiler from being able to do those things.
1:19:16
SPEAKER_00: Because if it can't tell what it's doing at runtime,
1:19:18
SPEAKER_00: if it has to leave open the possibility that you substitute it in
1:19:21
SPEAKER_00: a different class here or something like that,
1:19:23
SPEAKER_00: then you end up in a situation where the compiler can't do any of that work.
1:19:27
SPEAKER_00: And people mistakenly think that this is just
1:19:29
SPEAKER_00: because like virtual function calls cost too much or something like that.
1:19:32
SPEAKER_00: That's not what it is.
1:19:33
SPEAKER_00: It's not the cost of the virtual function call.
1:19:35
SPEAKER_00: We could talk about that as a separate thing because you can analyze that cost as well.
1:19:41
SPEAKER_00: It doesn't have much to do with Specifically,
1:19:43
SPEAKER_00: whether it's virtual or not,
1:19:43
SPEAKER_00: it has to do with a lot of things like branch prediction
1:19:45
SPEAKER_00: and how much stuff was getting pushed in the stack and whatever else, right?
1:19:48
SPEAKER_00: But it's the cost of the compiler not being able to do any optimizations.
1:19:52
SPEAKER_00: That's the actual cost.
1:19:54
SPEAKER_00: And that cost can be severe.
1:19:56
SPEAKER_00: I showed only,
1:19:57
SPEAKER_00: I think,
1:19:57
SPEAKER_00: a pretty mild degradation compared to what you would actually see in production
1:20:01
SPEAKER_00: if you really had a huge number of things doing this.
1:20:04
SPEAKER_00: And I think it landed pretty well.
1:20:07
SPEAKER_00: It's a very widely viewed video and a lot of people seem to really like it.
1:20:10
SPEAKER_00: I thought it would be probably even more controversial than it was.
1:20:14
SPEAKER_00: So I was pleased with that.
1:20:16
SPEAKER_00: But yeah, I mean, all that stuff remains true today, I guess is what I'd say.
1:20:20
SPEAKER_00: And I think it's good for people to hear because they need to hear opposing viewpoints.
1:20:23
SPEAKER_00: I think you can write code.
1:20:26
SPEAKER_00: That is maintainable and easy to read,
1:20:30
SPEAKER_00: that doesn't follow those principles in that way,
1:20:33
SPEAKER_00: and that doesn't have those problems.
1:20:35
SPEAKER_00: I don't think you have to do those things,
1:20:37
SPEAKER_00: so I think it's worth exploring other options that are still maintainable,
1:20:41
SPEAKER_00: that are good code, but that allow the compiler to do the right thing.
1:20:56
SPEAKER_01: Used to be super popular in the,
1:20:58
SPEAKER_01: you know, like,
1:20:59
SPEAKER_01: especially when you're building services,
1:21:00
SPEAKER_01: some of those things,
1:21:01
SPEAKER_01: especially in the 2000s, kind of got a little bit out of fashion.
1:21:04
SPEAKER_01: And now it's unclear if it'll come back or not with agents or not.
1:21:08
SPEAKER_00: I don't have that much of a spicy take on that one.
1:21:10
SPEAKER_00: My take is very pragmatic,
1:21:12
SPEAKER_00: which is that if you can identify tests that will save time in total,
1:21:19
SPEAKER_00: that's usually what I try to emphasize.
1:21:20
SPEAKER_00: In other words,
1:21:21
SPEAKER_00: if the amount of time it takes to create and maintain
1:21:24
SPEAKER_00: the tests will actually save us total development time
1:21:28
SPEAKER_00: because they will identify bugs that would be hard for us to find in production
1:21:31
SPEAKER_00: in production.
1:21:33
SPEAKER_00: Or would be very costly if they got out, then great.
1:21:37
SPEAKER_00: And I've used them before.
1:21:38
SPEAKER_00: I talked about working at Rad Game Tools.
1:21:41
SPEAKER_00: I had a regression tester that I ran on the core libraries there that I had written.
1:21:46
SPEAKER_00: They're not called libraries,
1:21:47
SPEAKER_00: but the core routines to make sure that anything that I could be testing
1:21:51
SPEAKER_00: for our customers, I sort of was.
1:21:54
SPEAKER_00: And so I think there's good times for testing.
1:21:57
SPEAKER_00: I would say the part that I don't like about test driven development is the test driven part.
1:22:04
SPEAKER_00: I don't think development should ever be driven by tests.
1:22:06
SPEAKER_00: I think tests are a thing that Now,
1:22:08
SPEAKER_00: could that decision be that for this particular project,
1:22:10
SPEAKER_00: we are going to drive it primarily from the tests?
1:22:12
SPEAKER_00: Yes, that could be a decision that you make, but you shouldn't really
1:22:30
SPEAKER_00: Think of development as something that is primarily test driven,
1:22:34
SPEAKER_00: like by default,
1:22:35
SPEAKER_00: because that might be a very bad decision for some other project
1:22:39
SPEAKER_00: where it just ends up costing you more to have done it that way.
1:22:41
SPEAKER_00: So like with most things, I would advocate for a pragmatic approach to testing.
1:22:46
SPEAKER_00: You should understand the cost of testing,
1:22:50
SPEAKER_00: the cost of developing,
1:22:51
SPEAKER_00: maintaining the test,
1:22:52
SPEAKER_00: and the cost to your codebase if it makes it harder to change your codebase
1:22:55
SPEAKER_00: because tests have to be rewritten
1:22:56
SPEAKER_00: and you therefore don't make changes you should make.
1:22:59
SPEAKER_00: All of that stuff should be in your brain.
1:23:02
SPEAKER_00: And you should make an intelligent decision about what your testing strategy is.
1:23:06
SPEAKER_00: If that decision,
1:23:07
SPEAKER_00: intelligently made,
1:23:09
SPEAKER_00: turns out to be we are going to have a lot of testing on this project,
1:23:13
SPEAKER_00: that may well be a good decision.
1:23:14
SPEAKER_00: I don't think there's an absolute thing you can say about how many tests there should be.
1:23:20
SPEAKER_00: Some projects probably shouldn't have very much.
1:23:22
SPEAKER_00: Maybe some projects should have a lot.
1:23:23
SPEAKER_00: And I think knowing which of those you are doing is part of being
1:23:27
SPEAKER_00: a good software engineer, is I guess what I would say.
1:23:30
SPEAKER_01: You mentioned being a good software engineer,
1:23:32
SPEAKER_01: but before we get into what is a good software engineer,
1:23:35
SPEAKER_01: what does good code mean to you specifically?
1:23:38
SPEAKER_00: So good code to me usually means that you have written something that is
1:23:45
SPEAKER_00: as straightforward to what the machine actually needs to do to solve the problem
1:23:50
SPEAKER_00: as it can be.
1:23:52
SPEAKER_00: And also hopefully that you have,
1:23:55
SPEAKER_00: I guess I'll say,
1:23:57
SPEAKER_00: properly designed Identified ways of breaking it into easily digestible pieces
1:24:04
SPEAKER_00: and named those pieces in ways that are easy for someone to understand,
1:24:08
SPEAKER_00: especially yourself,
1:24:09
SPEAKER_00: because you are very likely to be someone who's going to have to modify it.
1:24:13
SPEAKER_00: That's the way I tend to code.
1:24:14
SPEAKER_00: I try to identify what do I actually need the computer to do.
1:24:18
SPEAKER_00: I try to write as simple as possible the thing that will do that.
1:24:22
SPEAKER_00: And then I try to put that in terms that are, you know, I would say least redundant.
1:24:28
SPEAKER_00: I don't want to see the equation for Euclidean distance scattered throughout my code.
1:24:33
SPEAKER_00: I want to have a function that's like compute that distance and I want to use it.
1:24:36
SPEAKER_00: I want it to then be nicely broken into the pieces that it represents.
1:24:41
SPEAKER_00: And I want those pieces to be reassemblable properly by the compiler in a way
1:24:45
SPEAKER_00: that will produce code that runs very efficiently.
1:24:48
SPEAKER_00: And so that's what I'm usually trying to do when I'm trying to program.
1:24:53
SPEAKER_00: For me, I have never really understood the sort of mentality of there's a difference
1:25:00
SPEAKER_00: between code that is like well architected by some principles and code
1:25:06
SPEAKER_00: that runs quickly because In my experience,
1:25:09
SPEAKER_00: usually the code that is architected properly is also the code that runs quickly.
1:25:13
SPEAKER_00: And yes,
1:25:14
SPEAKER_00: there is a point where if we decide that something absolutely has to get as close
1:25:21
SPEAKER_00: to theoretical maximum as it possibly can, Yes,
1:25:24
SPEAKER_00: we will start to make that code harder to read and modify
1:25:28
SPEAKER_00: because we are now like really over specializing it for this piece of hardware
1:25:33
SPEAKER_00: or whatever.
1:25:34
SPEAKER_00: That's true.
1:25:35
SPEAKER_00: But that point is like, you know, way out on the curve.
1:25:39
SPEAKER_00: It's not the common case.
1:25:41
SPEAKER_00: Most of the time,
1:25:42
SPEAKER_00: assuming you just want code that runs pretty, pretty darn well on most hardware.
1:25:46
SPEAKER_00: The simple, readable version of the code is actually very fast.
1:25:50
SPEAKER_00: It's only once you think you need to have 27 factories and 8,000 microservices
1:25:55
SPEAKER_00: and all these things running that it starts
1:25:57
SPEAKER_00: to be this thing that's like good architecture,
1:26:00
SPEAKER_00: but also like hard to modify, hard to read, run slowly, write all these things.
1:26:06
SPEAKER_00: So I tend to think of like good code.
1:26:08
SPEAKER_00: There's like this nice nexus of runs pretty,
1:26:11
SPEAKER_00: pretty darn well,
1:26:12
SPEAKER_00: easy to read,
1:26:13
SPEAKER_00: easy to maintain,
1:26:15
SPEAKER_00: isn't And related to this, what is a good software engineer to you?
1:26:18
SPEAKER_00: Is it just someone who writes good code or it goes beyond that?
1:26:39
SPEAKER_00: I would say it really depends on the environment a little bit,
1:26:44
SPEAKER_00: because I think I've seen a lot of different kinds of good software engineers,
1:26:49
SPEAKER_00: and so I would liken it more to a,
1:26:53
SPEAKER_00: you know,
1:26:53
SPEAKER_00: if you want a sports You'd imagine something more like a baseball team
1:26:57
SPEAKER_00: where it's like, what's a good baseball player?
1:27:00
SPEAKER_00: Well, it's like, are we talking about a pitcher or a designated hitter?
1:27:03
SPEAKER_00: Right.
1:27:04
SPEAKER_00: And it changes quite dramatically.
1:27:06
SPEAKER_00: So there might be some things like, hey,
1:27:07
SPEAKER_00: if someone's pleasant to work with,
1:27:09
SPEAKER_00: But when you're talking about things that are more specific
1:27:12
SPEAKER_00: to just software engineering and not just being a good employee or something like
1:27:15
SPEAKER_00: that,
1:27:29
SPEAKER_00: I would say I've seen a couple of different kinds.
1:27:31
SPEAKER_00: I've seen people who are like the utility infielder.
1:27:33
SPEAKER_00: They're people who just like they can identify and go and try to fix a problem and succeed.
1:27:38
SPEAKER_00: Even if the code base is kind of wacky and out there,
1:27:41
SPEAKER_00: they're good at getting the lay of the land very quickly
1:27:44
SPEAKER_00: of identifying something that's going on.
1:27:46
SPEAKER_00: And they're not afraid to go in and like,
1:27:48
SPEAKER_00: okay, this is kind of this code base kind of ugly here.
1:27:51
SPEAKER_00: It's okay.
1:27:51
SPEAKER_00: I'm going to patch around.
1:27:52
SPEAKER_00: I'm going to do what I need to do and get things done.
1:27:54
SPEAKER_00: That's a great engineer to have around.
1:27:56
SPEAKER_00: I've also seen great engineers who are the exact opposite of that.
1:28:00
SPEAKER_00: They are just like,
1:28:01
SPEAKER_00: I take this one particular problem that we have,
1:28:06
SPEAKER_00: and eight months later,
1:28:08
SPEAKER_00: I have ground out every last thing there is to know about this.
1:28:13
SPEAKER_00: And sometimes to the point of producing new algorithms that no one's even known
1:28:18
SPEAKER_00: before, that are these breakthrough things, right?
1:28:21
SPEAKER_00: And that's a great software engineer to have on a project if you're going
1:28:24
SPEAKER_00: to be having that kind of thing.
1:28:25
SPEAKER_00: And so I've seen a lot of different people
1:28:28
SPEAKER_00: that I would consider great software engineers,
1:28:29
SPEAKER_00: and they aren't all the same person, right?
1:28:33
SPEAKER_00: So I think that it's kind of important if you're asking it from the standpoint
1:28:37
SPEAKER_00: of like, Hey, you need to put together a team to go build this project.
1:28:41
SPEAKER_00: What's a great software engineer?
1:28:43
SPEAKER_00: I would say the best advice you could give someone in that position is think about the roles.
1:28:48
SPEAKER_00: Think about what kinds of roles there are going to be here.
1:28:52
SPEAKER_00: And don't think great software engineer.
1:28:54
SPEAKER_00: Think great that role, right?
1:28:57
SPEAKER_00: Who is going to be a great pitcher?
1:28:58
SPEAKER_00: Who's going to be a great first baseman?
1:28:59
SPEAKER_00: Who's going to be a great outfielder?
1:29:00
SPEAKER_00: Who's going to be a great this, that, the other thing?
1:29:02
SPEAKER_00: Great third base coach, whatever it is, right?
1:29:05
SPEAKER_00: And that's what you're trying to put together if you're trying to build a team.
1:29:08
SPEAKER_01: Yeah.
1:29:08
SPEAKER_01: So like, it's just not one size fits all.
1:29:10
SPEAKER_01: But I still want to push you a little bit.
1:29:12
SPEAKER_01: Like, what are things that you think are non-negotiable for someone to be
1:29:16
SPEAKER_01: a great software engineer?
1:29:18
SPEAKER_01: We talked about the things that we talked about, which is a recurring theme with you.
1:29:23
SPEAKER_01: What you can do is just going deeper and deeper and understanding the next and next layer.
1:29:27
SPEAKER_01: Understand if you're doing web development, understand React.
1:29:31
SPEAKER_01: Once you understand React, understand what's going on in the DOM, go all the way to assembly.
1:29:35
SPEAKER_01: Once you've done there,
1:29:36
SPEAKER_01: understand how the CPU Is doing operations and branch predictions and some
1:29:40
SPEAKER_01: of those things.
1:29:41
SPEAKER_01: Like to me, that's a skill of like curiosity, driving deeper, crafts, whatever you call it.
1:29:47
SPEAKER_01: There are different ways we could do it.
1:29:49
SPEAKER_01: But along these lines,
1:29:50
SPEAKER_01: what are those traits that you think,
1:29:52
SPEAKER_01: no matter what kind of role we're talking about,
1:29:54
SPEAKER_01: but if you think back of some of the different types of roles that you work with,
1:29:58
SPEAKER_01: like you see some overlap that they all had something.
1:30:02
SPEAKER_00: I would say that it's pretty unusual,
1:30:05
SPEAKER_00: I guess,
1:30:06
SPEAKER_00: that I can't think of someone I would think of as a great software engineer
1:30:10
SPEAKER_00: who didn't know how to read assembly or something.
1:30:14
SPEAKER_00: That is true.
1:30:15
SPEAKER_00: It might be that having that curiosity about how things work and knowing
1:30:19
SPEAKER_00: at some level what's going on is kind of Maybe something that's going
1:30:23
SPEAKER_00: to be very common to a great software engineer,
1:30:26
SPEAKER_00: but I would just underscore the point.
1:30:28
SPEAKER_00: The degree to which they are employing that knowledge may vary quite a bit.
1:30:32
SPEAKER_00: For some of them, that may be their bread and butter, and they're doing that all day.
1:30:35
SPEAKER_00: For others, it's just ridiculous.
1:30:37
SPEAKER_00: And that's great,
1:30:43
SPEAKER_00: but they may not really be doing all that much actually at that kind of level
1:30:50
SPEAKER_00: or thinking about at that level.
1:30:51
SPEAKER_00: They're just going like, yeah,
1:30:52
SPEAKER_00: I know we got to kind of push, okay,
1:30:54
SPEAKER_00: this stuff's going to have to be done in batch because I just kind of know
1:30:57
SPEAKER_00: that that's how the machine's going to have to handle it.
1:30:59
SPEAKER_00: So I'll make sure I write the code that way or whatever.
1:31:02
SPEAKER_00: Yes, but so there's a little bit of that.
1:31:05
SPEAKER_00: The other thing that I would say maybe is actually like not being dogmatic
1:31:11
SPEAKER_00: about things that they haven't actually themselves proved out is probably
1:31:15
SPEAKER_00: a huge one.
1:31:16
SPEAKER_00: I find there's a lot And sometimes it is like you could demonstrate
1:31:32
SPEAKER_00: So I would say people who actually focus on what works in practice.
1:31:49
SPEAKER_00: Is a huge plus.
1:31:52
SPEAKER_00: And you can apply that anywhere into anything, right?
1:31:55
SPEAKER_00: Not just saying, oh,
1:31:56
SPEAKER_00: the flavor of the month is that we're writing everything with classes
1:31:59
SPEAKER_00: and virtual functions and hierarchies or whatever.
1:32:01
SPEAKER_00: It's like,
1:32:01
SPEAKER_00: did you actually determine that that results in less code or that
1:32:04
SPEAKER_00: the Do we do any testing to figure out if this is helping us rather than hurting us?
1:32:08
SPEAKER_00: And the answer oftentimes is no.
1:32:10
SPEAKER_00: Or if it was at all,
1:32:12
SPEAKER_00: it was extremely shoddily done and you would not take those results as conclusive
1:32:16
SPEAKER_00: in any way.
1:32:17
SPEAKER_00: And so it's like being more skeptical about coding practices and actually trying
1:32:21
SPEAKER_00: to focus on what is working in practice and what we can demonstrate and measure
1:32:25
SPEAKER_00: in some kind of a process.
1:32:28
SPEAKER_00: Repeatable way is, I think, a really great thing for a software engineer to have as well.
1:32:32
SPEAKER_00: So people who don't tend to fall prey to that,
1:32:34
SPEAKER_00: just like I watched some presentation and someone at Google says,
1:32:37
SPEAKER_00: always call memset or never use if statements or whatever it is.
1:32:40
SPEAKER_00: Like, if that's the level that you're thinking at,
1:32:43
SPEAKER_00: then I probably am not going to put you in that category
1:32:46
SPEAKER_00: of really good software engineer because that's not how it works.
1:32:49
SPEAKER_01: Plus, it's not that hard to try these things out or set up or run an experiment.
1:32:56
SPEAKER_01: Now, the final topic I wanted to touch on,
1:32:58
SPEAKER_01: which I deliberately didn't get into until now,
1:33:01
SPEAKER_01: is AI and how it's changing your work.
1:33:04
SPEAKER_01: And I'd like to start with that.
1:33:06
SPEAKER_01: In the work that you're doing at Molly Rocket with this project that is
1:33:11
SPEAKER_01: yet unreleased, how are you using AI tools if you're using them at all?
1:33:17
SPEAKER_00: We are not using them at all.
1:33:19
SPEAKER_01: So you're doing it just like before, you're writing your code.
1:33:23
SPEAKER_01: What made you decide to take this path?
1:33:26
SPEAKER_00: Well, we're a little bit different, obviously, in the sense for two reasons.
1:33:31
SPEAKER_00: One is that we,
1:33:34
SPEAKER_00: like I said,
1:33:34
SPEAKER_00: we are kind of,
1:33:35
SPEAKER_00: we have like sort of two projects here and the Substack is our primary focus
1:33:40
SPEAKER_00: and this other one is a thing that we're doing because we want to do it.
1:33:44
SPEAKER_00: And when you think about that perspective, it's like, well, why did you want to do it?
1:33:47
SPEAKER_00: Well, the reason that I want to program like things in a game is because I want to program them.
1:33:54
SPEAKER_00: If I just wanted an AI to program them,
1:33:56
SPEAKER_00: I probably,
1:33:57
SPEAKER_00: you know, first of all, we'd probably just go use a licensed engine, right?
1:33:59
SPEAKER_00: Like I wouldn't even bother asking AI to do it.
1:34:02
SPEAKER_01: I'd just go get the Unreal Engine, right?
1:34:05
SPEAKER_00: Or something like that and so on.
1:34:07
SPEAKER_00: So I think a little bit of that decision is probably not that relevant
1:34:10
SPEAKER_00: to your audience because it's more about what do you want to do?
1:34:13
SPEAKER_00: Why are you spending this time, right?
1:34:15
SPEAKER_00: It's a philosophical question, not a productivity question.
1:34:19
SPEAKER_00: So it's not like I evaluated it and said,
1:34:21
SPEAKER_00: I don't think this will save us time or I have questions about the copyrightability
1:34:26
SPEAKER_00: of it or the ethics of it or all the sorts of things
1:34:28
SPEAKER_00: that you could rightfully evaluate AI on.
1:34:31
SPEAKER_00: It wasn't necessarily that.
1:34:33
SPEAKER_00: It's more just like this does not further the goals of the project to use it.
1:34:38
SPEAKER_00: So it kind of was a non-issue at that point, right?
1:34:40
SPEAKER_00: Stepping out a little bit more to a broader philosophical framework about AI,
1:34:45
SPEAKER_00: I guess what I would also say is I think that regardless
1:34:50
SPEAKER_00: of what you think will happen with AI in the future,
1:34:53
SPEAKER_00: because obviously we don't really have any weight.
1:34:56
SPEAKER_00: I
1:35:00
SPEAKER_00: think there will probably also be at some point a notion
1:35:05
SPEAKER_00: of like traditional handcrafting that will come into play because we've seen this
1:35:10
SPEAKER_00: in most other times when you automate something.
1:35:14
SPEAKER_00: If you automate making furniture and you have like Ikea or whatever,
1:35:18
SPEAKER_00: that doesn't mean that there isn't some weird guy down in the industrial district
1:35:23
SPEAKER_00: of your city making crazy wood tables with iron and welding and something.
1:35:27
SPEAKER_00: And that That's just a thing that people are still doing, and some people want that table.
1:35:32
SPEAKER_00: I don't necessarily have an explanation for it,
1:35:35
SPEAKER_00: and I'm not trying to argue that it has more or less value,
1:35:39
SPEAKER_00: but it's just something that happens, right?
1:35:41
SPEAKER_00: And so if I imagine what I love about computers and what I want to do
1:35:47
SPEAKER_00: with computers,
1:35:48
SPEAKER_00: and you asked me,
1:35:49
SPEAKER_00: move that into some other context, which of these people would you be?
1:35:53
SPEAKER_00: My answer is always, I'd be the organic farming guy.
1:35:56
SPEAKER_00: I'd be the guy who's making the weird table in the industrial district.
1:36:00
SPEAKER_00: I have no interest in managing a division at IKEA.
1:36:03
SPEAKER_00: I literally couldn't care less about that, right?
1:36:06
SPEAKER_00: And so I think for me,
1:36:08
SPEAKER_00: another reason why I'm not that interested in pursuing AI is because I would like
1:36:14
SPEAKER_00: to be part of whatever the set of people are who are going
1:36:18
SPEAKER_00: to keep this traditional craft alive Just because that's something humans do,
1:36:24
SPEAKER_00: not because we're trying to say that that's the right business case, right?
1:36:28
SPEAKER_00: If that makes sense.
1:36:29
SPEAKER_01: And at this point,
1:36:29
SPEAKER_01: there is a bit of a tradition,
1:36:31
SPEAKER_01: if you will,
1:36:31
SPEAKER_01: even if we assume that these machines will do as good or better than humans did
1:36:36
SPEAKER_01: for like What, 60 plus years?
1:36:39
SPEAKER_01: We've all only exclusively handwritten software because that's how it got done, right?
1:36:44
SPEAKER_01: Like a lot of us,
1:36:45
SPEAKER_01: anyone who started coding before 2023 or at the end of 2022
1:36:49
SPEAKER_01: or probably honestly 2024 when these things have gotten like decently good,
1:36:53
SPEAKER_01: you just wrote it by hand, a lot of it or tap complete still counts.
1:36:57
SPEAKER_00: Yeah, and I guess I would say,
1:36:59
SPEAKER_00: like, again, it's just, you know, why do that, right, is the question.
1:37:03
SPEAKER_00: It's like, I don't know why humans do that.
1:37:05
SPEAKER_00: Humans do that because it's something humans do, right?
1:37:07
SPEAKER_00: Humans like to do things themselves sometimes.
1:37:10
SPEAKER_00: You know, people can...
1:37:12
SPEAKER_00: Buy a hat.
1:37:13
SPEAKER_00: They can buy a wool hat trivially or they can buy whatever.
1:37:17
SPEAKER_00: And then someone's out there knitting a hat right now.
1:37:20
SPEAKER_00: That's just it's just something humans do.
1:37:22
SPEAKER_00: They like to make things by hand sometimes and at varying levels of handmade.
1:37:27
SPEAKER_00: You know, there's some people just buy the wool or whatever or buy the pre-made yard.
1:37:31
SPEAKER_00: Some people raise the sheep or whatever and shear it, right?
1:37:35
SPEAKER_00: Like you can go arbitrarily far and you can find somebody who's going all the way, right?
1:37:40
SPEAKER_00: Even further than probably I would ever go if I was in that thing.
1:37:43
SPEAKER_00: So, you know, you could imagine someone making their own hardware these days, right?
1:37:47
SPEAKER_00: I'm not doing that.
1:37:49
SPEAKER_00: And so that's kind of my take on it.
1:37:51
SPEAKER_00: So I'm kind of the last one to ask about AI coding or what you might want to do
1:37:56
SPEAKER_00: with it, I really have nothing of value to add.
1:37:59
SPEAKER_01: Yeah, but I am interested in asking you through the lens of the games industry,
1:38:04
SPEAKER_01: and we touched on games engines arriving,
1:38:08
SPEAKER_01: and now so many more people can make games.
1:38:11
SPEAKER_01: Not everyone, but it's a lot easier to enter.
1:38:14
SPEAKER_01: What are you observing in terms of most people outside of
1:38:18
SPEAKER_01: who are still handcrafting code because they want
1:38:20
SPEAKER_01: to are using these AI coding agents for two reasons.
1:38:23
SPEAKER_01: Either it just makes sense and they realize, well,
1:38:27
SPEAKER_01: this thing can now generate code as good as I did,
1:38:30
SPEAKER_01: which was a turning point in January.
1:38:31
SPEAKER_01: I had the turning point actually myself.
1:38:33
SPEAKER_01: Or some are actually just pushed with corporate mandates of like you need
1:38:37
SPEAKER_01: to use these tools and eventually they kind of get on board,
1:38:40
SPEAKER_01: whether willingly or unwillingly.
1:38:42
SPEAKER_01: But so many folks are having AI write the code for them.
1:38:46
SPEAKER_01: They're prompting it, but they're doing it.
1:38:49
SPEAKER_01: What do you observe of the effect having from your vantage point?
1:38:54
SPEAKER_01: May that be on quality, craftsmanship, on output, speed, etc.
1:39:01
SPEAKER_01: What are you seeing?
1:39:03
SPEAKER_00: I think it's a little too early to assess,
1:39:05
SPEAKER_00: to be honest,
1:39:06
SPEAKER_00: because kind of as you pointed out,
1:39:08
SPEAKER_00: obviously there's been people who maybe,
1:39:12
SPEAKER_00: you know,
1:39:13
SPEAKER_00: we might derogatorily call AI shills who have been saying that it was producing
1:39:18
SPEAKER_00: as good a code as humans for, you know, two years now or something like that.
1:39:22
SPEAKER_00: Right.
1:39:22
SPEAKER_01: Yeah.
1:39:23
SPEAKER_00: But in reality,
1:39:24
SPEAKER_00: the people whose opinion I would trust more,
1:39:28
SPEAKER_00: none of them thought it was really all that usable until much more recently, right?
1:39:33
SPEAKER_00: And so we really haven't,
1:39:34
SPEAKER_00: they haven't had very many months to actually be thinking Figuring out how
1:39:38
SPEAKER_00: to use this thing or to determine to what extent they can use it and how,
1:39:43
SPEAKER_00: what it's best at,
1:39:45
SPEAKER_00: what the workflow looks like that makes it produce the best results.
1:39:48
SPEAKER_00: It seems like at the moment,
1:39:49
SPEAKER_00: I would say probably need to give it at least another six months,
1:39:53
SPEAKER_00: if not another year or something to Let everyone kind of shake out,
1:39:57
SPEAKER_00: like, what are actually the best ways to use this thing?
1:40:01
SPEAKER_00: I know tons of people in the game industry are using it,
1:40:04
SPEAKER_00: so I know that they are doing various things with it.
1:40:08
SPEAKER_00: Whether those things are the same sorts of things they will eventually think are
1:40:14
SPEAKER_00: the way they like— The things that they're doing right now may be like,
1:40:18
SPEAKER_00: oh, that was kind of dumb.
1:40:19
SPEAKER_00: Like, you shouldn't have used it that way.
1:40:20
SPEAKER_00: You should do this other thing with it, and it's way more productive or something.
1:40:24
SPEAKER_00: So I feel like it's probably too early to assess.
1:40:27
SPEAKER_00: We haven't seen any real, like,
1:40:30
SPEAKER_00: obvious, like, oh, wow, like,
1:40:31
SPEAKER_00: you know, the,
1:40:32
SPEAKER_00: you know,
1:40:33
SPEAKER_00: Fortnite ship's Once a week now and it's bug free,
1:40:36
SPEAKER_00: like nothing particularly interesting has happened in terms of output there.
1:40:40
SPEAKER_00: But again, it's been what, like five months or something.
1:40:43
SPEAKER_00: So it's just it's way it's way too too early to see how it actually gets integrated
1:40:48
SPEAKER_00: into a reliable process.
1:40:50
SPEAKER_00: Right.
1:40:50
SPEAKER_01: Yeah.
1:40:51
SPEAKER_01: And there are I know there are some companies who are now tying up,
1:40:54
SPEAKER_01: let's say, agents fixing bugs, but that's only a few months old.
1:40:57
SPEAKER_01: The oldest software that's widespread that is written now.
1:41:01
SPEAKER_01: Close to 100% by agents is from the labs, OpenAI's Codex and Entropics Cloud Code.
1:41:07
SPEAKER_01: But even there, it's been since November or some parts of it, December.
1:41:12
SPEAKER_01: So like maybe six months.
1:41:14
SPEAKER_01: And it's different, right?
1:41:15
SPEAKER_01: That is a product they're selling.
1:41:16
SPEAKER_01: So there's, I'm not sure we'll know for sure.
1:41:21
SPEAKER_01: Like, is it truly 100%?
1:41:22
SPEAKER_01: How much?
1:41:23
SPEAKER_01: You know, there's a...
1:41:24
SPEAKER_00: Sure, sure.
1:41:43
SPEAKER_00: Most of these things are currently presented as tools,
1:41:46
SPEAKER_00: meaning a human has to operate them at least in some way,
1:41:50
SPEAKER_00: like at least setting it up to do what it's going to do.
1:41:54
SPEAKER_00: And therefore, you have to give it some time.
1:41:59
SPEAKER_00: You know, nobody currently is selling a product where it's just like, oh, just...
1:42:03
SPEAKER_00: Turn this thing on and it will just ship Fortnite by itself forever
1:42:06
SPEAKER_00: and you can just get rid of all your engineers.
1:42:08
SPEAKER_00: Like no one's actually selling that product yet, right?
1:42:10
SPEAKER_00: We could evaluate that product because we'd be like, did anyone do it?
1:42:12
SPEAKER_00: Did it start shipping Fortnite on its own, right?
1:42:14
SPEAKER_00: So if it's still something where humans have to kind of figure out how they want
1:42:19
SPEAKER_00: to like slot it into what they're doing,
1:42:21
SPEAKER_00: then it's entirely possible that the reason...
1:42:24
SPEAKER_00: That we haven't seen some big uptick in productivity that would be obvious to
1:42:29
SPEAKER_00: an external observer is because it's going to take a while for people to shake
1:42:33
SPEAKER_00: that out.
1:42:34
SPEAKER_00: Or maybe the AIs need to get a little bit better.
1:42:36
SPEAKER_00: Maybe we've got to go through some more update steps or whatever.
1:42:39
SPEAKER_00: I'm not sure.
1:42:40
SPEAKER_00: So there's all that on the table.
1:42:42
SPEAKER_00: Then there's another possibility,
1:42:43
SPEAKER_00: which is that it actually already has worked,
1:42:46
SPEAKER_00: but just the productivity boost isn't as big as would be obvious.
1:42:50
SPEAKER_00: If people got 10% more productive,
1:42:53
SPEAKER_00: that would still be pretty impressive because it's hard to get a 10%
1:43:05
SPEAKER_00: It's not really externally observable that clearly if that's what you got,
1:43:09
SPEAKER_00: but it may have happened, right?
1:43:11
SPEAKER_00: So it's really hard for all of those reasons.
1:43:14
SPEAKER_00: At some point,
1:43:15
SPEAKER_00: if the AIs are really fantastic and people figure out how to use them really well,
1:43:20
SPEAKER_00: it should be obvious.
1:43:22
SPEAKER_00: It should be like five people are now shipping four at night instead of 5,000
1:43:25
SPEAKER_00: or whatever, right?
1:43:27
SPEAKER_00: But until that point,
1:43:28
SPEAKER_00: it's really hard to know because it's just like,
1:43:30
SPEAKER_00: especially if it was small, it'd be hard for us to see.
1:43:33
SPEAKER_01: Well, this is anecdotal,
1:43:34
SPEAKER_01: but I'm getting a lot of data points and messages from software engineers
1:43:37
SPEAKER_01: and managers.
1:43:38
SPEAKER_01: One impact it's having is there's this kind of like AI fatigue slash burnout
1:43:43
SPEAKER_01: from software developers who are like, look, I am good at coding.
1:43:47
SPEAKER_01: I've always been good at it.
1:43:49
SPEAKER_01: I enjoyed the work to various extents.
1:43:52
SPEAKER_01: But since this AI thing happened since the end of the year,
1:43:55
SPEAKER_01: beginning of the year,
1:43:56
SPEAKER_01: since it's actually I'm now prompting and now all my code is generated,
1:44:00
SPEAKER_01: whether that's corporate mandates or is just faster.
1:44:03
SPEAKER_01: I'm starting to lose my drive.
1:44:06
SPEAKER_01: Why am I here?
1:44:08
SPEAKER_01: Anyone could do this.
1:44:09
SPEAKER_01: And I think there's a sense of I'm using a lot less of what I'm capable of.
1:44:13
SPEAKER_01: There's all this pressure from above to be more productive with it.
1:44:18
SPEAKER_01: I'm seeing more and more signs that it's,
1:44:21
SPEAKER_01: whatever we call it, burnout, AI fatigue, et cetera, loss of motivation.
1:44:25
SPEAKER_01: I haven't seen a technology,
1:44:27
SPEAKER_01: I don't remember technology having this widespread impact everywhere.
1:44:30
SPEAKER_01: I'm hearing from folks at some of the leading,
1:44:34
SPEAKER_01: not AI companies per se,
1:44:36
SPEAKER_01: but big enough database providers who are now hugely into AI and they're powering
1:44:41
SPEAKER_01: a lot of the things, traditional companies, modern companies, everywhere.
1:44:46
SPEAKER_01: Have you observed some of this thing?
1:44:49
SPEAKER_01: Would you have any advice or any pointers to folks who are feeling like this right now?
1:44:56
SPEAKER_00: I guess I would say observed no, heard about yes, I guess is what I would say.
1:45:04
SPEAKER_00: Like I have talked to people who have been like such and such has been having
1:45:09
SPEAKER_00: a really hard time with this or such and such.
1:45:12
SPEAKER_00: I've definitely heard that interacted directly with someone, not currently no.
1:45:18
SPEAKER_00: And part of that is probably largely because most of the people I talk to Have
1:45:24
SPEAKER_00: a fair amount of latitude with what they do and how they do it.
1:45:29
SPEAKER_00: A lot of the people that I talk to on a daily basis are able
1:45:33
SPEAKER_00: to make their own decisions about what they want to do with AI and so on.
1:45:37
SPEAKER_00: And so I don't necessarily hear from as many people who are going to be in
1:45:40
SPEAKER_00: a position where some manager told them, This is just what you have to do.
1:45:44
SPEAKER_01: This is very interesting because one thing that keeps coming back,
1:45:47
SPEAKER_01: and Armin Ronitur was telling me the same thing on the podcast,
1:45:50
SPEAKER_01: is he's observed that autonomy at your work,
1:45:54
SPEAKER_01: how autonomous you are at your work,
1:45:55
SPEAKER_01: how many decisions you can make on what you work on, how you do your work.
1:45:59
SPEAKER_01: The people who have a lot of that are typically like,
1:46:03
SPEAKER_01: oh great,
1:46:03
SPEAKER_01: I can use this for this,
1:46:04
SPEAKER_01: I can use this tool,
1:46:06
SPEAKER_01: but the people who are told You know,
1:46:08
SPEAKER_01: like beforehand, you're given a ticket or the PM tells you this.
1:46:12
SPEAKER_01: They don't have much wiggle room.
1:46:13
SPEAKER_01: And now those folks are seeing it way more as a threat because,
1:46:16
SPEAKER_01: of course,
1:46:16
SPEAKER_01: subconsciously or consciously,
1:46:18
SPEAKER_01: they're thinking, well, this thing could automate my job.
1:46:21
SPEAKER_01: It's now or it made it made from that little effort I had to do that.
1:46:25
SPEAKER_01: It took it away as well.
1:46:26
SPEAKER_01: So I wonder if there's a connection here.
1:46:28
SPEAKER_00: I mean, that sounds totally logical, right?
1:46:30
SPEAKER_00: If you're somebody with a high degree of autonomy, then when are you going to reach for an AI?
1:46:35
SPEAKER_00: Well, whenever there's something that you didn't want to do, right?
1:46:38
SPEAKER_00: So kind of by definition,
1:46:39
SPEAKER_00: I think at that point, you're going to have a much more positive experience.
1:46:43
SPEAKER_00: Yeah.
1:46:43
SPEAKER_00: Yeah.
1:46:43
SPEAKER_00: Yeah.
1:47:00
SPEAKER_00: Whereas, yeah,
1:47:01
SPEAKER_00: if you're just told like you had this thing that you wanted to do
1:47:03
SPEAKER_00: and you were told you can't just do it yourself,
1:47:05
SPEAKER_00: you have to do it with the AI,
1:47:07
SPEAKER_00: you know,
1:47:07
SPEAKER_00: and by the way,
1:47:08
SPEAKER_00: we just had layoffs or whatever,
1:47:10
SPEAKER_00: you know,
1:47:10
SPEAKER_00: a lot of that stuff obviously could totally change your mental reaction thing
1:47:14
SPEAKER_00: because now it's not you deciding to use an AI
1:47:17
SPEAKER_00: because there's something you didn't want to do that you thought the AI could do
1:47:20
SPEAKER_00: for you.
1:47:21
SPEAKER_00: Now it's you just being told that you're supposed to be using this AI
1:47:24
SPEAKER_00: to automate whatever your job used to be.
1:47:26
SPEAKER_00: You can see pretty obviously why that would have different psychological effects
1:47:29
SPEAKER_00: on people, right?
1:47:30
SPEAKER_01: So I guess it might be just an idea for folks in this situation
1:47:34
SPEAKER_01: that now you might want to evaluate your current position or
1:47:39
SPEAKER_01: if you're interviewing your next position based on how much autonomy will you have?
1:47:43
SPEAKER_01: Because the more autonomy you'll have, the more likely.
1:47:45
SPEAKER_01: You're going to have control over how you're using this stuff,
1:47:48
SPEAKER_01: how much you can experiment versus being given a mandate that,
1:47:51
SPEAKER_01: I don't know,
1:47:51
SPEAKER_01: we're expecting you to have this output increase or output change, whatever that is.
1:47:56
SPEAKER_01: I wonder if this will reevaluate some of,
1:47:58
SPEAKER_01: you know, like what is considered an attractive position?
1:48:01
SPEAKER_01: Because like,
1:48:01
SPEAKER_01: for example,
1:48:01
SPEAKER_01: big tech was considered a great place to work because high compensation,
1:48:06
SPEAKER_01: pretty clear expectations, like easy to understand career advancement.
1:48:10
SPEAKER_01: Yeah.
1:48:11
SPEAKER_01: But now they're the ones who are starting to measure your AI usage,
1:48:15
SPEAKER_01: which is going to like giving a kind of a bit of a handcuff
1:48:18
SPEAKER_01: of what we're expecting you to do.
1:48:19
SPEAKER_01: Or there's where you might have,
1:48:22
SPEAKER_01: there might be mass layoffs, which again, you have no control over, right?
1:48:25
SPEAKER_01: Like it's getting one more or inside of metaphors reassignments
1:48:28
SPEAKER_01: of like you will now do labeling for X months.
1:48:32
SPEAKER_00: I mean, you could sort of think of,
1:48:34
SPEAKER_00: you know,
1:48:34
SPEAKER_00: could we coin the phrase,
1:48:36
SPEAKER_00: are you using an AI to do your Is it an AI using you to do your job?
1:48:41
SPEAKER_00: Because at some point, it definitely felt like Meta, for example, from your reports on it.
1:48:48
SPEAKER_00: And I have seen the same thing said by other people.
1:48:51
SPEAKER_00: So it does not sound like a one source kind of thing.
1:48:54
SPEAKER_00: It sounds like this was kind of just accepted as fact that they kind
1:48:58
SPEAKER_00: of just were using you as AI training.
1:49:01
SPEAKER_00: You're just there to train the AI to do it so that we don't need you anymore.
1:49:07
SPEAKER_00: Thinking about that from a perspective of choosing your job,
1:49:10
SPEAKER_00: it does make some sense if you have any latitude, right?
1:49:14
SPEAKER_00: But yeah.
1:49:15
SPEAKER_01: As closing, what are one or two books that you would recommend that had an impact on you?
1:49:20
SPEAKER_00: I'm going to have a hot take here if I might, because it's sort of a push I've been on recently.
1:49:29
SPEAKER_00: I don't think people should necessarily take a book recommendation from me.
1:49:35
SPEAKER_00: I wanna recommend that people read a paper.
1:49:38
SPEAKER_00: I'm trying to get more people to just read papers because I realized I read a ton of papers.
1:49:45
SPEAKER_00: Like I am constantly reading Papers on things that I am interested in.
1:49:50
SPEAKER_00: Like if I'm gonna go do some programming in an area that I haven't done before,
1:49:55
SPEAKER_00: I will read a ton of papers.
1:49:56
SPEAKER_00: I'll crawl the references on papers.
1:49:58
SPEAKER_00: I'll read a survey and go gather all those references and read those references
1:50:01
SPEAKER_00: and crawl them back.
1:50:02
SPEAKER_00: And I find that I learn a ton that way.
1:50:04
SPEAKER_00: And I feel like a lot of programmers just don't do that.
1:50:07
SPEAKER_00: And so my recommendation would, you don't have to read a specific paper.
1:50:10
SPEAKER_00: I'm not gonna give you a specific paper to read this one.
1:50:13
SPEAKER_00: Just think about the domain you're programming in Do a search on Google Scholar
1:50:19
SPEAKER_00: for some part of that that you're interested in.
1:50:21
SPEAKER_00: Try reading a paper, following the references, see what you think.
1:50:25
SPEAKER_00: I think it's a great thing to do,
1:50:27
SPEAKER_00: and I get a tremendous amount of not just enjoyment from the education of it,
1:50:31
SPEAKER_00: but also just like...
1:50:32
SPEAKER_00: I have more knowledge about what I'm doing pretty much every time I do this,
1:50:36
SPEAKER_00: even if it's just to learn a little bit more about the historical record
1:50:39
SPEAKER_00: of how things got discovered.
1:50:41
SPEAKER_00: But a lot of times it's just like I learned about whole new techniques I just was not aware of.
1:50:47
SPEAKER_00: Way too much out there for any one person to know.
1:50:49
SPEAKER_00: Again, since I don't currently use AI in my workflow,
1:50:55
SPEAKER_00: I couldn't say,
1:50:56
SPEAKER_00: but my assumption would be that AIs would also be very good
1:51:00
SPEAKER_00: at helping you find some papers to read if you were interested.
1:51:02
SPEAKER_00: Thanks so much for having me.
1:51:03
SPEAKER_00: It's been a pleasure.
1:51:25
SPEAKER_01: I've been wanting
1:51:25
SPEAKER_01: to talk about performance with Casey for such a long time
1:51:27
SPEAKER_01: and I'm glad that we finally made it happen.
1:51:29
SPEAKER_01: I kind of wish the industry had more people as excited and interested
1:51:32
SPEAKER_01: in high performance code as Casey is.
1:51:34
SPEAKER_01: If you made it to the end of this episode, you might just be one of them.
1:51:38
SPEAKER_01: I appreciate that Casey did not beat around the bush.
1:51:40
SPEAKER_01: If you care about performance,
1:51:41
SPEAKER_01: you want to be able to read assembly and spend some time reading it.
1:51:45
SPEAKER_01: Reading assembly is several times easier than writing it.
1:51:47
SPEAKER_01: If you can read assembly, you can see what's happening at the machine level.
1:51:51
SPEAKER_01: And it's a lot easier to understand,
1:51:52
SPEAKER_01: for example,
1:51:52
SPEAKER_01: why a programming language like Python is much slower than something like Rust or C
1:51:56
SPEAKER_01: when you see the assembly code for simple operations.
1:51:59
SPEAKER_01: I was chuckling when Casey talked about these blog posts
1:52:01
SPEAKER_01: about how we rewrote our services in a new language
1:52:04
SPEAKER_01: and got 10x performance improvement and how those rewrites are usually not about
1:52:07
SPEAKER_01: the new language with fixing the architecture that caused the performance issues
1:52:11
SPEAKER_01: to start with.
1:52:12
SPEAKER_01: And although we did not talk much about AI,
1:52:14
SPEAKER_01: I found it amusing for Casey to say that
1:52:16
SPEAKER_01: the games industry had its AI moment years ago,
1:52:19
SPEAKER_01: when game engines became accessible to pretty much anyone wanting to build a game.
1:52:23
SPEAKER_01: Before, large teams were needed to build both a game engine and a game,
1:52:26
SPEAKER_01: and now teams of one or two can create full-blown games.
1:52:29
SPEAKER_01: After a brief spike of positive effects,
1:52:32
SPEAKER_01: with lots of new,
1:52:32
SPEAKER_01: good games released,
1:52:34
SPEAKER_01: games have flooded the market in such great numbers that it's now impossible for
1:52:37
SPEAKER_01: a new game to become hit organically.
1:52:39
SPEAKER_01: So marketing and distribution becomes mandatory even for great games.
1:52:43
SPEAKER_01: For more deep dives related to game development and performance software,
1:52:46
SPEAKER_01: check out the link to Pragmatic Engineer deep dives on these topics.
1:52:49
SPEAKER_01: If you've enjoyed this podcast,
1:52:51
SPEAKER_01: please do subscribe on your favorite podcast platform and on YouTube.
1:52:54
SPEAKER_01: And a big thank you if you also leave a rating on the show.
1:52:56
SPEAKER_01: Appreciate it and see you in the next one.