Latest / The Pragmatic Engineer / Stop being skeptical about AI for development with Charity Majors

Transcript

Stop being skeptical about AI for development with Charity Majors · The Pragmatic Engineer

• 0:00
  SPEAKER_01: Why are there firmly two camps within software engineering when it comes to AI,
• 0:04
  SPEAKER_01: those hating its effects and those who are AI-pilled?
• 0:07
  SPEAKER_01: Charity Majors emphasizes both camps and thinks they are talking alongside one another.
• 0:12
  SPEAKER_01: Charity is a co-founder and CTO of Honeycomb,
• 0:14
  SPEAKER_01: previously worked at Facebook and Parse,
• 0:16
  SPEAKER_01: and is one of my favorite voices in engineering.
• 0:18
  SPEAKER_01: Today,
• 0:19
  SPEAKER_01: we discuss what it would take for us engineers to ship code we have never read
• 0:23
  SPEAKER_01: and why this is more of a when question, not an if question.
• 0:26
  SPEAKER_01: Why are reliability is quietly getting worse across the industry
• 0:29
  SPEAKER_01: and why it will take some time to recover.
• 0:31
  SPEAKER_01: Career advice in this age of AI.
• 0:33
  SPEAKER_01: Why middle managers should consider going back to being IC
• 0:36
  SPEAKER_01: and why junior engineers will be okay.
• 0:38
  SPEAKER_01: If you want to hear from someone who was skeptical about AI in 2025
• 0:41
  SPEAKER_01: but has changed her mind based on the evidence, this episode is for you.
• 0:45
  SPEAKER_01: In today's episode,
• 0:46
  SPEAKER_01: Charity will say,
• 0:47
  SPEAKER_01: spoiler alert,
• 0:48
  SPEAKER_01: that the question is not if we will stop reading code written by AI, but when.
• 0:51
  SPEAKER_01: And we should take lessons from Ops and QA on how they prove that software
• 0:54
  SPEAKER_01: that others wrote works in prod.
• 0:57
  SPEAKER_01: And she's got a very good point.
• 0:59
  SPEAKER_01: As any Ops Engineer or S3 will tell you,
• 1:01
  SPEAKER_01: that's how software has always been written,
• 1:03
  SPEAKER_01: by unreliable agents from their point of view.
• 1:05
  SPEAKER_01: That is, software engineers like me, your colleagues, or you.
• 1:08
  SPEAKER_01: And let's face it, you probably haven't read all the code in your codebase either.
• 1:12
  SPEAKER_01: This is where I need to mention our presenting sponsor Antithesis.
• 1:15
  SPEAKER_01: Antithesis verifies software written by unreliable agents.
• 1:18
  SPEAKER_01: It runs your whole system in a hostile simulation and roots out the bugs for you.
• 1:22
  SPEAKER_01: It does this by using an approach called deterministic simulation testing or DST.
• 1:27
  SPEAKER_01: Antithesis turbocharges testing by running your whole system under aggressive fault injection.
• 1:31
  SPEAKER_01: Imagine Antithesis hundreds or thousands of versions of the Mario game running,
• 1:35
  SPEAKER_01: each instance aggressively trying to break the game
• 1:37
  SPEAKER_01: with increasingly weird input combinations.
• 1:40
  SPEAKER_01: If it finds a breakage, this is where the determinism comes in.
• 1:43
  SPEAKER_01: Instead of you having to try to reproduce a tricky bug you saw in production,
• 1:47
  SPEAKER_01: Antithesis can provide you with a perfect deterministic replay
• 1:49
  SPEAKER_01: of anything it finds every time.
• 1:51
  SPEAKER_01: With Antistesis,
• 1:52
  SPEAKER_01: you can specify properties at the whole system level
• 1:55
  SPEAKER_01: and Antistesis will actively try to disprove them.
• 1:58
  SPEAKER_01: So you can be confident that if your system holds up in Antistesis,
• 2:01
  SPEAKER_01: it will hold up in production.
• 2:02
  SPEAKER_01: Head over to antistesis.com slash pragmatic to learn more.
• 2:06
  SPEAKER_01: Charity, it's so nice to do this in person.
• 2:08
  SPEAKER_00: You're in my city.
• 2:09
  SPEAKER_00: This is amazing.
• 2:11
  SPEAKER_02: So today I wanted to kick off with AI,
• 2:14
  SPEAKER_02: but before we kick off with AI,
• 2:15
  SPEAKER_02: I just want to make it kind of clear for people who don't know you that,
• 2:18
  SPEAKER_02: you know, you're not an AI hater or an AI lover.
• 2:20
  SPEAKER_02: You actually built a lot of cool stuff pre-AI, right?
• 2:23
  SPEAKER_02: Starting at, we just saw Linden Labs.
• 2:25
  SPEAKER_02: Was that your first job?
• 2:27
  SPEAKER_00: My first job at Linden Lab right across the street.
• 2:29
  SPEAKER_02: Right across the street.
• 2:30
  SPEAKER_02: We were just talking about that.
• 2:31
  SPEAKER_02: So you were building Second Life?
• 2:33
  SPEAKER_00: Yeah, we were building Second Life.
• 2:34
  SPEAKER_00: Yeah.
• 2:35
  SPEAKER_02: And then from there on,
• 2:36
  SPEAKER_02: one of the big hits was Parse,
• 2:38
  SPEAKER_02: the developer tool,
• 2:39
  SPEAKER_02: which was beloved by developers, best backend for mobile services I used to use it.
• 2:45
  SPEAKER_02: And then what happened?
• 2:46
  SPEAKER_02: Facebook bought you.
• 2:47
  SPEAKER_00: Facebook bought it.
• 2:48
  SPEAKER_00: Yeah, it was my first great lesson, and most acquisitions fail.
• 2:53
  SPEAKER_00: Most were terrible.
• 2:55
  SPEAKER_00: This one failed.
• 2:56
  SPEAKER_00: They shut it down.
• 2:57
  SPEAKER_00: But ultimately,
• 2:58
  SPEAKER_00: I'm very grateful to have had the experience because if it wasn't for that,
• 3:02
  SPEAKER_00: I've always been a startup kid.
• 3:04
  SPEAKER_00: And so nobody knew my name.
• 3:05
  SPEAKER_00: And it wasn't until I was leaving Facebook that investors were like,
• 3:08
  SPEAKER_00: oh, would you like some money?
• 3:09
  SPEAKER_00: And that's that's how we started Honeycomb.
• 3:11
  SPEAKER_02: And then you saw stuff at Facebook, right?
• 3:13
  SPEAKER_02: It inspired you.
• 3:14
  SPEAKER_00: Yeah.
• 3:15
  SPEAKER_00: Yeah.
• 3:15
  SPEAKER_00: Facebook.
• 3:16
  SPEAKER_00: There is a tool called Scuba.
• 3:19
  SPEAKER_00: And so we were in a weird position.
• 3:21
  SPEAKER_00: We were building on AWS, Ruby on Rails, all this stuff.
• 3:24
  SPEAKER_00: And then we got to use the internal Facebook tools.
• 3:27
  SPEAKER_00: And Facebook had this tool called Scuba.
• 3:29
  SPEAKER_00: And it was, we were experiencing hockey stick growth.
• 3:32
  SPEAKER_00: It was just like, we had over a million mobile apps hosted on Parse by the time I left.
• 3:38
  SPEAKER_00: And every single week, a new one would break.
• 3:40
  SPEAKER_00: It would hit the top 10 on iTunes or something out of nowhere.
• 3:43
  SPEAKER_00: It would just be like, oh, wait.
• 3:45
  SPEAKER_00: And these apps...
• 3:47
  SPEAKER_00: Needle in the haystack,
• 3:48
  SPEAKER_00: you know,
• 3:48
  SPEAKER_00: and it went from,
• 3:49
  SPEAKER_00: it would take hours or weeks,
• 3:51
  SPEAKER_00: we have to get lucky,
• 3:52
  SPEAKER_00: we finally find,
• 3:52
  SPEAKER_00: because it's not,
• 3:53
  SPEAKER_00: it might be one app that's spamming your logs, but that might not be the reason.
• 3:57
  SPEAKER_00: They might all be backed up behind the reason, you know.
• 4:00
  SPEAKER_00: We started getting our data sets into Scuba and finding them,
• 4:04
  SPEAKER_00: it just went from being a really hard engineering problem with a lot of luck
• 4:07
  SPEAKER_00: to just being like, report problem.
• 4:09
  SPEAKER_00: Click, click, click, oh, there it is.
• 4:11
  SPEAKER_00: And it was just mind blown, like,
• 4:13
  SPEAKER_00: you just want That was a huge problem for our entire existence,
• 4:17
  SPEAKER_00: and then it was solved.
• 4:18
  SPEAKER_02: And then when you started Honeycomb,
• 4:21
  SPEAKER_02: was this a bit of inspiration that you wanted to build something
• 4:23
  SPEAKER_02: that feels like
• 4:24
  SPEAKER_00: Scuba did?
• 4:27
  SPEAKER_00: I was planning to go be an engineering manager, an engineer at Slack or Stripe or something.
• 4:31
  SPEAKER_00: And I was just like, oof, I would be so much less powerful as an engineer without this.
• 4:40
  SPEAKER_00: And so, you know, the grand plan in the beginning, I'm just like, well, all startups fail.
• 4:44
  SPEAKER_00: So, you know, we'll fail, but...
• 4:46
  SPEAKER_00: I'll go sit in a corner and write Go code for a year or two,
• 4:49
  SPEAKER_00: and then I'll open source it and I can take it with me wherever I go.
• 4:54
  SPEAKER_00: That's how Honeycomb started.
• 4:56
  SPEAKER_02: And we'll get back to like observability or Honeycomb.
• 4:59
  SPEAKER_02: But before we do now with AI, you know, it's changing everything.
• 5:04
  SPEAKER_02: But I kind of had a bit of a blast from the past,
• 5:06
  SPEAKER_02: which is one of the first places we connected was in 2020.
• 5:09
  SPEAKER_02: So almost five years ago or so, someone submitted a question to both my blog and your blog.
• 5:15
  SPEAKER_02: And it was the question was like, can you measure individual developer productivity?
• 5:20
  SPEAKER_02: Now, I wrote an answer and you wrote an answer.
• 5:23
  SPEAKER_02: And I wanted to ask you, that was five years ago, no AI, no nothing.
• 5:27
  SPEAKER_02: Today, someone shoots you a question saying,
• 5:30
  SPEAKER_02: hey, Charity, can you measure one of an engineer's individual productivity?
• 5:34
  SPEAKER_02: You know, they're using AI tools and all this stuff.
• 5:37
  SPEAKER_02: What would you tell them?
• 5:38
  SPEAKER_00: I would tell them.
• 5:39
  SPEAKER_00: God, I don't even remember what I said.
• 5:40
  SPEAKER_00: I remember that blog post, but...
• 5:42
  SPEAKER_02: We both agreed, by the way, that it was...
• 5:45
  SPEAKER_02: That you can measure some dimensions,
• 5:47
  SPEAKER_02: and they're not going to give you the full thing,
• 5:49
  SPEAKER_02: and they will,
• 5:49
  SPEAKER_02: for example,
• 5:50
  SPEAKER_02: not tell you how a team is doing,
• 5:52
  SPEAKER_02: if someone is actually a really key part of the team,
• 5:56
  SPEAKER_02: and that as long as you measure individual things,
• 5:58
  SPEAKER_02: we both agreed that you need to be in the details to know.
• 6:02
  SPEAKER_02: And as a good manager or a good team lead, you will know.
• 6:06
  SPEAKER_00: You will know, but you have to have Gator to back it up.
• 6:09
  SPEAKER_00: It's like...
• 6:10
  SPEAKER_00: Color and a painting on the wall.
• 6:12
  SPEAKER_00: And is it Goodhart's Law?
• 6:15
  SPEAKER_00: Yes, it's Goodhart's Law.
• 6:17
  SPEAKER_00: So like never go well, it's this thing that matters, right?
• 6:21
  SPEAKER_00: You need to actually understand,
• 6:23
  SPEAKER_00: but you need it to not just be your opinion that was tossed off because you have
• 6:28
  SPEAKER_00: an opinion about some,
• 6:29
  SPEAKER_00: you know,
• 6:30
  SPEAKER_00: we're all,
• 6:31
  SPEAKER_00: we have biases, we are selective, you know, you need to look at the picture.
• 6:36
  SPEAKER_00: I also believe that,
• 6:37
  SPEAKER_00: you know,
• 6:37
  SPEAKER_00: there's been this whole push towards individual output,
• 6:40
  SPEAKER_00: but teams are still what matter.
• 6:44
  SPEAKER_00: And honestly,
• 6:44
  SPEAKER_00: if there's one thing that I am encouraged and excited about with the AI movement,
• 6:49
  SPEAKER_00: I think it's forcing us all to ask ourselves early and often,
• 6:55
  SPEAKER_00: what does good look like?
• 6:56
  SPEAKER_00: What does good mean?
• 6:58
  SPEAKER_00: What does productivity mean?
• 7:00
  SPEAKER_00: What would better look like?
• 7:01
  SPEAKER_00: What would great look like?
• 7:03
  SPEAKER_00: You know, and these questions are hard.
• 7:05
  SPEAKER_00: I think it's telling that we all jumped so fast to speed.
• 7:11
  SPEAKER_00: Oh, fast.
• 7:12
  SPEAKER_00: We can do it fast.
• 7:14
  SPEAKER_00: Same thing faster, you know.
• 7:15
  SPEAKER_00: Boom.
• 7:16
  SPEAKER_00: And I've come to feel like that is a very immature description of what better is.
• 7:24
  SPEAKER_02: Yeah, just today I saw the Antropic team posted a podcast with Spotify's head
• 7:32
  SPEAKER_02: of engineering or VP of engineering,
• 7:34
  SPEAKER_02: I'm not sure which one,
• 7:34
  SPEAKER_02: in which they talk that, wow,
• 7:36
  SPEAKER_02: Spotify with CloudCo,
• 7:37
  SPEAKER_02: they're shipping 4,500 changes per day,
• 7:39
  SPEAKER_02: per week, I'm not sure which one, but they talked about speed.
• 7:42
  SPEAKER_02: And I was kind of thinking like my experience has been different because I struggled
• 7:46
  SPEAKER_02: to publish any like some of my episodes did not go on Spotify because it was down.
• 7:50
  SPEAKER_02: And yeah, they're talking about speed, but we're not talking about quality.
• 7:54
  SPEAKER_02: We're not talking about more functionality,
• 7:57
  SPEAKER_02: better functionality or just things that people want.
• 8:01
  SPEAKER_02: And in the comments,
• 8:01
  SPEAKER_02: some people are asking like,
• 8:02
  SPEAKER_02: OK, so what exactly does that mean that they're shipping more frequently?
• 8:06
  SPEAKER_00: Yeah.
• 8:07
  SPEAKER_00: Do customers really want the buttons on their app to move around all the time?
• 8:11
  SPEAKER_00: I don't think they do.
• 8:13
  SPEAKER_02: Yeah, it's an interesting one.
• 8:14
  SPEAKER_00: It's the easiest thing to measure.
• 8:16
  SPEAKER_02: Let's jump back to last year in 2025.
• 8:19
  SPEAKER_02: You wrote a blog post right at the end of the year looking back saying that 2025
• 8:25
  SPEAKER_02: for AI was what 2010 was for the cloud.
• 8:28
  SPEAKER_02: Can we talk about that before we go into like this year,
• 8:31
  SPEAKER_02: but like last year, like how was your perspective?
• 8:34
  SPEAKER_02: Of course, you were working at observability company.
• 8:36
  SPEAKER_02: AI will give you lots of like business as well, but you said it went mainstream, right?
• 8:41
  SPEAKER_02: Last year.
• 8:42
  SPEAKER_02: Yeah.
• 8:43
  SPEAKER_00: In March of 2025, Fred Hebert and I gave a keynote at SRECon.
• 8:47
  SPEAKER_00: We gave a closing talk.
• 8:49
  SPEAKER_00: And it's Fred and I standing in front of a...
• 8:51
  SPEAKER_00: The term vibe coding had just been invented.
• 8:54
  SPEAKER_02: Oh, yes.
• 8:55
  SPEAKER_00: And we were like, you guys should try vibe coding.
• 8:58
  SPEAKER_00: Pause.
• 8:59
  SPEAKER_00: Groans.
• 9:00
  SPEAKER_00: Audible groans.
• 9:00
  SPEAKER_00: Just like people laughing like, ha, ha, ha.
• 9:03
  SPEAKER_00: Our big pitch was that people should learn AI because you can complain better if you learn it.
• 9:07
  SPEAKER_00: Which is legit.
• 9:09
  SPEAKER_00: I mean, I really mean it.
• 9:11
  SPEAKER_00: But at the time...
• 9:14
  SPEAKER_00: I think I still saw it as a really big feature,
• 9:17
  SPEAKER_00: or like bigger than a programming language,
• 9:19
  SPEAKER_00: like the cloud, but not like generational, you know, not changing everything.
• 9:25
  SPEAKER_00: And I think that was accurate.
• 9:28
  SPEAKER_00: For me, it was November of 2025 when they released Opus 4.5,
• 9:33
  SPEAKER_00: but I actually wrote about this recently,
• 9:35
  SPEAKER_00: a couple blog posts back, about how in retrospect...
• 9:39
  SPEAKER_00: You could see it coming sooner.
• 9:41
  SPEAKER_00: You could see, and it wasn't actually the models.
• 9:44
  SPEAKER_00: It was the harnesses.
• 9:46
  SPEAKER_00: It was all the tooling.
• 9:47
  SPEAKER_00: And it was people starting to say that around July.
• 9:51
  SPEAKER_00: They were like, this is coming faster than you think, and this is what it's going to look like.
• 9:54
  SPEAKER_02: And it was people who were saying it,
• 9:55
  SPEAKER_02: the robots who were playing with either a cloth coat or maybe pie or open coat.
• 10:00
  SPEAKER_02: So the harnesses, you're right.
• 10:01
  SPEAKER_00: They were getting better at the tooling.
• 10:03
  SPEAKER_00: You know,
• 10:03
  SPEAKER_00: it went from just being kind of a shell script that would try again to like,
• 10:06
  SPEAKER_00: they built a lot of stuff around it.
• 10:09
  SPEAKER_00: And then, you know, the opus thing kind of, it was a weird time at the beginning.
• 10:13
  SPEAKER_00: It's been a weird time every time for a long time.
• 10:15
  SPEAKER_00: But the early months of this year,
• 10:17
  SPEAKER_00: it felt like everyone around me was just trying it again and changing their mind.
• 10:22
  SPEAKER_00: Everyone.
• 10:23
  SPEAKER_02: Yeah.
• 10:24
  SPEAKER_02: I think we were just talking about right before we started recording that both you
• 10:28
  SPEAKER_02: and me respect people who do change their minds.
• 10:31
  SPEAKER_00: And I don't think we were wrong to be skeptical that the first time.
• 10:35
  SPEAKER_00: It's a pretty extraordinary claim that the AI is going to write code about as well
• 10:40
  SPEAKER_00: as the median software engineer can, you know, for limited bounds of that.
• 10:45
  SPEAKER_02: Well, especially because if we look back at the history of software engineering,
• 10:48
  SPEAKER_02: this claim has happened again and again.
• 10:49
  SPEAKER_02: You know, neural nets should have been doing something magical.
• 10:53
  SPEAKER_00: There's a sticker in your pack that says we already have a programming language that...
• 10:57
  SPEAKER_00: It lets you, COBOL is the punchline.
• 11:00
  SPEAKER_00: So I don't think we were wrong to be skeptical.
• 11:02
  SPEAKER_02: And also, don't forget no code and low code.
• 11:05
  SPEAKER_00: Oh, yeah.
• 11:05
  SPEAKER_02: I mean, we know it turned out to be a joke, but the promise was the same.
• 11:09
  SPEAKER_02: And we were skeptical, and we were right.
• 11:11
  SPEAKER_02: And now we're skeptical again, and we were wrong.
• 11:15
  SPEAKER_00: What I was saying in that piece, though, was I think we were right to be skeptical that time.
• 11:18
  SPEAKER_00: But now I see the same thing playing out with,
• 11:24
  SPEAKER_00: would you be willing to ship it some code that you didn't read?
• 11:27
  SPEAKER_00: Yeah.
• 11:27
  SPEAKER_00: Yeah.
• 11:27
  SPEAKER_00: Yeah.
• 11:28
  SPEAKER_00: Yeah.
• 11:49
  SPEAKER_02: You're right.
• 11:50
  SPEAKER_02: If I could have a way to,
• 11:52
  SPEAKER_02: for example,
• 11:53
  SPEAKER_02: I could see the change, I could tell that this was tested in a harness or something.
• 12:01
  SPEAKER_02: Same way where,
• 12:02
  SPEAKER_02: for example, pre-AI,
• 12:03
  SPEAKER_02: if there was a team member who said,
• 12:06
  SPEAKER_02: I vouch for this and I've hammered it and I trust that person.
• 12:09
  SPEAKER_02: So you're right.
• 12:10
  SPEAKER_02: There's these things which are Of course,
• 12:14
  SPEAKER_02: never, but I've thought that AI or something can do anything like that.
• 12:17
  SPEAKER_02: But if it could, you're not entering, right?
• 12:19
  SPEAKER_00: Or, for example,
• 12:21
  SPEAKER_00: if you and the AI would both do it in tandem for a few months and you would be like,
• 12:26
  SPEAKER_00: you would get to, how much are they catching?
• 12:28
  SPEAKER_00: How much am I catching?
• 12:29
  SPEAKER_00: Is it about the same?
• 12:29
  SPEAKER_00: Is it more?
• 12:30
  SPEAKER_00: Is it less?
• 12:30
  SPEAKER_00: And you're training it and it's getting better.
• 12:32
  SPEAKER_00: Whether it takes five days or five years or whatever,
• 12:35
  SPEAKER_00: I think it's pretty clear that directionally that's where we're going.
• 12:38
  SPEAKER_00: And the other thing that I would say is, this is good for us.
• 12:41
  SPEAKER_00: Have you spent much time with the Phoenix architecture stuff
• 12:44
  SPEAKER_00: that Chad Fowler has been writing about?
• 12:46
  SPEAKER_02: You have been quoting.
• 12:47
  SPEAKER_00: I've been quoting liberally.
• 12:49
  SPEAKER_00: I should probably let you get to it in your own order.
• 12:51
  SPEAKER_00: But I just feel like anyone who's ever done a painful rewrite should be on board with us.
• 12:58
  SPEAKER_02: Yeah, but here's a quote from Jeff Fowler.
• 13:01
  SPEAKER_02: Immutable infrastructure,
• 13:02
  SPEAKER_02: stateless services, containers, blue-green deployments, infrastructure as a code.
• 13:07
  SPEAKER_02: These ideas all share a common premise.
• 13:09
  SPEAKER_02: Never fix a running thing.
• 13:10
  SPEAKER_02: Replace it.
• 13:11
  SPEAKER_02: AI pushes this premise beyond infrastructure and into application code itself.
• 13:16
  SPEAKER_02: When rewriting is cheap, editing in place becomes risky.
• 13:19
  SPEAKER_02: Mutation accumulates entropy.
• 13:21
  SPEAKER_02: Replacements resets it.
• 13:23
  SPEAKER_00: Yes, code is cash.
• 13:24
  SPEAKER_02: This is a very interesting idea because...
• 13:27
  SPEAKER_02: You've compared,
• 13:29
  SPEAKER_02: Chaff compared,
• 13:30
  SPEAKER_02: and you've also,
• 13:31
  SPEAKER_02: of course,
• 13:32
  SPEAKER_02: shared this,
• 13:33
  SPEAKER_02: that when we look at how infrastructure changed before,
• 13:37
  SPEAKER_02: you know, like specifically a server, you need to configure it.
• 13:40
  SPEAKER_02: I think we call it like pet.
• 13:41
  SPEAKER_02: Pets versus server.
• 13:43
  SPEAKER_02: Having pets versus.
• 13:44
  SPEAKER_01: Yeah.
• 13:45
  SPEAKER_02: And at some point we stopped configuring individually.
• 13:48
  SPEAKER_02: We stopped like fixing individual machines.
• 13:50
  SPEAKER_02: We just like throw it away and have a new thing.
• 13:53
  SPEAKER_02: Yep.
• 13:54
  SPEAKER_02: And with code,
• 13:55
  SPEAKER_02: we've always been used to the history of the profession,
• 13:58
  SPEAKER_02: you know, 60 plus years or maybe a bit longer, is that we edit code.
• 14:03
  SPEAKER_02: And are you thinking this might...
• 14:06
  SPEAKER_00: Because of the economics of it.
• 14:07
  SPEAKER_00: I mean, if you think about it,
• 14:09
  SPEAKER_00: you could generate 10,000 variants
• 14:12
  SPEAKER_02: of a function faster
• 14:15
  SPEAKER_02: than you could write it once.
• 14:17
  SPEAKER_00: And so when you start thinking about it that way,
• 14:19
  SPEAKER_00: it's like, well, okay, we're going to need a lot of evals.
• 14:21
  SPEAKER_00: We're going to need a lot of tests.
• 14:24
  SPEAKER_00: But the generation is so cheap that it really, I think it forces us in that direction.
• 14:31
  SPEAKER_00: And I think that the expensiveness of writing code and maintaining code and
• 14:37
  SPEAKER_00: the expense of software has always been bound up in its maintenance.
• 14:41
  SPEAKER_00: And those lines of code...
• 14:44
  SPEAKER_00: The reason that we trust something is because we've been using it.
• 14:51
  SPEAKER_00: There's this deep thing about production.
• 14:55
  SPEAKER_00: It's like, well, it's trusted.
• 14:56
  SPEAKER_00: We know.
• 14:57
  SPEAKER_00: And I've been, I know that as well as anyone.
• 15:02
  SPEAKER_00: And I will also say this,
• 15:03
  SPEAKER_00: anyone who's ever done a hard database migration should have some real humility
• 15:07
  SPEAKER_00: about our ability to extrapolate those contracts, store them.
• 15:11
  SPEAKER_00: Like, I am not one of those people who's like, this is good, we're going to generate all code.
• 15:15
  SPEAKER_00: I don't know how much code.
• 15:17
  SPEAKER_00: I believe that we can go some distance in that direction and it will be good for us.
• 15:21
  SPEAKER_00: I don't know how far we can go.
• 15:23
  SPEAKER_00: I believe we can go farther than we are now.
• 15:25
  SPEAKER_00: I just, man,
• 15:27
  SPEAKER_00: the last project I did at Parse,
• 15:30
  SPEAKER_00: So we had spent like six months writing the original Ruby on Rails API.
• 15:34
  SPEAKER_02: Yep.
• 15:35
  SPEAKER_00: Spent two years rewriting it in Golang.
• 15:37
  SPEAKER_00: Wow.
• 15:38
  SPEAKER_00: Yeah, it was...
• 15:40
  SPEAKER_02: And was it two years because new stuff kept being added?
• 15:43
  SPEAKER_00: To some extent.
• 15:45
  SPEAKER_00: And also Golang was a pretty immature language at the time.
• 15:48
  SPEAKER_00: We had to write, you know, the MongoDB drivers and like all the other...
• 15:51
  SPEAKER_00: A bunch of things.
• 15:52
  SPEAKER_00: And also just like when you're writing in Ruby and MongoDB and JavaScript
• 15:56
  SPEAKER_00: and everything is, you know, there's no type safety and it's just painful.
• 16:01
  SPEAKER_00: Just, you know,
• 16:01
  SPEAKER_00: and the strangler figs that they do where you build the architecture outside
• 16:06
  SPEAKER_00: the architecture and you literally find the contracts with your users
• 16:10
  SPEAKER_00: by breaking them.
• 16:12
  SPEAKER_00: One after the other.
• 16:14
  SPEAKER_00: Like that just does not seem like the ideal artifact.
• 16:16
  SPEAKER_00: We should be able to store them somewhere.
• 16:18
  SPEAKER_00: We should be able to have architecture diagrams that we can review.
• 16:22
  SPEAKER_00: And discuss that generate that code to spec.
• 16:26
  SPEAKER_02: This is very interesting because some of these ideas, they've been around decades ago.
• 16:30
  SPEAKER_02: Specifically,
• 16:32
  SPEAKER_02: you know,
• 16:32
  SPEAKER_02: if we had Grady Booch as a third person sitting here,
• 16:35
  SPEAKER_02: the idea of like, hey, we can have architecture diagrams that translate to code.
• 16:40
  SPEAKER_02: UML...
• 16:42
  SPEAKER_02: That's my hope.
• 17:06
  SPEAKER_00: I mean, I'm just barely old enough that my first job was like 17 at university.
• 17:12
  SPEAKER_00: I was a sysadmin.
• 17:14
  SPEAKER_00: I remember when, you know, I wasn't really aware of what was going on.
• 17:18
  SPEAKER_00: I was just a kid.
• 17:19
  SPEAKER_00: But yeah,
• 17:19
  SPEAKER_00: I remember how stressful it was and how people were agonizing
• 17:23
  SPEAKER_00: about how we'll never be able to get that information back.
• 17:26
  SPEAKER_00: And everyone adapted just fine.
• 17:30
  SPEAKER_00: I think I read the systems that, you know, they built the systems that replaced them.
• 17:34
  SPEAKER_00: But Not as in replace them and work them out of a job.
• 17:37
  SPEAKER_00: They built the systems and they spent their time writing code instead
• 17:41
  SPEAKER_00: of like running updates by hand on every server in the closet.
• 17:46
  SPEAKER_02: And I guess this is an interesting one because clearly like the sysadmin role
• 17:49
  SPEAKER_02: and profession has been, it doesn't exist today.
• 17:52
  SPEAKER_02: It's kind of, let's just say it has been eliminated.
• 17:55
  SPEAKER_02: However, the people who are sysadmins, they did understand the operating systems.
• 17:59
  SPEAKER_02: They understood hardware.
• 18:00
  SPEAKER_02: Yes.
• 18:01
  SPEAKER_02: They were in a really good position to adopt.
• 18:03
  SPEAKER_02: And a lot of them just became either software engineers, product managers.
• 18:07
  SPEAKER_02: I know someone who became a tech salesperson.
• 18:09
  SPEAKER_02: Yeah, yeah.
• 18:12
  SPEAKER_00: And I will hold that our generation of engineers are still the best debuggers.
• 18:17
  SPEAKER_00: I'm glad that people don't all have to learn about CPU and memory
• 18:21
  SPEAKER_00: and all this stuff, but there's value in knowing that stuff.
• 18:25
  SPEAKER_00: It comes in handy.
• 18:26
  SPEAKER_01: I think there's some analogies
• 18:27
  SPEAKER_00: there to the generation of code stuff.
• 18:30
  SPEAKER_02: Also, you took a bunch of inspiration in your recent writing about both sysadmins
• 18:35
  SPEAKER_02: but also QA, and you wrote something interesting.
• 18:37
  SPEAKER_02: You said, lines of code are not the ideal artifact to review.
• 18:41
  SPEAKER_02: And I'll quote a little bit from you.
• 18:42
  SPEAKER_02: The tools to do this don't exist yet, but many of the ideas do exist.
• 18:46
  SPEAKER_02: Most come from operations in QA,
• 18:48
  SPEAKER_02: two domains that software engineering has historically been rather snobbish about.
• 18:52
  SPEAKER_02: Shall we revisit our relationship to QA and Ops?
• 18:57
  SPEAKER_02: Where I feel we always put ourselves as software engineers here and ops and QA somewhere.
• 19:05
  SPEAKER_02: And maybe time to eat some humble pie.
• 19:07
  SPEAKER_00: Ops equals toil.
• 19:10
  SPEAKER_00: Right?
• 19:11
  SPEAKER_00: Yeah.
• 19:12
  SPEAKER_00: I think it's time.
• 19:13
  SPEAKER_00: I mean, ops and QA have always been more concerned with what is.
• 19:18
  SPEAKER_00: Software engineering has always been much more concerned with how should it be.
• 19:21
  SPEAKER_02: Mm-hmm.
• 19:22
  SPEAKER_02: So Ops and QA have always been more concerned about validating,
• 19:27
  SPEAKER_02: about correctness, about does it work as expected?
• 19:30
  SPEAKER_02: Does it work?
• 19:31
  SPEAKER_02: Does it work to start with?
• 19:32
  SPEAKER_00: Does it work?
• 19:33
  SPEAKER_00: Yeah.
• 19:33
  SPEAKER_00: Yeah.
• 19:34
  SPEAKER_00: I mean,
• 19:35
  SPEAKER_00: it's always weird to me just how much software engineers really seem to believe that
• 19:38
  SPEAKER_00: the world exists in the repo.
• 19:41
  SPEAKER_00: It doesn't.
• 19:43
  SPEAKER_00: It's production, you know?
• 19:44
  SPEAKER_02: The code has
• 19:46
  SPEAKER_00: part of the information.
• 19:48
  SPEAKER_00: Some of it, it's very necessary.
• 19:49
  SPEAKER_00: We need that.
• 19:50
  SPEAKER_00: But like,
• 19:51
  SPEAKER_00: I know
• 19:51
  SPEAKER_00: some software engineers who,
• 19:53
  SPEAKER_00: and okay, some places don't even let software engineers look at production.
• 19:58
  SPEAKER_00: Just like, how?
• 19:59
  SPEAKER_00: I know a lot of people are very upset about AI,
• 20:01
  SPEAKER_00: but the things that get me very excited,
• 20:04
  SPEAKER_00: genuinely excited about AI are that it is pushing the discipline
• 20:08
  SPEAKER_00: in directions we have desperately needed to go for a very long time.
• 20:13
  SPEAKER_00: Production is not what happens after development.
• 20:16
  SPEAKER_00: It is a stage of development.
• 20:18
  SPEAKER_02: And you've been saying this consistently for pre-AI,
• 20:21
  SPEAKER_02: I'm just going to say for those who don't,
• 20:22
  SPEAKER_02: because I remember we've,
• 20:24
  SPEAKER_02: I think we also bonded a little bit over,
• 20:26
  SPEAKER_02: there was this thing called trending on Twitter when it was still Twitter
• 20:29
  SPEAKER_02: and it was tech Twitter.
• 20:31
  SPEAKER_02: Everyone was there who mattered.
• 20:32
  SPEAKER_02: And there was a trend going, it's Friday, don't deploy.
• 20:37
  SPEAKER_02: Something that there was maybe a hashtag even like,
• 20:39
  SPEAKER_02: I'm not sure, don't deploy Friday or something like that.
• 20:42
  SPEAKER_02: And the point was, it was well-meaning.
• 20:44
  SPEAKER_02: It said like, look,
• 20:45
  SPEAKER_02: when you deploy off and there's an outage,
• 20:47
  SPEAKER_02: And on the weekend, we don't want to do so.
• 20:49
  SPEAKER_02: They were saying every Friday, it went viral saying don't deploy on Fridays.
• 20:52
  SPEAKER_02: And you came in and you said, you know what?
• 20:55
  SPEAKER_02: You should be able to deploy anytime without fear because you should be able
• 21:00
  SPEAKER_02: to just know, you know, however that might be, CI, CD.
• 21:04
  SPEAKER_02: And then on top of this,
• 21:05
  SPEAKER_02: you were like, no,
• 21:06
  SPEAKER_02: like you should actually just not even have a user acceptance testing environment
• 21:09
  SPEAKER_02: at UAT.
• 21:10
  SPEAKER_02: You should just deploy its production, like intestine production, right?
• 21:13
  SPEAKER_00: As soon as you merge, it should be going out.
• 21:16
  SPEAKER_00: You should have to stop the train to make your code and not go into production
• 21:20
  SPEAKER_00: as soon as you've merged.
• 21:21
  SPEAKER_00: Absolutely.
• 21:22
  SPEAKER_02: And one more interesting thing is you had a long train of thought about AI and what it could be.
• 21:28
  SPEAKER_02: One thing you said is our brains are not built for validation.
• 21:31
  SPEAKER_02: Almost everyone I talked to,
• 21:32
  SPEAKER_02: including Andres Heisberg,
• 21:35
  SPEAKER_02: he said that, look, it's very clear that code generation is cheap.
• 21:39
  SPEAKER_02: We are generating more code.
• 21:41
  SPEAKER_02: And the bottleneck for human engineers is for code review.
• 21:45
  SPEAKER_02: And everyone's trying to figure out how do we make code review easier?
• 21:48
  SPEAKER_02: How do we build nicer tools?
• 21:50
  SPEAKER_02: Uber has built amazing tools to try to surface important code reviews.
• 21:54
  SPEAKER_02: But everyone's pushing, like, all right, let's do more code review.
• 21:58
  SPEAKER_02: As an engineer, I'll be honest, like I never liked doing a code review.
• 22:02
  SPEAKER_02: When there's very little to do and it's with someone I care about, I'll entertain it.
• 22:06
  SPEAKER_00: It's more of a coaching opportunity then, right?
• 22:09
  SPEAKER_02: But as soon as there's an AI, it's kind of like, I don't know.
• 22:12
  SPEAKER_02: I don't really care.
• 22:13
  SPEAKER_02: I'm just being honest here.
• 22:15
  SPEAKER_00: Do you care?
• 22:16
  SPEAKER_00: I don't.
• 22:17
  SPEAKER_00: I've never.
• 22:18
  SPEAKER_00: So one of the problems is that I think code review means so many things to
• 22:22
  SPEAKER_00: so many people in so many places.
• 22:24
  SPEAKER_00: And so there's a lot of projection going on.
• 22:26
  SPEAKER_00: A lot of people are,
• 22:27
  SPEAKER_00: if you say that you don't want code review,
• 22:29
  SPEAKER_00: you're saying you don't want to talk to your coworkers,
• 22:31
  SPEAKER_00: you don't want to mentor juniors, you don't want to.
• 22:34
  SPEAKER_00: Which is not true.
• 22:35
  SPEAKER_00: We've just bundled so many things into this, like, you know, it's like...
• 22:39
  SPEAKER_02: Hugely overloaded.
• 22:40
  SPEAKER_00: Hugely overloaded.
• 22:41
  SPEAKER_00: And some of those things are really good.
• 22:43
  SPEAKER_00: Some of those things could be done better in other ways, you know?
• 22:47
  SPEAKER_00: Some of those things are very cultural, very specific.
• 22:50
  SPEAKER_00: My friend David Pohl,
• 22:52
  SPEAKER_00: who I worked with at Parse, and he's now working at GitHub on pull requests.
• 22:57
  SPEAKER_02: Amazing.
• 22:58
  SPEAKER_02: Love the Parse Mafia.
• 22:59
  SPEAKER_00: Yeah, exactly.
• 23:00
  SPEAKER_00: Exactly.
• 23:02
  SPEAKER_00: He's like, to me, the code review is when we decide, do we want this in our product or not?
• 23:08
  SPEAKER_00: I'm like, well, that is a, that's a great, great discussion.
• 23:11
  SPEAKER_00: That is what humans are good at.
• 23:13
  SPEAKER_00: We should talk about, is this mental model coherent?
• 23:16
  SPEAKER_00: Should we add this?
• 23:17
  SPEAKER_00: Should we not?
• 23:18
  SPEAKER_00: Like, love that.
• 23:20
  SPEAKER_00: Architectures,
• 23:20
  SPEAKER_00: you know, but like, the code is not necessarily a great artifact for all of those.
• 23:26
  SPEAKER_00: So should we be talking to people?
• 23:28
  SPEAKER_00: Uh, yes.
• 23:30
  SPEAKER_00: Is the code review the right form factor?
• 23:34
  SPEAKER_00: Maybe.
• 23:34
  SPEAKER_00: But I think that the emotional reaction that's when people are getting to it,
• 23:38
  SPEAKER_00: like the validation in my book is at the very bottom of the list.
• 23:42
  SPEAKER_02: I'd like to like touch, like stay here a bit more.
• 23:45
  SPEAKER_02: Can you break out the parts?
• 23:46
  SPEAKER_02: Because it feels to me code reviews are overloaded,
• 23:48
  SPEAKER_02: but the parts of code review or the things that you have seen are good things
• 23:53
  SPEAKER_02: and maybe we don't need to do as code review and the things that are just like,
• 23:56
  SPEAKER_02: just have never been that good and maybe we just need to throw it away.
• 23:59
  SPEAKER_00: Yeah, I mean, I think doing what this inner product is, that is great.
• 24:03
  SPEAKER_00: I mean, ideally you'd talk about that before you write the code for it, but you know, whatever.
• 24:08
  SPEAKER_00: And, you know, is this API design?
• 24:11
  SPEAKER_00: You know, those are great conversations.
• 24:13
  SPEAKER_00: Reading for...
• 24:15
  SPEAKER_00: Syntax and bugs and that sort of thing.
• 24:18
  SPEAKER_00: It's not evil, but it feels like it could be.
• 24:20
  SPEAKER_00: It's a teaching opportunity if that's the best teaching opportunity you have.
• 24:24
  SPEAKER_00: And I guess some folks at some point maybe need them, but it doesn't feel high.
• 24:27
  SPEAKER_02: It's like a great use of It feels the only time where it's useful is
• 24:32
  SPEAKER_02: if someone joins a team and initially it can be a little bit of feedback,
• 24:37
  SPEAKER_02: especially when there's like nothing is written down,
• 24:39
  SPEAKER_02: there's no guidance, there's no linting rules that would give you that.
• 24:41
  SPEAKER_00: Well, see, that's, again, yes, we can fill in the cracks if we haven't built the guardrails.
• 24:48
  SPEAKER_00: We can fill in the cracks all kinds of ways with it.
• 24:50
  SPEAKER_00: With our own time.
• 24:51
  SPEAKER_00: But there are so many things I think that we never think to extract out of
• 24:57
  SPEAKER_00: the process of building and validating software.
• 25:00
  SPEAKER_00: We rely on us.
• 25:02
  SPEAKER_00: So I am a huge fan of Intercom, now FIN, their engineering org.
• 25:07
  SPEAKER_00: And I have been forever.
• 25:09
  SPEAKER_00: Like I noticed their CTO a decade ago had this saying, shipping is your company's heartbeat.
• 25:17
  SPEAKER_00: They ship a Ruby monolith like 10, 15 minutes, hundreds of times a day.
• 25:24
  SPEAKER_00: That is not trivial.
• 25:25
  SPEAKER_00: It's not a trivial thing to do, right?
• 25:28
  SPEAKER_00: So they're kind of the high watermark from my mind right now for teams
• 25:33
  SPEAKER_00: that were founded pre-AI,
• 25:35
  SPEAKER_00: have a lot of engineering discipline, who have become AI native.
• 25:39
  SPEAKER_00: And they wrote a great post about how they do PRs that are You've also written
• 25:45
  SPEAKER_00: that non-deterministic systems require more engineering discipline.
• 26:10
  SPEAKER_01: Not less.
• 26:10
  SPEAKER_02: So like, what is the thing about these non-deterministic, specific AI, right?
• 26:15
  SPEAKER_02: We're talking about AI, let's just name it.
• 26:17
  SPEAKER_02: That is, we see that AI does amplify Both discipline and lack of discipline.
• 26:23
  SPEAKER_02: Why do we need more?
• 26:25
  SPEAKER_02: And when you say discipline, what specifics are we talking about?
• 26:27
  SPEAKER_00: Well, I mean, tests and evals for one thing, right?
• 26:31
  SPEAKER_00: Like if we're treating the code like a trusted artifact and we're,
• 26:37
  SPEAKER_00: you know,
• 26:38
  SPEAKER_00: trying to predict everything with our human brains and everything,
• 26:41
  SPEAKER_00: then we're writing the test that we can predict that it might break,
• 26:44
  SPEAKER_00: you know, and then anytime the system breaks, we like try and write a test for that.
• 26:48
  SPEAKER_00: But It's not an especially high bar.
• 26:51
  SPEAKER_00: And so I think the sort of the behavioral tests or the,
• 26:56
  SPEAKER_00: I don't remember where it starts with C,
• 26:59
  SPEAKER_00: but the QA folks have these suite of tests where it captures.
• 27:03
  SPEAKER_02: There's also smoke tests.
• 27:04
  SPEAKER_02: Yeah, there's so many.
• 27:06
  SPEAKER_02: There can be like performance tests.
• 27:07
  SPEAKER_02: There can be low tests.
• 27:09
  SPEAKER_02: There can be, yeah, there can be like just kind of fuss testing as well.
• 27:13
  SPEAKER_00: Something that's like, okay,
• 27:14
  SPEAKER_00: if I'm not going to read this code,
• 27:16
  SPEAKER_00: how do I know it's going to I think we're going to need the trust test
• 27:21
  SPEAKER_00: to go somewhere.
• 27:36
  SPEAKER_00: Right?
• 27:37
  SPEAKER_00: If you're debiting from this trust account and the creation of the code,
• 27:40
  SPEAKER_00: it has to get built up somewhere else.
• 27:43
  SPEAKER_00: And I feel like one of the things that I'm really excited about in
• 27:45
  SPEAKER_00: the coming months is just,
• 27:46
  SPEAKER_00: I actually really like thinking about it less as AI and more as deterministic
• 27:51
  SPEAKER_00: and non-deterministic systems that have to play nicely together
• 27:54
  SPEAKER_00: because determinism is not going anywhere.
• 27:56
  SPEAKER_00: It's incredibly valuable.
• 27:58
  SPEAKER_00: And we have to Learn to make AI kind of boring, you know?
• 28:03
  SPEAKER_00: It's a non-deterministic tool,
• 28:05
  SPEAKER_00: which means that it is all over the place,
• 28:07
  SPEAKER_00: but it's so valuable, but it's all over the place.
• 28:10
  SPEAKER_00: We have to learn how to give it carved pathways and places where we kind
• 28:15
  SPEAKER_00: of corral it,
• 28:15
  SPEAKER_00: where we use it in the way that it's a superpower and not in the way
• 28:20
  SPEAKER_00: that like erodes our foundations.
• 28:23
  SPEAKER_02: This is interesting because Martin Fowler,
• 28:24
  SPEAKER_02: a year ago when he was on the podcast,
• 28:26
  SPEAKER_02: the thing that he talked about is how the biggest change with AI is
• 28:29
  SPEAKER_02: the non-determinism.
• 28:31
  SPEAKER_02: And when we think back in the history of software, it's always been deterministic.
• 28:35
  SPEAKER_02: Same for neural nets,
• 28:36
  SPEAKER_02: but that was most of us software engineers didn't really touch too much of it
• 28:39
  SPEAKER_02: because it just wasn't that useful for us.
• 28:41
  SPEAKER_02: But we've been used to that when we programmed.
• 28:45
  SPEAKER_02: It just happened the same way.
• 28:46
  SPEAKER_02: Unit tests were easy because you just run them once.
• 28:48
  SPEAKER_02: You don't run them twice because why would you?
• 28:51
  SPEAKER_02: And I wonder if this is,
• 28:52
  SPEAKER_02: we need to just realize how big of a deal this change is,
• 28:57
  SPEAKER_02: and that any business that employs us, they want software that works the same way.
• 29:04
  SPEAKER_02: We just had a...
• 29:06
  SPEAKER_02: Recent posts on Hacker News,
• 29:08
  SPEAKER_02: there's this ATS application tracking system scoring system
• 29:11
  SPEAKER_02: that HackerRank outsourced, which scores your resume.
• 29:15
  SPEAKER_02: And so software engineers just like, and you can run it locally, it's open source.
• 29:18
  SPEAKER_02: You can use a local model.
• 29:20
  SPEAKER_02: I think they recommend Gemma, Google's small model.
• 29:23
  SPEAKER_02: And when you run it like 100 times,
• 29:26
  SPEAKER_02: it will score the same resume anywhere from like 66 points to 99 points.
• 29:30
  SPEAKER_02: And typically most companies have 85 set as the bar.
• 29:33
  SPEAKER_02: And you're like, hang on.
• 29:34
  SPEAKER_02: So we've turned what they were advertising as a tool to help your recruitment.
• 29:40
  SPEAKER_02: We just prove that it's just a coin flip.
• 29:42
  SPEAKER_02: That's bad.
• 29:43
  SPEAKER_00: Yeah.
• 29:43
  SPEAKER_00: And we have to be able to say that it's bad.
• 29:46
  SPEAKER_00: AI is not the right tool for every use case, you know.
• 29:50
  SPEAKER_00: And I think every company is going through this in microcosm.
• 29:53
  SPEAKER_00: And something I was saying to folks just earlier today, we are...
• 29:57
  SPEAKER_00: We've been doing a series of conversations on our AI norms and values.
• 30:01
  SPEAKER_00: And it was like a year ago, I don't trust us.
• 30:04
  SPEAKER_00: Like a year ago, if we were like, yes, we should use AI, no, we didn't know enough.
• 30:08
  SPEAKER_00: We've gone on such a journey over the past year and we know so much more now.
• 30:13
  SPEAKER_00: If one of my coworkers is like, AI is the wrong tool for this job, I'm like, I trust you.
• 30:19
  SPEAKER_00: You know, you got to get worse before you can get better.
• 30:22
  SPEAKER_02: So tell me about where you are right now with your,
• 30:26
  SPEAKER_02: how inside of Honeycomb you're thinking about AI,
• 30:29
  SPEAKER_02: how you're thinking about how to think about AI,
• 30:31
  SPEAKER_02: and what values you came up with that works right now for you.
• 30:34
  SPEAKER_00: Yeah, it starts with just acknowledging that the bar has gone up for all of us.
• 30:38
  SPEAKER_00: That's what happens when we get powerful new tools.
• 30:42
  SPEAKER_02: Has the bar gone up or has the, you know, the floor gone up?
• 30:47
  SPEAKER_00: That is a great question.
• 30:48
  SPEAKER_00: Maybe yes.
• 30:49
  SPEAKER_00: Maybe, yeah.
• 30:50
  SPEAKER_00: I don't know.
• 30:51
  SPEAKER_00: We're definitely in a sort of wandering in the wilderness phase.
• 30:56
  SPEAKER_00: But you can't not wander or you will be left behind.
• 30:58
  SPEAKER_00: You know,
• 30:59
  SPEAKER_00: we acknowledge that the bar is going up for all of this and that the only viable way
• 31:02
  SPEAKER_00: to define that bar is better outcomes.
• 31:06
  SPEAKER_00: And asking ourselves, like, is this good?
• 31:09
  SPEAKER_00: Is this better?
• 31:09
  SPEAKER_00: What does good look like?
• 31:11
  SPEAKER_00: Another thing that we point out is just there is no human in the loop.
• 31:20
  SPEAKER_00: Charity
• 31:28
  SPEAKER_01: just talks about owning the loop.
• 31:30
  SPEAKER_01: Owning the loop also means controlling what every agent inside of
• 31:33
  SPEAKER_01: that loop is allowed to do, which brings us to our season sponsor, WorkOS.
• 31:37
  SPEAKER_01: Today, agents are increasingly able to act on their own,
• 31:40
  SPEAKER_01: and the old auth model was never designed for that.
• 31:43
  SPEAKER_01: Who is this agent?
• 31:44
  SPEAKER_01: What's it allowed to touch?
• 31:46
  SPEAKER_01: On whose behalf?
• 31:47
  SPEAKER_01: You really don't want to get answers to these questions wrong.
• 31:50
  SPEAKER_01: WorkOS is built exactly to solve this problem.
• 31:52
  SPEAKER_01: WorkOS is fine-grained authorization, FGA,
• 31:55
  SPEAKER_01: designed for how agents actually operate,
• 31:57
  SPEAKER_01: plus SSO and SCIM, and not just user auth with agents bolted on after.
• 32:02
  SPEAKER_01: The fastest-growing AI companies, Entropic, OpenAI, Cursor, Perplexity, already trust WorkOS.
• 32:08
  SPEAKER_01: Check it out at workos.com.
• 32:10
  SPEAKER_01: I also want to talk about Buildkite,
• 32:11
  SPEAKER_01: the CI orchestration platform trusted by Cursor,
• 32:14
  SPEAKER_01: OpenAI, Entropic, NVIDIA, Uber, Canva, and more.
• 32:16
  SPEAKER_01: Charity talked about owning the loop.
• 32:18
  SPEAKER_01: But here's a challenge.
• 32:20
  SPEAKER_01: Thanks to AI, your agents are writing a lot more code.
• 32:22
  SPEAKER_01: To trust this code,
• 32:24
  SPEAKER_01: every change that an But
• 32:31
  SPEAKER_01: when agents are pushing 5, 10,
• 32:33
  SPEAKER_01: or 50 times the commit volume to your pipelines,
• 32:35
  SPEAKER_01: faster CI runners won't be enough to keep up with it.
• 32:38
  SPEAKER_01: Shaving 30 seconds off a single build is meaningless when the queue is 100 plus jobs deep.
• 32:42
  SPEAKER_01: What you really want is a CI system that gets faster as the volume grows and CI
• 32:47
  SPEAKER_01: that offers instant parallelization to give you unlimited concurrency and
• 32:51
  SPEAKER_01: to intelligently route changes at runtime.
• 32:54
  SPEAKER_01: This is what BuildKite does and why global software leaders
• 32:57
  SPEAKER_01: at Azure Buildkite continues to reliably grow.
• 32:59
  SPEAKER_01: Agents run on your infrastructure or on Buildkite.
• 33:18
  SPEAKER_01: Any cloud, any chip, your secrets, your scale.
• 33:21
  SPEAKER_01: Every artifact and log is captured,
• 33:23
  SPEAKER_01: so when something fails, either you or your agents have immediate insight for why.
• 33:28
  SPEAKER_01: As you're entering the context you give to your agents,
• 33:30
  SPEAKER_01: think about how you'll verify what they hand back.
• 33:33
  SPEAKER_01: If your system is buckling under the increased volume, head to buildkite.com slash pragmatic.
• 33:37
  SPEAKER_01: 30-day all-access trial, no credit card, and an actual human engineer on standby.
• 33:42
  SPEAKER_01: His name's Ola, and he's very helpful.
• 33:44
  SPEAKER_01: And with this, let's get back to charity and communication norms with AI.
• 33:48
  SPEAKER_00: I think there was this frenzy of, oh my
• 33:50
  SPEAKER_02: God, I can do this.
• 33:51
  SPEAKER_02: Oh my God, it's so cool.
• 33:53
  SPEAKER_00: And I know you have also become very weary of this slot.
• 34:00
  SPEAKER_00: I just don't even read it anymore.
• 34:01
  SPEAKER_00: As soon as I can tell.
• 34:02
  SPEAKER_02: As soon as I recognize this might have been AI.
• 34:04
  SPEAKER_02: It's like trash.
• 34:06
  SPEAKER_00: Here's a baseline.
• 34:08
  SPEAKER_00: You cannot send anyone something you haven't read.
• 34:11
  SPEAKER_00: And in fact,
• 34:11
  SPEAKER_00: if it would take them longer to read it than it took you to make it,
• 34:14
  SPEAKER_00: it's probably slot.
• 34:16
  SPEAKER_00: That's really disrespectful, actually.
• 34:18
  SPEAKER_00: And I think like just like asking someone like you're asking.
• 34:22
  SPEAKER_02: Anytime I give you something, I'm asking for your time and attention.
• 34:25
  SPEAKER_00: And if I'm giving you something that I don't even know what's in it and I
• 34:29
  SPEAKER_00: and I'm putting it on you, it costs you instead of me.
• 34:33
  SPEAKER_00: That is not good.
• 34:36
  SPEAKER_00: I also think that even before that,
• 34:38
  SPEAKER_00: it's like I've noticed as I start working on these norms and values,
• 34:41
  SPEAKER_00: I'm noticing myself.
• 34:43
  SPEAKER_00: As I start to ask someone a question without trying to look up the answer,
• 34:48
  SPEAKER_00: ooh, I shouldn't do that.
• 34:49
  SPEAKER_00: Or if I'm giving someone something that I kind of generated and I'm like,
• 34:53
  SPEAKER_00: ooh, you know, so part of it is just self-awareness.
• 34:57
  SPEAKER_02: It's interesting because everything you talked about,
• 34:59
  SPEAKER_02: it reminds me of when a new joiner would join a team,
• 35:03
  SPEAKER_02: a junior engineer,
• 35:04
  SPEAKER_02: a new grad,
• 35:05
  SPEAKER_02: either they had emotional intelligence or they picked up on really quickly that,
• 35:09
  SPEAKER_02: for example,
• 35:10
  SPEAKER_02: you go and ask a senior of their time,
• 35:13
  SPEAKER_02: Once you put in a little bit of work and you start to respect their time as well.
• 35:16
  SPEAKER_02: And obviously it doesn't start like that.
• 35:17
  SPEAKER_02: We don't want them, but there's this balance.
• 35:19
  SPEAKER_02: And I almost feel it's the same thing.
• 35:21
  SPEAKER_02: We're like, look, like respect your colleagues, respect fellow humans.
• 35:24
  SPEAKER_02: If you are communicating with them, make sure that you're not wasting their attention.
• 35:29
  SPEAKER_02: Because now I guess attention is where we're kind of running low.
• 35:33
  SPEAKER_02: Like we have all of these,
• 35:35
  SPEAKER_02: like a bunch of people have a bunch of agents doing,
• 35:37
  SPEAKER_02: but the point is that's kind of the currency.
• 35:39
  SPEAKER_02: And as long as you respect that, it doesn't matter.
• 35:42
  SPEAKER_02: I think we're not talking about don't use AI for this or that.
• 35:45
  SPEAKER_02: Use it as much as you want or make yourself more efficient.
• 35:48
  SPEAKER_02: Just don't degrade because it really degrades those personal skills, right?
• 35:52
  SPEAKER_00: You can use AI as a shortcut to help you not have to think too much.
• 35:57
  SPEAKER_00: And you can use AI to help you think more deeply and more rigorously.
• 36:02
  SPEAKER_00: And most of those use cases have their place.
• 36:04
  SPEAKER_00: But when it comes to your core job functions, We primarily want the second one, right?
• 36:10
  SPEAKER_00: And especially if you're involving someone else and you're asking them to review or,
• 36:13
  SPEAKER_00: you know, and this is not absolutist.
• 36:15
  SPEAKER_00: Like there are people who English is a second language and they use it.
• 36:18
  SPEAKER_00: People who like neurodivergent and that is, again, that is still being respectful, you know?
• 36:24
  SPEAKER_00: So it's not like,
• 36:25
  SPEAKER_00: like you said,
• 36:25
  SPEAKER_00: it's not no AI,
• 36:27
  SPEAKER_00: but it's like make reasonable asks of each other and,
• 36:31
  SPEAKER_00: you know,
• 36:32
  SPEAKER_00: We don't need to reinvent a new bar for quality or respect
• 36:36
  SPEAKER_00: because we have great bars already for quality and respect.
• 36:40
  SPEAKER_00: We just need to apply for a while there.
• 36:42
  SPEAKER_00: I think that there was a bit of, oh my God, this is so cool.
• 36:45
  SPEAKER_00: Do you see what this cool thing can do?
• 36:47
  SPEAKER_00: And I think we're all just like so over it.
• 36:49
  SPEAKER_02: The reason I really respect that you came from the sys-dev background,
• 36:56
  SPEAKER_02: you also, you're very involved in SRE.
• 36:58
  SPEAKER_02: These are all folks who have been pretty skeptical of AI.
• 37:01
  SPEAKER_02: And you mentioned how you're seeing two camps, two very clear camps.
• 37:06
  SPEAKER_02: There's like kind of the AI pills folks who get it.
• 37:10
  SPEAKER_02: And then the people who seem to just hate AI.
• 37:14
  SPEAKER_02: And you said that you're not seeing these two camps have any sort of way to go
• 37:19
  SPEAKER_02: between any feedback.
• 37:20
  SPEAKER_02: Can we talk about what you're seeing and maybe where you see some of these camps forming?
• 37:27
  SPEAKER_00: See, the problem is that neither side is making it up.
• 37:31
  SPEAKER_00: They are seeing really scary events.
• 37:37
  SPEAKER_00: Trends, they're grappling with real hard problems that are getting worse.
• 37:41
  SPEAKER_00: You know,
• 37:41
  SPEAKER_00: and on the enthusiast side,
• 37:43
  SPEAKER_00: it's like they're acutely conscious that it's a bit of a race and that we need
• 37:50
  SPEAKER_00: to push ourselves out of our comfort zone.
• 37:52
  SPEAKER_00: And they see other companies moving faster, catching up, leapfrogging.
• 37:57
  SPEAKER_00: They're really worried about, you know, we're falling behind.
• 38:01
  SPEAKER_00: And the first thing, I don't want to make it sound like false equivalents because...
• 38:05
  SPEAKER_00: While there are elements of this that are true,
• 38:07
  SPEAKER_00: I think every company is more one or more the other, but like they're not wrong.
• 38:12
  SPEAKER_00: They're not wrong.
• 38:13
  SPEAKER_00: We've never seen technological change this fast.
• 38:16
  SPEAKER_00: We're on the inside of an exponential curve,
• 38:18
  SPEAKER_00: which is very rare and it never usually lasts that long,
• 38:22
  SPEAKER_00: but it's still happening, you know?
• 38:25
  SPEAKER_00: Things that are happening that shock us and we would be wise to prepare for them.
• 38:29
  SPEAKER_00: So like that's real.
• 38:30
  SPEAKER_00: That's real.
• 38:31
  SPEAKER_00: And these folks are usually at most companies,
• 38:34
  SPEAKER_00: usually they are the small minority and they are constantly feeling outmanned.
• 38:38
  SPEAKER_00: One of the things that's ironic though is that both
• 38:40
  SPEAKER_00: of these sides feel like they are the tiny minority and they're outmanned
• 38:43
  SPEAKER_00: and they're being suppressed and they are standing up for what is truth and valor in
• 38:48
  SPEAKER_00: the face of the big AI folks or the big skeptics.
• 38:52
  SPEAKER_00: But the other side,
• 38:53
  SPEAKER_00: and this often starts to come down to the group that is on call and the group
• 38:58
  SPEAKER_00: that is not.
• 39:00
  SPEAKER_02: Ooh, yep.
• 39:01
  SPEAKER_00: Because the people who, the buck stops with them.
• 39:05
  SPEAKER_00: They are seeing melting mental models.
• 39:07
  SPEAKER_00: They're seeing slop.
• 39:08
  SPEAKER_00: They're seeing all their hard work just dissolve.
• 39:12
  SPEAKER_00: And they don't see any end in sight.
• 39:15
  SPEAKER_02: So just to be clear,
• 39:15
  SPEAKER_02: we're seeing that the people who are on call for a lot
• 39:17
  SPEAKER_02: of these systems are seeing more incidents.
• 39:20
  SPEAKER_02: They're seeing carelessness being caused by it.
• 39:22
  SPEAKER_02: They're actually seeing that since that group started to use more AI,
• 39:26
  SPEAKER_02: our systems are getting way worse.
• 39:27
  SPEAKER_02: Way worse.
• 39:28
  SPEAKER_00: Yeah.
• 39:29
  SPEAKER_00: And that's very real.
• 39:30
  SPEAKER_00: I'm not making it out.
• 39:31
  SPEAKER_02: No, no, no.
• 39:32
  SPEAKER_02: Actually, I was just talking to someone inside of Meta.
• 39:35
  SPEAKER_02: There's been this big drama where people have been reassigned.
• 39:37
  SPEAKER_02: Oh, God, I know.
• 39:37
  SPEAKER_00: I saw your post.
• 39:38
  SPEAKER_02: So, not just my post since,
• 39:40
  SPEAKER_02: and I haven't written about this since,
• 39:42
  SPEAKER_02: and I'm not sure when this podcast came out,
• 39:44
  SPEAKER_02: I might have not talked about it,
• 39:45
  SPEAKER_02: is inside of Meta, they track SevZeros, which is the highest severity.
• 39:50
  SPEAKER_02: Oh, I know.
• 39:50
  SPEAKER_02: I remember.
• 39:51
  SPEAKER_02: You remember SevZeros.
• 39:52
  SPEAKER_02: There has been a flurry of SevZeros, so many of them, and you cannot hide.
• 39:58
  SPEAKER_02: Like, this is, you know, Meta, like, this is black or white.
• 40:02
  SPEAKER_02: And the past about two months, it's been crazy.
• 40:06
  SPEAKER_02: And just so it happens,
• 40:07
  SPEAKER_02: it's happening inside of Instagram,
• 40:09
  SPEAKER_02: it's happening inside of WhatsApp,
• 40:10
  SPEAKER_02: where the trust and safety,
• 40:13
  SPEAKER_02: basically the reliability folks, have been axed, removed.
• 40:19
  SPEAKER_02: So it's impossible to deny the connection.
• 40:23
  SPEAKER_02: Of course, it's not a direct one, but again, each one has it as a postmortem.
• 40:27
  SPEAKER_02: But Meta has not had this bad for...
• 40:30
  SPEAKER_02: Closer to a decade.
• 40:32
  SPEAKER_01: Yeah.
• 40:33
  SPEAKER_00: Move fast and break things.
• 40:34
  SPEAKER_02: And you put two plus two together.
• 40:36
  SPEAKER_02: And when I told this story at a conference,
• 40:38
  SPEAKER_02: people came up to me and they said,
• 40:40
  SPEAKER_02: I'm so glad you talked about this because my company,
• 40:44
  SPEAKER_02: different company,
• 40:44
  SPEAKER_02: often VC funded or publicly traded, like the same
• 40:47
  SPEAKER_01: thing is happening.
• 40:48
  SPEAKER_01: People are like whispering to me, like we are not metal, but the same thing is happening.
• 40:52
  SPEAKER_00: Same thing is happening.
• 40:53
  SPEAKER_01: And you know what they all told me?
• 40:54
  SPEAKER_01: They told me, I thought it's just us.
• 40:56
  SPEAKER_02: Or I thought it's us and then my buddy who works at this other company.
• 41:00
  SPEAKER_02: And suddenly it's like, oh, it's all of us.
• 41:02
  SPEAKER_02: No, it's all of us.
• 41:03
  SPEAKER_00: Yeah.
• 41:03
  SPEAKER_00: No, it's a real thing.
• 41:05
  SPEAKER_00: And the intercom folks,
• 41:07
  SPEAKER_00: you know, what I love about them is they publish the real gnarly stuff, right?
• 41:15
  SPEAKER_02: They don't color it out.
• 41:15
  SPEAKER_00: They don't color it out.
• 41:17
  SPEAKER_00: And they showed that for 18 months, reliability and code quality went down.
• 41:23
  SPEAKER_00: And it had just started to...
• 41:26
  SPEAKER_00: Possibly be going back up.
• 41:27
  SPEAKER_02: But it's still not there where it was.
• 41:29
  SPEAKER_02: It's still not there where it was.
• 41:30
  SPEAKER_02: And they're very honest about it.
• 41:31
  SPEAKER_00: And they're honest about it.
• 41:33
  SPEAKER_02: Finally.
• 41:34
  SPEAKER_02: Right?
• 41:35
  SPEAKER_00: This is the thing.
• 41:36
  SPEAKER_00: Like, stop, like,
• 41:37
  SPEAKER_00: spitting in my, and telling me that, you know, like, it's just, this is my thing.
• 41:42
  SPEAKER_00: It's like, we need to hear the wins.
• 41:46
  SPEAKER_00: We need to hear what's, we need to hear about what's possible.
• 41:49
  SPEAKER_00: We need to hear what's exciting.
• 41:51
  SPEAKER_00: But you gotta couple it with the costs.
• 41:55
  SPEAKER_00: You gotta couple it with a, is it worth it?
• 41:58
  SPEAKER_00: You've got to couple it with what are we doing?
• 42:00
  SPEAKER_00: What is happening?
• 42:01
  SPEAKER_00: And I feel like part of the reason that both of these sides are getting
• 42:05
  SPEAKER_00: so frustrated is because they're not connecting at all.
• 42:08
  SPEAKER_00: And so the people who are seeing really incredible,
• 42:12
  SPEAKER_00: there are some really incredible things happening in software right now,
• 42:14
  SPEAKER_00: like with rewrites and with, you know, automating away, like real toil.
• 42:19
  SPEAKER_00: Like not a single person that I've talked to would give it up.
• 42:23
  SPEAKER_00: Yeah.
• 42:23
  SPEAKER_00: It's amazing.
• 42:24
  SPEAKER_00: Like they don't get so excited.
• 42:27
  SPEAKER_00: Nobody wants to take it away.
• 42:29
  SPEAKER_00: But half of the people are seeing the wins and they're not connecting it to
• 42:34
  SPEAKER_00: the cost,
• 42:35
  SPEAKER_00: which makes them think that their co-workers are just fuck nuts,
• 42:39
  SPEAKER_00: who are just like, oh, they just don't want to lose their jobs.
• 42:41
  SPEAKER_00: They're just afraid of getting automated out of existence.
• 42:44
  SPEAKER_00: They're just blah, blah, blah, blah, blah.
• 42:46
  SPEAKER_00: Like, no, dude, you be on call and then see how you feel, you know?
• 42:49
  SPEAKER_00: And there's a mirror effect happening where the folks who are on call,
• 42:53
  SPEAKER_00: who are responsible for this stuff,
• 42:55
  SPEAKER_00: They don't actually believe that these winds are real.
• 42:57
  SPEAKER_00: They think they're all cooked because they're not hearing
• 43:01
  SPEAKER_00: the quiet part said out loud that,
• 43:03
  SPEAKER_00: yeah, we're seeing this wind, but this is what it costs.
• 43:05
  SPEAKER_00: We're still cleaning this up.
• 43:07
  SPEAKER_00: And so that's my beg to everyone who loves Geerge's podcast and listens
• 43:15
  SPEAKER_00: to this is
• 43:16
  SPEAKER_02: tell the whole story.
• 43:19
  SPEAKER_00: Talk about the costs.
• 43:22
  SPEAKER_00: We're all in it together.
• 43:23
  SPEAKER_02: Yeah, because you're right.
• 43:25
  SPEAKER_02: This technology is not going anywhere.
• 43:26
  SPEAKER_02: It will make really big positive change at a bunch of places.
• 43:30
  SPEAKER_02: It's here.
• 43:30
  SPEAKER_00: But it's not magic.
• 43:31
  SPEAKER_02: It's not magic.
• 43:32
  SPEAKER_02: And I think this is what you said.
• 43:34
  SPEAKER_02: In Make AI Boring Again,
• 43:36
  SPEAKER_02: another great article of yours, what you said is AI is just technology.
• 43:41
  SPEAKER_02: Just technology.
• 43:42
  SPEAKER_02: And you were arguing that let's just realize it's technology,
• 43:46
  SPEAKER_02: it's a tool, and let's learn to use it well.
• 43:49
  SPEAKER_02: Now, one other thing you said,
• 43:51
  SPEAKER_02: which is very interesting, is software will be the killer app with AI.
• 43:55
  SPEAKER_00: Yeah, I think so.
• 43:57
  SPEAKER_02: Which is very unique.
• 43:58
  SPEAKER_02: Let's talk a little bit about that.
• 43:59
  SPEAKER_00: Software is made of logic and language.
• 44:02
  SPEAKER_00: AI is made of logic and language.
• 44:04
  SPEAKER_00: And because of that,
• 44:07
  SPEAKER_00: we can bake in guardrails,
• 44:09
  SPEAKER_00: we can bake in checks,
• 44:10
  SPEAKER_00: we can bake in validation that we,
• 44:15
  SPEAKER_00: I don't know how we do that in other parts of our lives or other applications.
• 44:18
  SPEAKER_00: And so it totally makes sense to me that Software is what AI is best at.
• 44:24
  SPEAKER_00: I mean, you see like in the courts,
• 44:25
  SPEAKER_00: they're starting to get lawsuits for the court is suing lawyers
• 44:29
  SPEAKER_00: who are submitting briefs that have hallucinated crap in them.
• 44:34
  SPEAKER_00: How do you check for that?
• 44:35
  SPEAKER_00: You know,
• 44:36
  SPEAKER_00: with the same,
• 44:37
  SPEAKER_00: we have structured data,
• 44:39
  SPEAKER_00: we have,
• 44:40
  SPEAKER_00: you know, a whole, and I just don't know how you account for that in the same way.
• 44:44
  SPEAKER_02: It might also mean that whatever will work outside of the software industry for AI,
• 44:49
  SPEAKER_02: it will be a subset of what will work in the second...
• 45:02
  SPEAKER_02: Good luck.
• 45:03
  SPEAKER_02: You will not be able to do it because we have the domain where you can validate stuff.
• 45:08
  SPEAKER_02: We have incredible training data on code that compiles, right?
• 45:11
  SPEAKER_00: Yes, yes.
• 45:12
  SPEAKER_02: Like in a bunch of places, you might have like training data like with magazines.
• 45:16
  SPEAKER_02: You might have like low quality magazines or whatnot, right?
• 45:19
  SPEAKER_02: You see what I mean?
• 45:19
  SPEAKER_00: I mean, back to your point about humans like their determinism.
• 45:23
  SPEAKER_00: They like things to happen the same way.
• 45:25
  SPEAKER_02: And it's very interesting because as I think of it, you know, one of my businesses is writing.
• 45:29
  SPEAKER_02: I write a newsletter that is, I like to think it's good and it's worth reading.
• 45:34
  SPEAKER_00: It is.
• 45:34
  SPEAKER_02: And I would have said, if you asked me, what is AI really good at?
• 45:38
  SPEAKER_02: Now, obviously it's good at coding, but before that it was good at writing.
• 45:41
  SPEAKER_02: It was like, my mind was blown that it can actually control the language.
• 45:45
  SPEAKER_02: When all the newer models come out,
• 45:47
  SPEAKER_02: I do this test where I say like,
• 45:48
  SPEAKER_02: all right, like, you know, write an article in the style of The Pragmatic Engineer.
• 45:53
  SPEAKER_02: And every single time I can tell it's AI generated because it's repetitive.
• 45:57
  SPEAKER_02: It has this thing.
• 45:58
  SPEAKER_02: So my point is AI is actually not as good as writing prose as it's a lot better in writing code.
• 46:04
  SPEAKER_00: Way better at writing.
• 46:05
  SPEAKER_02: When I ask to write code, like I often I'm like, yeah, this is something I could have written.
• 46:09
  SPEAKER_02: Whereas when I asked it to write words,
• 46:10
  SPEAKER_02: I'm like, I would have never written this and it has training data on me.
• 46:13
  SPEAKER_02: So who knows?
• 46:14
  SPEAKER_02: This might prove that software is the best fit.
• 46:17
  SPEAKER_00: I think it is.
• 46:18
  SPEAKER_00: Software is a simplified version of language for a purpose.
• 46:25
  SPEAKER_00: At first,
• 46:26
  SPEAKER_00: everybody was trying to come up with ways to be more efficient and write with AI
• 46:29
  SPEAKER_00: and everything.
• 46:30
  SPEAKER_00: And I sunk a lot of cycles into that.
• 46:32
  SPEAKER_00: And I have decided not to think anymore because writing is thinking on paper.
• 46:38
  SPEAKER_00: And there's no shortcuts.
• 46:39
  SPEAKER_00: I use it for structure.
• 46:39
  SPEAKER_00: I'll be like, hey, read this and give me feedback and stuff, but you
• 47:06
  SPEAKER_02: Also, I think we should not forget that as we improve our skills,
• 47:12
  SPEAKER_02: our capability, our experience, our thoughts, we do become more valuable.
• 47:18
  SPEAKER_02: And I have this idea,
• 47:19
  SPEAKER_02: and this might be a flawed idea,
• 47:21
  SPEAKER_02: but I think it will be correct, that five years from now, how will people be hired?
• 47:26
  SPEAKER_02: Now, of course,
• 47:27
  SPEAKER_02: we know the tools will be better and all that,
• 47:28
  SPEAKER_02: but in the end, I think it will be like this.
• 47:30
  SPEAKER_02: Someone's sitting here and I'm going to be interviewing with you.
• 47:33
  SPEAKER_02: I'm going to be trying to get into your company, probably Honeycomb, right?
• 47:37
  SPEAKER_02: And we will be having a conversation and you will judge me based on how I respond.
• 47:43
  SPEAKER_02: And the more I have spent thinking and bettering myself,
• 47:48
  SPEAKER_02: the more valuable I will be to you because you will have all these candidates
• 47:51
  SPEAKER_02: and some of them will have outsourced or other things to AI and they will have
• 47:54
  SPEAKER_02: a blank because that thing is off.
• 47:57
  SPEAKER_02: Guess who you will want to work with, right?
• 47:59
  SPEAKER_00: I am so excited about leaning into the parts of being human together.
• 48:04
  SPEAKER_00: I don't like the feeling of chatting all day back and forth between agents and people on Slack.
• 48:10
  SPEAKER_00: Like, it feels way too similar.
• 48:12
  SPEAKER_00: It's just gross.
• 48:13
  SPEAKER_00: Honeycomb is a fully distributed company, which was never...
• 48:17
  SPEAKER_00: We always wanted to have a hybrid model, but the office has not come back.
• 48:20
  SPEAKER_00: And I feel all kinds of ways about this because I love not leaving the house.
• 48:25
  SPEAKER_00: But at the same time, I crave this more full, like, I'm so glad you're here.
• 48:31
  SPEAKER_00: It's so nice to see you.
• 48:33
  SPEAKER_02: We were just talking how it is different.
• 48:35
  SPEAKER_02: We've done a podcast remote and it was a decent one, but this is more enjoyable.
• 48:40
  SPEAKER_00: Yes.
• 48:41
  SPEAKER_00: And so part of what I hope we do is just remember that we're in charge of the machines.
• 48:48
  SPEAKER_00: They serve us.
• 48:49
  SPEAKER_00: And this is still what matters.
• 48:50
  SPEAKER_02: Now, I want to pull back to something different.
• 48:53
  SPEAKER_02: I'll just talk a bit more about ops and DevOps and give one of your spicy stakes.
• 48:57
  SPEAKER_02: So now that we have AI,
• 48:58
  SPEAKER_02: we can actually just, you know, badmouth some of the other thing or just be real.
• 49:02
  SPEAKER_02: Let's talk about DevOps.
• 49:03
  SPEAKER_02: Just can we go back a little bit in time?
• 49:05
  SPEAKER_02: You were there.
• 49:06
  SPEAKER_02: Why was it created?
• 49:08
  SPEAKER_02: And in the end, there was this massive DevOps movement in the 2010s.
• 49:11
  SPEAKER_02: Do you think it succeeded?
• 49:13
  SPEAKER_02: Do you think it failed?
• 49:14
  SPEAKER_02: Yeah.
• 49:15
  SPEAKER_00: So before DevOps,
• 49:17
  SPEAKER_00: we needed a DevOps because there was devs and ops and there was the proverbial wall
• 49:23
  SPEAKER_00: that code got thrown over, right?
• 49:25
  SPEAKER_02: And ops were the people who were in charge of the IT,
• 49:28
  SPEAKER_02: they deployed, they managed the servers, they set the Linux version.
• 49:32
  SPEAKER_00: And crafted Linux, you know, pluggable storage models and everything.
• 49:37
  SPEAKER_00: That was always a bad idea because it's split brain.
• 49:41
  SPEAKER_00: Half of you are writing the code and the other half are understanding it.
• 49:44
  SPEAKER_00: I would argue that you can't really understand the code you write unless you're operating it.
• 49:48
  SPEAKER_00: So, you know,
• 49:49
  SPEAKER_00: the DevOps movement did a lot of good trying to knit back together that sort
• 49:53
  SPEAKER_00: of original sin.
• 49:56
  SPEAKER_00: And, you know, around the time that I was a sysadmin, there was this big push.
• 50:00
  SPEAKER_00: All right, ops people, learn to code.
• 50:02
  SPEAKER_00: And great, I'm glad that happened.
• 50:04
  SPEAKER_00: Everyone who works with computers should be writing code.
• 50:06
  SPEAKER_00: I feel like the wave after that It was a little less successful,
• 50:11
  SPEAKER_00: which was like,
• 50:11
  SPEAKER_00: okay, software engineers, time to learn to understand your code in production.
• 50:17
  SPEAKER_00: But I also think that, in my mind, 20 years of DevOps was really about one thing.
• 50:25
  SPEAKER_00: Trying to create one feedback loop that connected people writing code to To
• 50:31
  SPEAKER_00: that code in production.
• 50:32
  SPEAKER_00: And it failed.
• 50:33
  SPEAKER_00: I mean, it failed.
• 50:34
  SPEAKER_00: To this day.
• 50:36
  SPEAKER_00: They're done by two different domains.
• 50:38
  SPEAKER_00: There are some people who...
• 50:41
  SPEAKER_02: And I'll show you this diagram that you drew.
• 50:44
  SPEAKER_02: We now added agents.
• 50:45
  SPEAKER_02: We'll put it on the, so viewers can see it.
• 50:48
  SPEAKER_02: This is your, I think it's a really nice draw up of how there is no feedback loop.
• 50:53
  SPEAKER_02: Like the office people, or oftentimes we call it platform teams, they manage the infralayer.
• 50:59
  SPEAKER_02: Engineers deploy there.
• 51:00
  SPEAKER_00: And so to be clear, I think that's actually good and fine and healthy.
• 51:02
  SPEAKER_00: I think that there are separations of concerns where you can't expect anyone to do everything.
• 51:08
  SPEAKER_00: And the nice separation of concern is,
• 51:10
  SPEAKER_00: do I own,
• 51:11
  SPEAKER_00: am I responsible for the stability of the things that you put code on,
• 51:15
  SPEAKER_00: or am I responsible for the code that I put on the thing, right?
• 51:18
  SPEAKER_00: That is a nice seam because you want the infrastructure to Be stable,
• 51:26
  SPEAKER_00: be like to protect itself, to be resilient and all these things.
• 51:29
  SPEAKER_00: And you want your code like to be oriented towards is every single user having
• 51:34
  SPEAKER_00: a good experience.
• 51:35
  SPEAKER_00: You can have one of those things be true and the other not be true.
• 51:38
  SPEAKER_00: Like they are decoupleable.
• 51:39
  SPEAKER_02: And actually, this is like even the most modern companies.
• 51:43
  SPEAKER_02: I often refer to Entropic as this company which operates in a very different way
• 51:47
  SPEAKER_02: to most companies.
• 51:48
  SPEAKER_02: They're very successful despite doing a lot of different things.
• 51:51
  SPEAKER_02: However, internally, they have platform teams.
• 51:54
  SPEAKER_02: They have the cloud platform teams.
• 51:55
  SPEAKER_02: And then they have applied AI.
• 51:58
  SPEAKER_02: Which is more of the feature teams, the integration.
• 52:02
  SPEAKER_02: And the two, I talked to both of them, they just have a very different outlook.
• 52:04
  SPEAKER_02: They have a very different view on even basic stuff like, will software engineers be obsolete?
• 52:10
  SPEAKER_02: The people on the platform team were like, no, we're working really hard.
• 52:13
  SPEAKER_02: And on the apply, they're like, well,
• 52:14
  SPEAKER_02: maybe it will happen.
• 52:15
  SPEAKER_00: That does not surprise me one tiny biota.
• 52:19
  SPEAKER_02: But so this company, Anthropic, that started with a blank page, they arrived at the same place.
• 52:24
  SPEAKER_02: Yeah.
• 52:25
  SPEAKER_00: Yeah, no,
• 52:26
  SPEAKER_00: I think it's the right separation of concern,
• 52:28
  SPEAKER_00: and I'm not trying to erase it,
• 52:29
  SPEAKER_00: but I think that to be a good engineer, you need fast feedback loops.
• 52:35
  SPEAKER_00: And this is part and parcel with the whole, oh, the source of truth is the code.
• 52:39
  SPEAKER_00: If that's where you live, if you live in the land of how it should theoretically work, no.
• 52:45
  SPEAKER_00: And I think that with agents...
• 52:48
  SPEAKER_00: They're breaking that, right?
• 52:50
  SPEAKER_00: They're breaking that and they're forcing another thing on the observability trip is
• 52:55
  SPEAKER_00: a lot of people, if you say like, what is observability?
• 52:57
  SPEAKER_00: They'll be like, ah, well, there's three pillars.
• 52:59
  SPEAKER_00: There's metrics, logs, and traces.
• 53:00
  SPEAKER_00: We talked about this last time.
• 53:02
  SPEAKER_02: Metrics and logs, I would say, are
• 53:04
  SPEAKER_00: system exhaust.
• 53:05
  SPEAKER_00: They're the exhaust pipe.
• 53:06
  SPEAKER_00: And they're never going away because every team runs a ton of third-party software.
• 53:12
  SPEAKER_00: They didn't write it.
• 53:13
  SPEAKER_00: They don't own it.
• 53:14
  SPEAKER_00: They just have to run it.
• 53:15
  SPEAKER_00: And it's outputting shit.
• 53:17
  SPEAKER_02: Yeah.
• 53:17
  SPEAKER_00: And you just got to put it somewhere.
• 53:19
  SPEAKER_02: You observe it.
• 53:19
  SPEAKER_02: You see what.
• 53:20
  SPEAKER_02: Yeah, yeah, yeah.
• 53:21
  SPEAKER_02: And then you do stuff with it.
• 53:22
  SPEAKER_00: Yeah.
• 53:23
  SPEAKER_00: And, you know, you should put it somewhere cheap.
• 53:25
  SPEAKER_00: There's a ton of it.
• 53:26
  SPEAKER_00: It's not super high value, but you definitely need it.
• 53:28
  SPEAKER_00: Right.
• 53:28
  SPEAKER_00: And you can't do anything about it.
• 53:29
  SPEAKER_00: You just take it and put it somewhere.
• 53:32
  SPEAKER_00: Then there's your code, there's your crown jewels, the code that makes you a company.
• 53:39
  SPEAKER_00: And for that code, your telemetry should be a product decision.
• 53:45
  SPEAKER_00: It should be you store it once with all the connective tissue because the value
• 53:53
  SPEAKER_00: of rich data goes up, not linearly, not even exponentially, combinatorially.
• 53:58
  SPEAKER_00: If you have a wide event or a trace with 29 bits of data and you add a 30th,
• 54:04
  SPEAKER_00: that 30th is more valuable than all the others.
• 54:07
  SPEAKER_00: Like it is just so powerful.
• 54:10
  SPEAKER_00: And with non-deterministic software,
• 54:12
  SPEAKER_00: you know right up front you can't predict what it's going to do.
• 54:16
  SPEAKER_00: You have to.
• 54:17
  SPEAKER_00: Like that is a product decision to capture that trace.
• 54:21
  SPEAKER_02: So let's talk specifically about modern observability and companies
• 54:24
  SPEAKER_02: that are either building AI-related code or just complicated code
• 54:29
  SPEAKER_02: that they're generating.
• 54:30
  SPEAKER_02: In the old world, again, I'm just being observability 101 back in the day.
• 54:35
  SPEAKER_02: The way I would have written the code is you write the code and you think,
• 54:38
  SPEAKER_02: hmm, something funny might be going on here.
• 54:40
  SPEAKER_02: Let me do a log or an info or a warn.
• 54:43
  SPEAKER_02: And then I would also try to maybe if we're printing some production,
• 54:47
  SPEAKER_02: I realize like, okay, well, I guess it's crashing and we don't have any logs there.
• 54:50
  SPEAKER_02: So I guess it's some other part.
• 54:52
  SPEAKER_02: Let me put a tool that will like log everything and I'll have a bunch of stuff.
• 54:56
  SPEAKER_02: Now, this is the simplest way of thinking.
• 54:59
  SPEAKER_02: In kind of a modern business where I'm like, I know this is high value stuff.
• 55:04
  SPEAKER_02: What are ways that I can go about that's actually maybe
• 55:06
  SPEAKER_02: a bit
• 55:10
  SPEAKER_00: Auto-instrumentation has gotten so good in recent years.
• 55:14
  SPEAKER_02: If you're using OpenTelemetry,
• 55:16
  SPEAKER_02: and everyone should be using OpenTelemetry,
• 55:19
  SPEAKER_00: all of the common patterns, all of the models are trained on them.
• 55:22
  SPEAKER_00: So it is literally faster and easier to build with instrumentation than not to.
• 55:28
  SPEAKER_02: And with instrumentation,
• 55:31
  SPEAKER_02: once I have the code in a compile step or an extra step,
• 55:34
  SPEAKER_02: it just adds it to the right lines.
• 55:36
  SPEAKER_00: This is what's important, right?
• 55:37
  SPEAKER_00: It's part of just developer intent, right?
• 55:40
  SPEAKER_00: This is how you declare your intent and that's how you check up on your intent in production.
• 55:44
  SPEAKER_02: It's honestly gotten so much simpler.
• 55:46
  SPEAKER_00: You know,
• 55:46
  SPEAKER_00: I don't fault developers or anyone else for not kind of closing that loop
• 55:51
  SPEAKER_00: with DevOps because the fact is it was prohibitively hard and time-consuming
• 55:57
  SPEAKER_00: and difficult because You know,
• 56:00
  SPEAKER_00: you're an old school software engineer and you sit down, write some code.
• 56:04
  SPEAKER_00: You're like, ah, here, I should instrument it and look at it in production.
• 56:07
  SPEAKER_00: So you're like, okay, I've got a bit of data and I want to do something with it.
• 56:12
  SPEAKER_00: All right.
• 56:13
  SPEAKER_00: Is it a metric, a log, a trace, an exception, an error, a profiling?
• 56:20
  SPEAKER_02: You know, just, okay, if it's a metric, is it a
• 56:23
  SPEAKER_00: Is it a counter?
• 56:24
  SPEAKER_00: Is it a gauge?
• 56:25
  SPEAKER_02: Is it a, you know, just like
• 56:27
  SPEAKER_00: all down, it takes so, and then, well, what type of data is it?
• 56:30
  SPEAKER_00: Is it going to have high cardinality?
• 56:32
  SPEAKER_00: Is it going to be a,
• 56:32
  SPEAKER_02: you know,
• 56:33
  SPEAKER_02: just like,
• 56:34
  SPEAKER_02: and you can
• 56:34
  SPEAKER_00: blow, yeah, it's just like, if it's a log line, which log level do I do?
• 56:39
  SPEAKER_00: Do I append it to?
• 56:40
  SPEAKER_00: Like, it's just,
• 56:41
  SPEAKER_00: you could double, triple,
• 56:42
  SPEAKER_00: quadruple the amount of time that you spent writing the code trying
• 56:44
  SPEAKER_00: to instrument it, and then it still wouldn't be done.
• 56:47
  SPEAKER_00: Like, you deploy it,
• 56:48
  SPEAKER_00: and then it's like,
• 56:48
  SPEAKER_00: okay, I know the name of the thing that I added, but, How do I find it?
• 56:54
  SPEAKER_00: How do I display it?
• 56:55
  SPEAKER_00: How do I create a dashboard?
• 56:56
  SPEAKER_00: It's just like, that was prohibitively, that was really hard.
• 57:01
  SPEAKER_00: But now we can bring all of this to you right in your development environment.
• 57:06
  SPEAKER_00: It is easier and faster to instrument with telemetry than without it.
• 57:11
  SPEAKER_00: And you don't have to leave your development environment to go fast.
• 57:14
  SPEAKER_00: And get it,
• 57:15
  SPEAKER_00: you know,
• 57:16
  SPEAKER_00: you could have the agent,
• 57:17
  SPEAKER_00: like we've built some really cool shit at Honeycomb where it'll just,
• 57:20
  SPEAKER_00: it'll be like,
• 57:21
  SPEAKER_00: oh, hey, that thing that you wrote, you know, maybe you want to look at this.
• 57:25
  SPEAKER_00: And you can, you can control how gross it is.
• 57:28
  SPEAKER_00: You can, you know, but it's right there.
• 57:30
  SPEAKER_00: And that's how it should be.
• 57:31
  SPEAKER_00: It should be part of your development loop.
• 57:33
  SPEAKER_02: Can we talk about what spans are?
• 57:34
  SPEAKER_02: Because I'll quote Eric Riddock,
• 57:37
  SPEAKER_02: who recently wrote LinkedIn,
• 57:38
  SPEAKER_02: the basic idea of observability for applications is don't use logs or metrics,
• 57:42
  SPEAKER_02: just put it all in spans.
• 57:44
  SPEAKER_02: What are spans?
• 57:46
  SPEAKER_00: Spans are bits of a trace.
• 57:50
  SPEAKER_00: I mean, a trace is just a structured log with some fancy fields, right?
• 57:53
  SPEAKER_00: And so the span is the subset of the trace that makes up the entire duration.
• 58:00
  SPEAKER_00: And I don't know if you've...
• 58:03
  SPEAKER_00: The default building block has been the transaction for as long as the web has been around.
• 58:09
  SPEAKER_00: That doesn't work anymore.
• 58:11
  SPEAKER_02: With, specifically with AI.
• 58:13
  SPEAKER_00: Yeah.
• 58:14
  SPEAKER_00: We just, we just ship something called timeline that is like, it sits on top of spans.
• 58:20
  SPEAKER_00: So, you know, if, if you,
• 58:22
  SPEAKER_00: you know, if you,
• 58:22
  SPEAKER_00: if you run something like intercom,
• 58:24
  SPEAKER_00: you have got a chat thing and a customer's like, I'm,
• 58:26
  SPEAKER_02: conversation going on.
• 58:28
  SPEAKER_00: Yeah.
• 58:28
  SPEAKER_00: Customer's like, I'm complaining.
• 58:29
  SPEAKER_00: You're like, okay.
• 58:30
  SPEAKER_00: So you spin up an agent,
• 58:31
  SPEAKER_00: supervisor agent that spins up more agents and Each of them calls APIs.
• 58:35
  SPEAKER_00: Each of them calls like storage backends and stuff.
• 58:37
  SPEAKER_00: And then the customer has another.
• 58:39
  SPEAKER_00: It could span hours, right?
• 58:41
  SPEAKER_00: And you need to be able to zoom out and visualize the whole thing.
• 58:47
  SPEAKER_00: It's super cool.
• 58:49
  SPEAKER_02: And so this is a new primitive that you came up for.
• 58:52
  SPEAKER_02: These use cases where there's a conversation or like an element is involved
• 58:56
  SPEAKER_02: and you have like a
• 58:58
  SPEAKER_00: meta trace.
• 59:00
  SPEAKER_00: Oh.
• 59:00
  SPEAKER_02: Okay, yeah.
• 59:01
  SPEAKER_02: A trace of traces.
• 59:03
  SPEAKER_02: So we need these new building blocks to actually just be able to work with.
• 59:07
  SPEAKER_00: Yeah.
• 59:07
  SPEAKER_02: Interesting.
• 59:08
  SPEAKER_02: So I guess this is something to keep in mind.
• 59:10
  SPEAKER_02: Any engineer who's building on top of LMS, who is an AI engineer now, as we know?
• 59:15
  SPEAKER_00: It's either that or you've just got all these tabs open with traces.
• 59:18
  SPEAKER_00: You're just copy pasting IDs from one to the next.
• 59:21
  SPEAKER_02: Yeah.
• 59:22
  SPEAKER_02: Or if you're a large enough company, you might have built your own in-house tool.
• 59:26
  SPEAKER_02: But we know that it's doable, but it's painful.
• 59:30
  SPEAKER_02: It's doable.
• 59:31
  SPEAKER_02: It's painful.
• 59:31
  SPEAKER_00: I'm really looking forward to seeing over the next few months or year or whatever,
• 59:36
  SPEAKER_00: just the marriage of tests and evals from a telemetry perspective.
• 59:44
  SPEAKER_02: With agents and AI agents being around,
• 59:46
  SPEAKER_02: a lot of them are now very useful to connect to observability stores.
• 59:51
  SPEAKER_02: You can go and do stuff.
• 59:53
  SPEAKER_02: However,
• 59:54
  SPEAKER_02: one question that comes up is, well,
• 59:57
  SPEAKER_02: agents have a finite context window and with observability,
• 1:00:01
  SPEAKER_02: you can really easily overload that.
• 1:00:02
  SPEAKER_02: What are approaches you've seen of agents either using honeycombs
• 1:00:07
  SPEAKER_02: or some other data sources to make them productive?
• 1:00:12
  SPEAKER_02: Have you seen some patterns?
• 1:00:13
  SPEAKER_00: There's a lot of trash data out there.
• 1:00:17
  SPEAKER_00: And a lot of traditional telemetry data,
• 1:00:21
  SPEAKER_00: metrics, logs, traces,
• 1:00:23
  SPEAKER_00: results,
• 1:00:24
  SPEAKER_00: it tends to fill up your context window with crap when the most important part of
• 1:00:29
  SPEAKER_00: the data is, again, the relationships between the data.
• 1:00:33
  SPEAKER_00: So if you can,
• 1:00:35
  SPEAKER_00: and in fact,
• 1:00:36
  SPEAKER_00: one of the AI SRE startups posted this great A couple months ago about how they see
• 1:00:44
  SPEAKER_00: the agents that they deploy in the wild bypass the observability data most of
• 1:00:49
  SPEAKER_00: the time, and they go upstream to find richer, intact telemetry data.
• 1:00:56
  SPEAKER_00: So that's what I would say.
• 1:00:57
  SPEAKER_00: Either you give your agents, but it's the relationships that matter, right?
• 1:01:01
  SPEAKER_00: Because that's what actually helps the AI make decisions.
• 1:01:05
  SPEAKER_02: And when it comes to observability, I cannot not mention your book, Observability Engineering.
• 1:01:09
  SPEAKER_02: And you have a second edition.
• 1:01:11
  SPEAKER_02: Can you tell me why you felt the need to write it and what's new in it?
• 1:01:15
  SPEAKER_00: Oh, man.
• 1:01:17
  SPEAKER_00: The whole thing is new.
• 1:01:18
  SPEAKER_00: So O'Reilly,
• 1:01:20
  SPEAKER_00: any time a book is considered successful,
• 1:01:22
  SPEAKER_00: and if the topic is still relevant,
• 1:01:24
  SPEAKER_00: they'll ask if you want to write a second edition.
• 1:01:26
  SPEAKER_00: So it's not really.
• 1:01:28
  SPEAKER_00: But I was really excited to write it.
• 1:01:30
  SPEAKER_00: The first book...
• 1:01:33
  SPEAKER_00: I don't want to say I wasn't proud of it.
• 1:01:34
  SPEAKER_00: You're not like your children in your books.
• 1:01:36
  SPEAKER_00: You're not supposed to like say anything bad about them, you know, because it's fine.
• 1:01:42
  SPEAKER_00: But it was written 2019 to 2021.
• 1:01:45
  SPEAKER_00: The definition of observability meant one thing when we started and another
• 1:01:50
  SPEAKER_00: the time we ended.
• 1:01:51
  SPEAKER_00: And there was no point where I was like, oh, this book is great.
• 1:01:54
  SPEAKER_00: Let's ship it.
• 1:01:55
  SPEAKER_00: It was just like, oh, I can't do this anymore.
• 1:01:57
  SPEAKER_02: Just like please take it and
• 1:01:59
  SPEAKER_00: I hope that's enough.
• 1:02:00
  SPEAKER_00: Now it feels like the definition of observability is more settled.
• 1:02:03
  SPEAKER_00: It's everything else in the world that's like changing and crazy and all.
• 1:02:06
  SPEAKER_00: So I think it's a good book.
• 1:02:07
  SPEAKER_00: I hope it can help a bunch of folks.
• 1:02:10
  SPEAKER_00: It's got six parts.
• 1:02:11
  SPEAKER_00: So the first part is, and I wrote parts one and six.
• 1:02:15
  SPEAKER_00: First part is just kind of like grappling with what does it mean
• 1:02:17
  SPEAKER_00: to run deterministic and non-deterministic systems, you know?
• 1:02:20
  SPEAKER_00: And then,
• 1:02:21
  SPEAKER_00: you know,
• 1:02:21
  SPEAKER_00: my co-authors,
• 1:02:23
  SPEAKER_00: Liz and Austin and George,
• 1:02:25
  SPEAKER_00: The part two and three is how do you instrument your code
• 1:02:30
  SPEAKER_00: and how do you understand it?
• 1:02:32
  SPEAKER_00: And there are parallel tracks for doing this with or without AI.
• 1:02:36
  SPEAKER_00: And a couple of great guest columns from Jeremy.
• 1:02:40
  SPEAKER_00: And then parts four and five are we have a whole lineup of guest authors
• 1:02:47
  SPEAKER_00: and Use cases and deep dives.
• 1:02:50
  SPEAKER_00: I did one on frontend and mobile.
• 1:02:57
  SPEAKER_00: We've got some great ones on CICD.
• 1:02:59
  SPEAKER_00: ClickHouse did one on columnar storage.
• 1:03:02
  SPEAKER_00: Some really, really stellar things.
• 1:03:04
  SPEAKER_00: There's a chapter from Kesha at Finn on how they use it iteratively to do observability.
• 1:03:10
  SPEAKER_02: Wow, so this is a brand new book.
• 1:03:12
  SPEAKER_02: A lot of second editions are like, oh, we added two chapters.
• 1:03:17
  SPEAKER_00: This is an entire rewrite and it's twice as long.
• 1:03:19
  SPEAKER_00: The first one was 250 pages.
• 1:03:20
  SPEAKER_00: This one is 600 pages.
• 1:03:22
  SPEAKER_02: Okay, so I'm interested.
• 1:03:24
  SPEAKER_02: Now I'm going to get this book.
• 1:03:25
  SPEAKER_00: And the part six,
• 1:03:27
  SPEAKER_00: it's my baby,
• 1:03:28
  SPEAKER_00: and it was originally supposed to be three chapters
• 1:03:30
  SPEAKER_00: for observability engineering teams.
• 1:03:31
  SPEAKER_00: And it turned into,
• 1:03:32
  SPEAKER_00: it's a third of the book,
• 1:03:33
  SPEAKER_00: it's 200 pages, but it's topics for observability governance for leaders.
• 1:03:39
  SPEAKER_00: And it starts with an open letter
• 1:03:40
  SPEAKER_00: to CTOs telling them why all their big AI goals are blocked behind their ability
• 1:03:45
  SPEAKER_00: to make sense of their system.
• 1:03:47
  SPEAKER_00: You know, and then we talk about, you know, software delivery for...
• 1:03:52
  SPEAKER_00: No buzzwords, just systems theory.
• 1:03:55
  SPEAKER_00: If you like Donella Meadows stuff, then you will like it.
• 1:04:00
  SPEAKER_00: And then there's a chapter on how to quantify the impact of observability
• 1:04:04
  SPEAKER_00: for your finance,
• 1:04:07
  SPEAKER_00: how to treat observability as an investment versus a cost center,
• 1:04:10
  SPEAKER_00: and when you should use observability as a cost center,
• 1:04:13
  SPEAKER_00: and when you should treat it like an investment.
• 1:04:15
  SPEAKER_00: Because it inherits the type of software that you're observing, you know?
• 1:04:18
  SPEAKER_00: And there's a great guest chapter from Rick Clark
• 1:04:22
  SPEAKER_00: on Staff Plus Principal Distinguished Engineers who are trying
• 1:04:25
  SPEAKER_00: to drive Massive change without authority.
• 1:04:28
  SPEAKER_00: How do you do that?
• 1:04:30
  SPEAKER_00: And how is observability vital to that?
• 1:04:33
  SPEAKER_00: And then there's a chapter on build versus buy versus open source.
• 1:04:39
  SPEAKER_02: I mean, it sounds to me that anyone who is inside or wants to be inside
• 1:04:44
  SPEAKER_02: a platform engineering team,
• 1:04:45
  SPEAKER_02: may you be an engineer or a leader,
• 1:04:48
  SPEAKER_01: you probably want to read this
• 1:04:50
  SPEAKER_02: book.
• 1:04:51
  SPEAKER_00: And at the end,
• 1:04:51
  SPEAKER_00: there's a chapter that is possibly one of my favorites that It's called The Art
• 1:04:58
  SPEAKER_00: and Science of Vendor Partnerships.
• 1:05:00
  SPEAKER_00: And it's just talking about how we can't build all the software that we need.
• 1:05:04
  SPEAKER_00: And great vendor partnerships are ones where you have influence over their roadmap
• 1:05:09
  SPEAKER_00: and they trust you to do these things.
• 1:05:11
  SPEAKER_00: And talking about how most transformations fail.
• 1:05:16
  SPEAKER_00: The ones that succeed, succeed because someone on the inside has trust and credibility.
• 1:05:21
  SPEAKER_00: People believe when you say something, it is true.
• 1:05:24
  SPEAKER_00: You know, it cuts through bureaucracy like a hot knife through butter.
• 1:05:27
  SPEAKER_00: When it comes to partnering with,
• 1:05:29
  SPEAKER_00: you know, the sales org of another company, you do not have trust incredibly.
• 1:05:33
  SPEAKER_00: You work to build trust through reciprocity.
• 1:05:36
  SPEAKER_00: You learned just how much you can trust them over time, right?
• 1:05:41
  SPEAKER_00: But the best vendor relationships are the ones
• 1:05:44
  SPEAKER_00: where you genuinely feel like their successes are your successes.
• 1:05:48
  SPEAKER_00: Your successes are their successes.
• 1:05:50
  SPEAKER_00: You're happy to see each other because each of you are delighted
• 1:05:52
  SPEAKER_00: because you know you're getting something from them.
• 1:05:54
  SPEAKER_00: It feels like you are two different teams working at the same big company.
• 1:05:57
  SPEAKER_00: That is rare.
• 1:05:58
  SPEAKER_00: Doesn't usually happen.
• 1:05:59
  SPEAKER_00: And that's fine.
• 1:06:01
  SPEAKER_00: Most vendor relationships are ones where you shake hands,
• 1:06:03
  SPEAKER_00: you exchange money and services, and that's fine.
• 1:06:05
  SPEAKER_00: But I think in an era of AI, these are durable skills.
• 1:06:11
  SPEAKER_00: These are durable skills for very senior engineers who care about impact.
• 1:06:15
  SPEAKER_02: Senior engineers and also engineering leaders and anyone who wants to become
• 1:06:19
  SPEAKER_02: an engineering leader.
• 1:06:19
  SPEAKER_02: Because I guess, like, I mean,
• 1:06:21
  SPEAKER_02: both of us have been in engineering leadership,
• 1:06:24
  SPEAKER_02: like you've been in much higher positions than I have.
• 1:06:26
  SPEAKER_02: But I think it's fair to say that the way for you to get to that CTO role,
• 1:06:31
  SPEAKER_02: that head of engineering,
• 1:06:32
  SPEAKER_02: that director of engineering is to do the work for six or eight or six months,
• 1:06:38
  SPEAKER_02: a year to year and a half.
• 1:06:40
  SPEAKER_01: And to do so, you need to know these things.
• 1:06:42
  SPEAKER_02: I feel Observer of the Engineering would be underselling this book,
• 1:06:45
  SPEAKER_02: I'll be honest,
• 1:06:46
  SPEAKER_02: the title,
• 1:06:46
  SPEAKER_02: but I'm also going to get it and I'll probably think of ways to share a bit more.
• 1:06:51
  SPEAKER_02: But thank you for writing and thank you to all your co-authors.
• 1:06:55
  SPEAKER_02: But speaking of leadership...
• 1:06:58
  SPEAKER_02: I'd love to talk about a little bit of engineering leadership because there's a lot
• 1:07:02
  SPEAKER_02: of things that are changing.
• 1:07:04
  SPEAKER_02: But I loved one of your very recent takes on leadership and I'm going to quote you.
• 1:07:08
  SPEAKER_02: The most effective leaders are kind, caring humans and skilled business operators.
• 1:07:13
  SPEAKER_02: The second most effective leaders are terrible humans and skilled business operators.
• 1:07:18
  SPEAKER_02: And after that comes everyone else.
• 1:07:20
  SPEAKER_02: There are plenty of good,
• 1:07:21
  SPEAKER_02: kind humans who are sloppy operators and bad at business because being good
• 1:07:24
  SPEAKER_02: at business is very hard.
• 1:07:26
  SPEAKER_02: And you said that.
• 1:07:27
  SPEAKER_02: This in relation to what happened at Twitter slash X,
• 1:07:31
  SPEAKER_02: referring to Elon as a terrible human, but a skilled business operator.
• 1:07:36
  SPEAKER_00: Yeah, well,
• 1:07:38
  SPEAKER_00: I don't know that I would call him the skilled business operator,
• 1:07:41
  SPEAKER_00: but my point was that Twitter had 16 years to figure it out,
• 1:07:47
  SPEAKER_00: and everyone could see that they were not figuring it out.
• 1:07:49
  SPEAKER_00: Figuring out the business, specifically.
• 1:07:52
  SPEAKER_00: Figuring out the business, yeah, building products.
• 1:07:56
  SPEAKER_00: You know, reaching folks.
• 1:07:58
  SPEAKER_00: And you could argue that X has gotten better or worse,
• 1:08:01
  SPEAKER_00: but you can't argue that he is running it with 20% as many people.
• 1:08:06
  SPEAKER_02: Yep.
• 1:08:07
  SPEAKER_02: And it's working.
• 1:08:08
  SPEAKER_00: And it's working.
• 1:08:09
  SPEAKER_00: And some of that, you know, 30 engineers on the core product.
• 1:08:14
  SPEAKER_00: And another 30, but like 60 engineers, there were 1,700 before it.
• 1:08:19
  SPEAKER_00: And you could argue,
• 1:08:20
  SPEAKER_00: and I think it would be true, that it's some of the work that those engineers did.
• 1:08:25
  SPEAKER_00: But this is the point.
• 1:08:26
  SPEAKER_00: If we don't do it ourselves,
• 1:08:28
  SPEAKER_00: meaning hold ourselves to a high standard,
• 1:08:31
  SPEAKER_00: Build with efficiency, constantly be like trying to get better.
• 1:08:34
  SPEAKER_00: If we don't do it ourselves, someone will come and do it to us.
• 1:08:38
  SPEAKER_02: And this is what you also said,
• 1:08:39
  SPEAKER_02: you closed saying,
• 1:08:40
  SPEAKER_02: if we want to remain in leadership,
• 1:08:41
  SPEAKER_02: if we want to set the culture and the tone and take the ethical sense
• 1:08:44
  SPEAKER_02: that we believe in, we first have to win at the business.
• 1:08:47
  SPEAKER_02: And I think this is like,
• 1:08:48
  SPEAKER_02: especially now that there's so many changes happening and technology changes,
• 1:08:52
  SPEAKER_02: there's a whirlwind, business will go up and down.
• 1:08:54
  SPEAKER_02: I guess the reminder that you want to keep your eyes on the prize,
• 1:08:57
  SPEAKER_02: which is,
• 1:08:58
  SPEAKER_02: especially if you're a leader,
• 1:08:59
  SPEAKER_00: The 2010s,
• 1:09:00
  SPEAKER_00: there was so much money sloshing around in Silicon Valley and time started
• 1:09:04
  SPEAKER_00: to get tough and all of these companies canceled their DEI programs and blah,
• 1:09:09
  SPEAKER_00: blah.
• 1:09:10
  SPEAKER_00: Yeah, they never believed in that.
• 1:09:11
  SPEAKER_00: They were just trying to buy people off.
• 1:09:13
  SPEAKER_00: You know, and that is very telling to me.
• 1:09:16
  SPEAKER_00: And I have taken a lot of lessons away from that,
• 1:09:19
  SPEAKER_00: which is just that It's not enough to be a good person.
• 1:09:22
  SPEAKER_00: I believe that people who are kind and care about people can
• 1:09:26
  SPEAKER_00: and usually do do better than sociopaths in the same roles,
• 1:09:30
  SPEAKER_00: but only if they're good at business.
• 1:09:33
  SPEAKER_02: Learn the business, stay close to it.
• 1:09:35
  SPEAKER_00: You got to.
• 1:09:36
  SPEAKER_02: With Now that coding has become cheap,
• 1:09:40
  SPEAKER_02: now that engineers are running agents,
• 1:09:43
  SPEAKER_02: how do you see the role of good,
• 1:09:46
  SPEAKER_02: skilled engineering managers and engineering directors change?
• 1:09:50
  SPEAKER_02: What has changed?
• 1:09:51
  SPEAKER_00: Well, the first thing that's changed is I think everyone gets to be hands-on.
• 1:09:57
  SPEAKER_02: Specifically to generate some code, to ship the production to some extent.
• 1:10:02
  SPEAKER_00: You should know what it feels like to submit a diff, to get a PR through.
• 1:10:07
  SPEAKER_00: You know, you should know what it feels like.
• 1:10:09
  SPEAKER_00: It's just easier now than it's ever been to pick it back up, to fill in the blanks, you know.
• 1:10:15
  SPEAKER_00: And it's always been the case that leaders were better if they had a hand in it.
• 1:10:20
  SPEAKER_00: And now it's just, there's no excuse not to.
• 1:10:24
  SPEAKER_02: Teams are getting smaller.
• 1:10:26
  SPEAKER_00: In general, I think this should be a good thing.
• 1:10:28
  SPEAKER_00: If we can figure out how to own more surface area, it should be a good thing.
• 1:10:33
  SPEAKER_00: I worry that the way it's happening is it's being done by CEOs who are like, oh,
• 1:10:39
  SPEAKER_02: well,
• 1:10:40
  SPEAKER_02: this other company is doing it or it's magic or we're going to do layoffs
• 1:10:44
  SPEAKER_02: or like it.
• 1:10:45
  SPEAKER_00: And I really dislike the anti-management tone.
• 1:10:49
  SPEAKER_00: So like no argument that power tends to drift towards managers over time and needs
• 1:10:54
  SPEAKER_00: to get pushed back to engineers.
• 1:10:56
  SPEAKER_00: No argument there's a tendency to have too many managers.
• 1:11:00
  SPEAKER_00: You know,
• 1:11:01
  SPEAKER_00: the bureaucracy kind of like generates a sort of,
• 1:11:04
  SPEAKER_00: you know, it's easier to say yes than it is to say no.
• 1:11:07
  SPEAKER_00: And so these things happen.
• 1:11:08
  SPEAKER_00: So they need to be pushed back from time to time.
• 1:11:11
  SPEAKER_00: But I believe that Management and middle management is deeply essential.
• 1:11:18
  SPEAKER_00: And I look forward to seeing how that works out for them.
• 1:11:21
  SPEAKER_02: But the role of a manager, middle management,
• 1:11:24
  SPEAKER_00: in my view, is sense-making and context-giving.
• 1:11:29
  SPEAKER_00: Because I don't believe in a world where engineers are just given tasks.
• 1:11:33
  SPEAKER_00: Here's your GRI.
• 1:11:34
  SPEAKER_00: Go do the things.
• 1:11:35
  SPEAKER_00: AI can do that.
• 1:11:37
  SPEAKER_00: I want people who understand what we're trying to do.
• 1:11:41
  SPEAKER_00: Understand how we're trying to do it or who are there
• 1:11:43
  SPEAKER_00: to help us figure out how we're going to do it.
• 1:11:45
  SPEAKER_00: And you can't engage emotionally, creatively, collaboratively without understanding.
• 1:11:51
  SPEAKER_00: And that understanding is incredibly difficult to build and it's fragile
• 1:11:54
  SPEAKER_00: and it never lasts very long.
• 1:11:56
  SPEAKER_02: For those of us listening who are middle managers,
• 1:11:58
  SPEAKER_02: it's been a tough few years because what they're seeing is there's a push
• 1:12:06
  SPEAKER_02: to have fewer of them.
• 1:12:07
  SPEAKER_02: A lot of their colleagues,
• 1:12:09
  SPEAKER_02: if they're in unlucky places,
• 1:12:10
  SPEAKER_02: they were made redundant and many of them have struggled to get similar positions.
• 1:12:15
  SPEAKER_02: We're talking director positions.
• 1:12:16
  SPEAKER_02: We're talking head of engineering, senior engineering manager.
• 1:12:20
  SPEAKER_02: That role is disappearing faster than ever.
• 1:12:23
  SPEAKER_02: I think directors might still be there.
• 1:12:25
  SPEAKER_02: For folks who are in this position and they do like middle management,
• 1:12:30
  SPEAKER_02: they do believe they're good at it,
• 1:12:33
  SPEAKER_02: what do you think tactics could be to give them a bit more career options?
• 1:12:37
  SPEAKER_00: Tactically,
• 1:12:38
  SPEAKER_00: I would say go back to BNIC for a while even if you know it's not what you want
• 1:12:42
  SPEAKER_00: to do.
• 1:12:43
  SPEAKER_00: If you're at all capable, if you're not capable of it, then I would try to work it.
• 1:12:46
  SPEAKER_00: You've got to get AI on your resume.
• 1:12:48
  SPEAKER_00: You just have to.
• 1:12:50
  SPEAKER_00: And this is a huge career risk.
• 1:12:51
  SPEAKER_00: If you're working somewhere where you're not getting these skills, that is a massive risk.
• 1:12:57
  SPEAKER_00: I would do whatever I could.
• 1:12:58
  SPEAKER_02: And this is very interesting that you're saying get AI in your career
• 1:13:02
  SPEAKER_02: because I remember about a year,
• 1:13:03
  SPEAKER_02: year and a half ago, I started to pay attention to like, okay, this is happening.
• 1:13:08
  SPEAKER_02: And I remember a year ago, I wrote an article about how to become an AI engineer.
• 1:13:12
  SPEAKER_02: And I talked with engineers who just like at their workplace started to do AI
• 1:13:15
  SPEAKER_02: and now they're AI engineers.
• 1:13:17
  SPEAKER_02: Next thing I'm hearing right now is,
• 1:13:19
  SPEAKER_02: People who have like two to three years of AI engineering experience are so
• 1:13:23
  SPEAKER_02: in demand.
• 1:13:24
  SPEAKER_02: I'm doing research on a job market and they're like, this is the best job market ever.
• 1:13:29
  SPEAKER_02: However, you know, the people who are like, OK, I have none, but I want to get it.
• 1:13:33
  SPEAKER_02: They and let's say they're out of a job.
• 1:13:35
  SPEAKER_02: They're struggling because no one's giving them the benefit of a doubt.
• 1:13:38
  SPEAKER_00: It is really hard.
• 1:13:38
  SPEAKER_00: And I'm not saying it's right, but it's how it is.
• 1:13:41
  SPEAKER_02: And I guess
• 1:13:42
  SPEAKER_02: the reason we're ringing this alarm bell is we know this change has not been asked
• 1:13:45
  SPEAKER_02: fast.
• 1:13:46
  SPEAKER_02: So do it now because later... Do it now.
• 1:13:49
  SPEAKER_00: The next time you go out for a job interview, anyone, you're going to be asked.
• 1:13:52
  SPEAKER_00: And you're going to be filtered out if you don't have it.
• 1:13:55
  SPEAKER_00: And the delta between those who are just getting started
• 1:13:59
  SPEAKER_00: and those who've been doing it, it was here for a
• 1:14:02
  SPEAKER_02: little while.
• 1:14:02
  SPEAKER_00: It was very easy to get started.
• 1:14:04
  SPEAKER_00: No, it's here.
• 1:14:05
  SPEAKER_00: But it's opening, the longer it goes, the more, the harder it will be to catch up.
• 1:14:09
  SPEAKER_00: You just got to get, you just got to get some.
• 1:14:11
  SPEAKER_02: Let's talk about directors.
• 1:14:12
  SPEAKER_00: Yeah, directors are usually the ones who,
• 1:14:15
  SPEAKER_00: they have been in management for like 10 years, usually.
• 1:14:18
  SPEAKER_00: And there's a real feeling of fear, often, of like, God, tech has changed a lot in 10 years.
• 1:14:27
  SPEAKER_00: And...
• 1:14:28
  SPEAKER_00: And this is where I would say your body,
• 1:14:31
  SPEAKER_00: like the way we experience anxiety and
• 1:14:35
  SPEAKER_00: the way we experience excitement is physiologically almost the same.
• 1:14:39
  SPEAKER_00: Like I used to play piano, right?
• 1:14:41
  SPEAKER_00: And before a performance, I'd be like,
• 1:14:42
  SPEAKER_02: I'm excited.
• 1:14:43
  SPEAKER_00: I'm so excited to do this, you know, because I'm like trembling and sweat.
• 1:14:47
  SPEAKER_00: But like the difference is agency.
• 1:14:49
  SPEAKER_00: If you sit back and And wait for the water to come to you,
• 1:14:54
  SPEAKER_00: you're just going to be freaking out.
• 1:14:56
  SPEAKER_00: But if you run towards the waves, if you're like, just like run towards, try it.
• 1:15:02
  SPEAKER_00: You know,
• 1:15:03
  SPEAKER_00: if you have a job now and you're a director and you're afraid of it,
• 1:15:06
  SPEAKER_00: it's always seen as kind of noble.
• 1:15:08
  SPEAKER_00: When managers want to go back to being ICs, I think it's very well respected.
• 1:15:14
  SPEAKER_00: Own it.
• 1:15:16
  SPEAKER_00: Run towards the waves.
• 1:15:17
  SPEAKER_00: Own it.
• 1:15:18
  SPEAKER_00: Be part of the wave,
• 1:15:19
  SPEAKER_00: the frontier of people who are like,
• 1:15:21
  SPEAKER_00: I'm so, just tell yourself, it doesn't have to be true.
• 1:15:23
  SPEAKER_00: I'm so excited to be an IC again.
• 1:15:25
  SPEAKER_00: It's never been easier to go back and try.
• 1:15:27
  SPEAKER_00: I'm going to do it and I'm going to talk about my experience and tell everyone else about it.
• 1:15:32
  SPEAKER_00: Just, you got to own it.
• 1:15:34
  SPEAKER_00: Don't wait.
• 1:15:35
  SPEAKER_02: And then let's talk about junior engineers.
• 1:15:38
  SPEAKER_02: Obviously,
• 1:15:38
  SPEAKER_02: it's a harder time to get started as a junior,
• 1:15:41
  SPEAKER_02: but how do you think about the value that they bring?
• 1:15:44
  SPEAKER_00: The hardest thing about quantifying the value of junior engineers is
• 1:15:47
  SPEAKER_00: that we don't know how to quantify the value of any engineers.
• 1:15:50
  SPEAKER_00: It's so interesting because I feel like we're over here doing all this hand-wringing
• 1:15:55
  SPEAKER_00: about, will juniors be okay?
• 1:15:56
  SPEAKER_00: Will they ever learn the basics?
• 1:15:59
  SPEAKER_00: But like my friend Boris,
• 1:16:00
  SPEAKER_00: who has a new observability startup,
• 1:16:02
  SPEAKER_00: and he talks to these high school, college kids all the time.
• 1:16:05
  SPEAKER_00: He's like, they are cooking.
• 1:16:07
  SPEAKER_00: They are,
• 1:16:08
  SPEAKER_00: they don't know what the software development life cycle is,
• 1:16:10
  SPEAKER_00: but they are just like off to their, they are doing so much cool shit.
• 1:16:14
  SPEAKER_00: I believe that the kids are going to be okay.
• 1:16:15
  SPEAKER_00: We just have to hire them.
• 1:16:17
  SPEAKER_00: We just have to give them a shot.
• 1:16:19
  SPEAKER_00: They're going to come up with a lot of the conclusions and the ways and the hows
• 1:16:23
  SPEAKER_00: that are going to be things that we wouldn't have thought of.
• 1:16:26
  SPEAKER_00: But we just have to hire them.
• 1:16:28
  SPEAKER_00: We just have to be willing to give them a shot.
• 1:16:30
  SPEAKER_02: This week, NSF,
• 1:16:30
  SPEAKER_02: I've talked with a bunch of founders,
• 1:16:33
  SPEAKER_02: young startups,
• 1:16:34
  SPEAKER_02: and they've been telling me the stories of this open source contributor
• 1:16:37
  SPEAKER_02: who was outstanding, so they wanted to hire him or her.
• 1:16:41
  SPEAKER_02: Turns out it was a 17-year-old kid.
• 1:16:43
  SPEAKER_02: They still hire it, and now they tell me like, oh my gosh, the things they do.
• 1:16:46
  SPEAKER_02: So I think when you're saying the kids' kids are going to be fine,
• 1:16:48
  SPEAKER_02: just give them a shot, even if it's an internship.
• 1:16:53
  SPEAKER_02: Yes, totally.
• 1:16:57
  SPEAKER_02: Yeah.
• 1:16:59
  SPEAKER_02: And even if that person doesn't work out,
• 1:17:02
  SPEAKER_02: with an internship under their belt, so much better for everyone.
• 1:17:05
  SPEAKER_00: Totally, totally.
• 1:17:06
  SPEAKER_02: One question that came up when I asked that you're going to be
• 1:17:09
  SPEAKER_02: the show what I should ask, they said AI fatigue.
• 1:17:12
  SPEAKER_02: Someone asked,
• 1:17:13
  SPEAKER_02: can you please ask Charity as an engineer if I'm starting to get just really,
• 1:17:18
  SPEAKER_02: really drained of this?
• 1:17:19
  SPEAKER_02: Have you had this?
• 1:17:20
  SPEAKER_02: Do you see people having it?
• 1:17:22
  SPEAKER_02: And what is a good way to just deal with it?
• 1:17:25
  SPEAKER_02: We know it's here.
• 1:17:26
  SPEAKER_02: We know it's here to stay, but still.
• 1:17:27
  SPEAKER_00: I mean, my follow-up question would be like, which variety of AI fatigue is?
• 1:17:31
  SPEAKER_02: Okay, tell us a lot of varieties.
• 1:17:33
  SPEAKER_00: You know,
• 1:17:34
  SPEAKER_00: because for some people when they say AI fatigue,
• 1:17:37
  SPEAKER_00: they're talking about receiving swap.
• 1:17:41
  SPEAKER_00: Some people are talking about all the hype and the,
• 1:17:45
  SPEAKER_00: have you heard the phrase or the term doom trolling?
• 1:17:50
  SPEAKER_02: No.
• 1:17:51
  SPEAKER_00: Cal Newport is, I think, his name.
• 1:17:54
  SPEAKER_00: He's an AI researcher professor on the East Coast.
• 1:17:58
  SPEAKER_00: And it's his term for what the CEO of Anthropic and OpenAI keep doing about,
• 1:18:04
  SPEAKER_00: oh, my God,
• 1:18:05
  SPEAKER_02: this might be the end of blah, blah, blah.
• 1:18:07
  SPEAKER_00: And he's like, it's just doom trolling.
• 1:18:09
  SPEAKER_00: And they need to stop it because they're stressing everyone the fuck out.
• 1:18:13
  SPEAKER_00: And stop because it's just not responsible.
• 1:18:16
  SPEAKER_00: So, yeah, I think there's a lot of fatigue around that.
• 1:18:18
  SPEAKER_00: I think that a lot of people...
• 1:18:20
  SPEAKER_02: Their family members are afraid.
• 1:18:22
  SPEAKER_00: You know, it's just, it's always before the history of technology.
• 1:18:25
  SPEAKER_00: It's been something cool or fun or this will be the iPhone.
• 1:18:30
  SPEAKER_00: It'll make your life better.
• 1:18:30
  SPEAKER_00: And now it's just like fear.
• 1:18:33
  SPEAKER_00: It's pretty crappy.
• 1:18:35
  SPEAKER_00: So there's that.
• 1:18:36
  SPEAKER_00: There's the fatigue of, like,
• 1:18:38
  SPEAKER_00: I found myself being off social media because I'm just so tired of all of
• 1:18:42
  SPEAKER_00: the AI slop posts.
• 1:18:44
  SPEAKER_02: It's just like I'm not interested.
• 1:18:46
  SPEAKER_02: Yeah.
• 1:18:47
  SPEAKER_00: There are a lot of different varieties here and yes we are all feeling it.
• 1:18:51
  SPEAKER_00: So I guess I would repeat my call for us to remember that we are in control.
• 1:18:56
  SPEAKER_00: We are in charge.
• 1:18:58
  SPEAKER_00: I think the universal nature of the frustration means that this is a great time
• 1:19:02
  SPEAKER_00: to propose experiments where we take back control.
• 1:19:07
  SPEAKER_00: Maybe you and your team agree we don't actually want any more AI generated PR descriptions.
• 1:19:15
  SPEAKER_00: We don't None of us use AI on Wednesdays.
• 1:19:18
  SPEAKER_00: Maybe we take a week, you know, just like take control back, try something,
• 1:19:22
  SPEAKER_02: propose something.
• 1:19:22
  SPEAKER_02: I guess because change is so big, experimenting has never been easier.
• 1:19:27
  SPEAKER_02: And I guess most businesses,
• 1:19:28
  SPEAKER_02: most directors,
• 1:19:30
  SPEAKER_02: most leaders would welcome teams saying,
• 1:19:33
  SPEAKER_02: you know,
• 1:19:33
  SPEAKER_02: we're going to try out because their answer will probably be,
• 1:19:35
  SPEAKER_02: I mean, you're in this position.
• 1:19:36
  SPEAKER_02: Your answer, I guess, will be sure.
• 1:19:38
  SPEAKER_00: Better yet, don't even tell me.
• 1:19:40
  SPEAKER_00: Come and tell me what worked afterwards.
• 1:19:41
  SPEAKER_00: And what didn't.
• 1:19:42
  SPEAKER_02: And what you learned.
• 1:19:42
  SPEAKER_02: And what didn't.
• 1:19:43
  SPEAKER_00: I guess maybe.
• 1:19:58
  SPEAKER_02: Maybe we just forgot that there have been major changes in the industry.
• 1:20:03
  SPEAKER_02: I remember the iPhone change.
• 1:20:04
  SPEAKER_02: And I remember the people when the iPhone came out,
• 1:20:07
  SPEAKER_02: iPhone and Android,
• 1:20:08
  SPEAKER_02: the smartphones,
• 1:20:09
  SPEAKER_02: the people who were the most kick-ass iOS engineers, you know who they were?
• 1:20:15
  SPEAKER_02: There were typically like 18 or 19-year-old kids who went into this and they tried it out.
• 1:20:19
  SPEAKER_02: Guess what?
• 1:20:20
  SPEAKER_02: Two years later, they were the domain experts.
• 1:20:21
  SPEAKER_02: The staff engineer was a 22-year-old and then the entry-level engineer was a 40-year-old.
• 1:20:27
  SPEAKER_02: And again, not always.
• 1:20:28
  SPEAKER_02: But my point is when there's such big change,
• 1:20:31
  SPEAKER_02: you can actually become an expert by you taking charge.
• 1:20:38
  SPEAKER_02: And also no one's really going to tell you no because no one knows what's working.
• 1:20:42
  SPEAKER_00: Exactly.
• 1:20:42
  SPEAKER_00: There's some liberty there.
• 1:20:44
  SPEAKER_02: So as closing,
• 1:20:45
  SPEAKER_02: just to go back to a little bit of being human and slowing down,
• 1:20:49
  SPEAKER_02: what are one or two books that gave you something?
• 1:20:51
  SPEAKER_00: Ooh.
• 1:20:55
  SPEAKER_00: I really got a lot out of catastrophe ethics.
• 1:21:00
  SPEAKER_00: I haven't seen it mentioned in many places,
• 1:21:02
  SPEAKER_00: and I think it might be,
• 1:21:04
  SPEAKER_00: I think real philosophy nerds would be like,
• 1:21:06
  SPEAKER_00: eh, that's kind of a pop book, you know?
• 1:21:07
  SPEAKER_00: And I think the people who are not real philosophy books are like,
• 1:21:09
  SPEAKER_00: that's kind of a lot of philosophy.
• 1:21:11
  SPEAKER_00: But You know, he's a bioethicist, I think.
• 1:21:15
  SPEAKER_00: Travis Reeder,
• 1:21:16
  SPEAKER_00: our IED,
• 1:21:17
  SPEAKER_00: our Catastrophe Ethics,
• 1:21:18
  SPEAKER_00: and he talks about how the puzzle of modern life is
• 1:21:21
  SPEAKER_00: that it feels like everything we're implicated, every choice we make.
• 1:21:25
  SPEAKER_00: Are you going to use milk?
• 1:21:27
  SPEAKER_00: Well, you know, the cows were tortured.
• 1:21:28
  SPEAKER_00: Are you going to use almond milk?
• 1:21:29
  SPEAKER_00: Well, water is a problem.
• 1:21:30
  SPEAKER_00: Well, you use soy milk?
• 1:21:31
  SPEAKER_00: Like, well, hormones.
• 1:21:33
  SPEAKER_00: And it's just like there is no...
• 1:21:35
  SPEAKER_00: Whatever you do, you are hurting someone.
• 1:21:38
  SPEAKER_00: And it feels like the problems are so large that none of our decisions really matter.
• 1:21:45
  SPEAKER_00: And that tension,
• 1:21:46
  SPEAKER_00: like what,
• 1:21:47
  SPEAKER_00: and then he kind of walks through traditional ethical frameworks like utilitarianism
• 1:21:53
  SPEAKER_00: and stuff and just shows how there is no recipe anyone can follow
• 1:21:58
  SPEAKER_00: that doesn't lead you to some really stupid,
• 1:22:02
  SPEAKER_00: and he's like, this is just, No gods, no masters.
• 1:22:06
  SPEAKER_00: Which doesn't mean that everything's relative.
• 1:22:09
  SPEAKER_00: What it means is that the way to live an ethical life of integrity is you need
• 1:22:17
  SPEAKER_00: to educate yourself about the world.
• 1:22:19
  SPEAKER_00: You need to know things, right?
• 1:22:22
  SPEAKER_00: And then listen.
• 1:22:26
  SPEAKER_00: Where are you drawn?
• 1:22:27
  SPEAKER_00: What suffering really speaks to you?
• 1:22:29
  SPEAKER_00: What caused you?
• 1:22:31
  SPEAKER_00: Because no one can tell you what matters.
• 1:22:33
  SPEAKER_00: You have to decide what matters.
• 1:22:35
  SPEAKER_00: And so that introspection and...
• 1:22:40
  SPEAKER_00: It's so at odds with the sort of performative rage, you know, which I'm just so exhausted.
• 1:22:46
  SPEAKER_00: All right, so that's one.
• 1:22:47
  SPEAKER_00: Number two,
• 1:22:48
  SPEAKER_00: this is a book that I've recommended a couple times,
• 1:22:49
  SPEAKER_00: but I'm just going to keep recommending it because it's so good.
• 1:22:52
  SPEAKER_00: It's by Adam Becker, and it's called More Everything Forever.
• 1:22:55
  SPEAKER_00: And he is a journalist based in San Francisco.
• 1:23:00
  SPEAKER_00: He has a philosophy undergrad and a PhD in astrophysics,
• 1:23:03
  SPEAKER_00: and he just Demolishes all of the AI religion,
• 1:23:08
  SPEAKER_00: the singularity,
• 1:23:10
  SPEAKER_00: and the effect of altruism,
• 1:23:13
  SPEAKER_00: and accelerationism,
• 1:23:14
  SPEAKER_00: and the whole like, what if we could have infinite growth forever-ism?
• 1:23:18
  SPEAKER_00: And he's like, the heat death of the universe, you guys.
• 1:23:21
  SPEAKER_00: Literally the only thing we know about exponential growth is that it must end.
• 1:23:26
  SPEAKER_00: It must end in an S-curve or in a crash.
• 1:23:28
  SPEAKER_00: It must end.
• 1:23:29
  SPEAKER_00: And he's got this dry sense of humor And there are a couple times
• 1:23:34
  SPEAKER_00: where he's just like describing some of the very real things.
• 1:23:40
  SPEAKER_00: It's just like, why do Oxford ethicists want this?
• 1:23:44
  SPEAKER_00: He's talking about like taking over star systems.
• 1:23:47
  SPEAKER_00: It's just ridiculous.
• 1:23:49
  SPEAKER_00: And he also, he gets in a whack.
• 1:23:51
  SPEAKER_00: He's just like, Talks about these people who are working so hard on life extension.
• 1:23:55
  SPEAKER_00: And he's like, these are a bunch of sad little boys who miss their daddy.
• 1:24:00
  SPEAKER_00: And I was just like, oh, my God.
• 1:24:03
  SPEAKER_00: It is the oldest fear of humanity is a fear of death.
• 1:24:06
  SPEAKER_00: And you just see it.
• 1:24:07
  SPEAKER_00: You cannot see it.
• 1:24:08
  SPEAKER_00: So, yeah, those are my two.
• 1:24:09
  SPEAKER_00: They're both so good.
• 1:24:10
  SPEAKER_02: Charity, thank you so much.
• 1:24:12
  SPEAKER_02: This is finally made it happen.
• 1:24:13
  SPEAKER_00: Finally.
• 1:24:14
  SPEAKER_00: It's a good time.
• 1:24:16
  SPEAKER_01: I always really, really enjoy talking with Charity.
• 1:24:18
  SPEAKER_01: I hope we also liked it.
• 1:24:19
  SPEAKER_01: I appreciated how Charity talks about the trust account.
• 1:24:22
  SPEAKER_01: If we are debiting trust from the creation of code because AI wrote it
• 1:24:25
  SPEAKER_01: and no human read it, then that trust needs to be refilled somewhere else.
• 1:24:29
  SPEAKER_01: Testing evils and guardrails are all ways to add more trust that we lost by using AI.
• 1:24:34
  SPEAKER_01: I also appreciated how she talked with empathy about both AI camps.
• 1:24:37
  SPEAKER_01: The enthusiasts or AI-pailed folks are seeing the practical wins
• 1:24:41
  SPEAKER_01: while those operating production systems see the slop.
• 1:24:44
  SPEAKER_01: Neither side is wrong, but they should talk to each other more.
• 1:24:47
  SPEAKER_01: So if you see wins with AI,
• 1:24:48
  SPEAKER_01: share with the broader team,
• 1:24:49
  SPEAKER_01: but also talk about it when it creates more work,
• 1:24:51
  SPEAKER_01: reduces reliability, or when it degrades quality.
• 1:24:54
  SPEAKER_01: And for those of us feeling anxious about all of this change,
• 1:24:57
  SPEAKER_01: especially directors and managers, I'll leave you with charity's advice.
• 1:25:01
  SPEAKER_01: Anxiety and excitement are psychologically almost the same,
• 1:25:04
  SPEAKER_01: but a difference between them is agency.
• 1:25:07
  SPEAKER_01: So instead of waiting for change to come to you,
• 1:25:09
  SPEAKER_01: take charge however you can and make changes yourself.
• 1:25:12
  SPEAKER_01: Do check out the show notes below for related to Pragmatic Engineer deep dives
• 1:25:15
  SPEAKER_01: on how AI is changing software engineering and for another discussion with Charity
• 1:25:19
  SPEAKER_01: on observability.
• 1:25:20
  SPEAKER_01: And I can very much recommend her book, Observability Engineering, second edition.
• 1:25:23
  SPEAKER_01: If you enjoyed this podcast,
• 1:25:24
  SPEAKER_01: please do subscribe on your favorite podcast platform and on YouTube.
• 1:25:28
  SPEAKER_01: A special thank you if you also leave a rating on the show.
• 1:25:30
  SPEAKER_01: Thanks and see you in the next one.