Latest / The Pragmatic Engineer / Building resilient systems with Sam Newman

Transcript

Building resilient systems with Sam Newman · The Pragmatic Engineer

• 0:00
  SPEAKER_01: Sam Newman is the author who wrote probably the most widely read book on microservices.
• 0:04
  SPEAKER_01: And yet, as you heard, he said that microservices are an architecture of the last resort.
• 0:09
  SPEAKER_01: But why?
• 0:10
  SPEAKER_01: Sam was in the room when the term microservices was coined.
• 0:14
  SPEAKER_01: I find him to be one of the most thoughtful engineers on this topic,
• 0:17
  SPEAKER_01: as well as on resilient systems,
• 0:19
  SPEAKER_01: and his latest book, Building Resilient Distributed Systems, is out now.
• 0:22
  SPEAKER_01: Today, we cover specs versus code as a source of truth and why Sam says
• 0:27
  SPEAKER_01: that production is truth,
• 0:28
  SPEAKER_01: idempotency keys versus fingerprints,
• 0:30
  SPEAKER_01: and why you should avoid idempotency keys when possible.
• 0:33
  SPEAKER_01: The four dimensions of resilience,
• 0:35
  SPEAKER_01: robustness,
• 0:36
  SPEAKER_01: rebound,
• 0:36
  SPEAKER_01: graceful extensibility,
• 0:38
  SPEAKER_01: and sustained adaptability,
• 0:40
  SPEAKER_01: cognitive depth and cognitive surrender,
• 0:41
  SPEAKER_01: and Sam's advice on how to better use AI to avoid these and many more.
• 0:45
  SPEAKER_01: If you want to understand what it takes to build resilient systems,
• 0:48
  SPEAKER_01: from an expert who has been knee-deep in this topic for 10 plus years,
• 0:51
  SPEAKER_01: this episode is for you.
• 0:53
  SPEAKER_01: This episode is presented by TurboBuffer, the AI retrieval layer built on object storage.
• 0:58
  SPEAKER_01: It's fast, cheap, and extremely scalable.
• 1:00
  SPEAKER_01: Antropic, Cognition, Notion, and many others are happy customers.
• 1:03
  SPEAKER_01: Before we start,
• 1:04
  SPEAKER_01: let me mention Tyre,
• 1:05
  SPEAKER_01: our seasoned sponsor and our product and team I'm very bullish on.
• 1:08
  SPEAKER_01: You probably already know that Git hosting has a problem, AI agents.
• 1:12
  SPEAKER_01: Git is increasingly becoming a bottleneck for modern agent-heavy software development.
• 1:16
  SPEAKER_01: Engines like you or me are creating more code with agents,
• 1:18
  SPEAKER_01: those agents push more code,
• 1:20
  SPEAKER_01: and most of us are running parallel agents, which push way more code.
• 1:23
  SPEAKER_01: Github is clearly struggling to keep up and has frequent outages.
• 1:27
  SPEAKER_01: So what's the solution?
• 1:28
  SPEAKER_01: Entire was founded by Github's last CEO,
• 1:30
  SPEAKER_01: Tomasz Dymka, who, by the way, was a guest on a podcast when he was still at Github.
• 1:34
  SPEAKER_01: And Tomasz revealed Githosting for the agentic era from scratch.
• 1:37
  SPEAKER_01: Entire was built And this allows for fleets of agents to push in parallel.
• 1:45
  SPEAKER_01: And Entire's performance is next level.
• 1:47
  SPEAKER_01: The repo can handle 418 pushes per second,
• 1:50
  SPEAKER_01: which is up to 89 times faster than every competitor on the market.
• 1:54
  SPEAKER_01: When GitHub is down,
• 1:55
  SPEAKER_01: you can still keep working and you don't even need to migrate away from GitHub.
• 1:58
  SPEAKER_01: You just sign up to Entire and the platform mirrors your repo.
• 2:02
  SPEAKER_01: And here's a neat thing.
• 2:03
  SPEAKER_01: Have you ever wondered what prompt resulted in this specific code being generated?
• 2:07
  SPEAKER_01: I find that the prompt and conversation with the agent carries more information than
• 2:11
  SPEAKER_01: the PR itself oftentimes.
• 2:12
  SPEAKER_01: Entire captures all the prompt history with your agent right inside of the repo,
• 2:16
  SPEAKER_01: easy to check back.
• 2:18
  SPEAKER_01: And it's got a pretty innovative UI to show all of this.
• 2:20
  SPEAKER_01: If you're looking for Git hosting that works even when GitHub is down,
• 2:23
  SPEAKER_01: head to entire.io slash pragmatic,
• 2:26
  SPEAKER_01: install the CLI and mirror your repo with a click.
• 2:29
  SPEAKER_01: I've already done it and I love it.
• 2:31
  SPEAKER_01: Oh, and did I mention that it works with any agent and it's open source?
• 2:34
  SPEAKER_00: Sam, it's great to have you on the podcast.
• 2:36
  SPEAKER_00: Thank you so much for having me.
• 2:37
  SPEAKER_00: Happy to be here.
• 2:38
  SPEAKER_00: Really happy to be here.
• 2:39
  SPEAKER_00: Big fan of the show, as they say.
• 2:41
  SPEAKER_00: I'm not going to lie.
• 2:42
  SPEAKER_00: I've become a bit of a millennial and I think I've watched a lot of shorts rather
• 2:46
  SPEAKER_00: than watching the whole things now.
• 2:47
  SPEAKER_00: I'm becoming my son.
• 2:48
  SPEAKER_01: Yeah, millennial is a new boomer, apparently, I've learned.
• 2:52
  SPEAKER_00: I was happy enough being Gen Y.
• 2:53
  SPEAKER_00: Now I'm wherever I am now.
• 2:55
  SPEAKER_01: I'm quite fortunate.
• 2:56
  SPEAKER_01: Living in the UK...
• 3:11
  SPEAKER_00: We had at places an access to computers quite early on.
• 3:13
  SPEAKER_00: We had an Acorn Electron at home,
• 3:15
  SPEAKER_00: which was sort of one of the generations of the BBC computers that came after
• 3:19
  SPEAKER_00: the BBC B.
• 3:21
  SPEAKER_00: And then I got a Spectrum 128 3, which is the Spectrum that had the built-in tape drive.
• 3:27
  SPEAKER_01: This is way before the Commodores.
• 3:29
  SPEAKER_00: I think some of this was,
• 3:31
  SPEAKER_00: I think I'm pretty sure that the Acorn Electron was pre-Commodore,
• 3:35
  SPEAKER_00: but the Spectrum would have been around the same.
• 3:38
  SPEAKER_00: The UK was a bit more of a Spectrum.
• 3:41
  SPEAKER_00: And BBC Market, whereas the US, the Commodores and the Ataris and things were a bit bigger.
• 3:46
  SPEAKER_00: So that sort of got me a bit interested.
• 3:48
  SPEAKER_00: And then I always sort of stayed interested in it.
• 3:50
  SPEAKER_00: Dad had a computer.
• 3:51
  SPEAKER_00: I did the very boring thing.
• 3:52
  SPEAKER_00: I went to uni.
• 3:54
  SPEAKER_00: I did a software engineering degree at a place called De Montfort University,
• 3:58
  SPEAKER_00: which the running joke in my family is it's in the top five universities in
• 4:02
  SPEAKER_00: the Leicester area.
• 4:04
  SPEAKER_00: There are only two universities in Leicester.
• 4:06
  SPEAKER_00: I mean, I loved it, actually.
• 4:08
  SPEAKER_00: It was a great club.
• 4:09
  SPEAKER_00: It's basically an MIT syllabus.
• 4:11
  SPEAKER_00: And then I just did work after.
• 4:12
  SPEAKER_00: I got a great, back then we did these things called sandwich courses.
• 4:17
  SPEAKER_00: Sandwich course.
• 4:18
  SPEAKER_00: Yeah.
• 4:18
  SPEAKER_00: And so basically the idea was you would do two years of study.
• 4:21
  SPEAKER_00: You do a year in industry that was paid,
• 4:23
  SPEAKER_00: but you effectively going in as a student and then you did your final year.
• 4:28
  SPEAKER_00: Oh, nice.
• 4:28
  SPEAKER_00: Which is actually fantastic because I kind of was drifting a bit.
• 4:31
  SPEAKER_00: And when I did my year in industry, So many things clicked.
• 4:35
  SPEAKER_00: It meant there was much focus going in.
• 4:37
  SPEAKER_00: And I got to do work for a company called, back then they were called GC Alstom.
• 4:42
  SPEAKER_00: That was called Alstom.
• 4:44
  SPEAKER_00: And we did software for the European Space Agency and gas and steam turbines.
• 4:48
  SPEAKER_00: Yeah, it was really cool.
• 4:49
  SPEAKER_00: I joined about the first time at that, and then I went back after.
• 4:54
  SPEAKER_00: And when I went back, it was just around the time of the first dot-com boom.
• 4:58
  SPEAKER_00: And so everybody left.
• 4:59
  SPEAKER_00: So I went from being the person that was doing like, Nothing to the person who did everything.
• 5:03
  SPEAKER_00: But I remember my first week there, I was given a Fortran 77 codebase.
• 5:10
  SPEAKER_00: So Fortran 77 was released in the year I was born.
• 5:14
  SPEAKER_00: And every single for loop had a break condition
• 5:18
  SPEAKER_00: because we were basically doing analysis on these Unix machines.
• 5:21
  SPEAKER_00: If the models and the time on the machines was expensive,
• 5:25
  SPEAKER_00: And we were worried about accidental infinite loops.
• 5:28
  SPEAKER_00: This is how I was trying to solve the halting problem, basically.
• 5:30
  SPEAKER_00: So every for loop across these 8,000 files had a break condition of,
• 5:36
  SPEAKER_00: I think it was 99,999 loops.
• 5:38
  SPEAKER_00: And if it went over that, the program basically broke and aborted.
• 5:41
  SPEAKER_00: The models had got bigger,
• 5:43
  SPEAKER_00: the computer got more powerful, so it had to go through and change them more.
• 5:46
  SPEAKER_01: By hand back then?
• 5:47
  SPEAKER_00: By hand, well...
• 5:49
  SPEAKER_00: This is one of the nicest things that anyone's ever done to me in a professional career.
• 5:52
  SPEAKER_00: Because Fortran 77 is based on the old punch card limitations to an extent.
• 5:57
  SPEAKER_00: So your lines can't go very long.
• 6:00
  SPEAKER_00: So it's all a bit fiddly.
• 6:01
  SPEAKER_00: You can't just go and put a nine.
• 6:02
  SPEAKER_00: And my boss said, I reckon it would take about two weeks if you do it by hand.
• 6:07
  SPEAKER_00: Or, and he slid a book across to me, you can read this.
• 6:09
  SPEAKER_00: And he gave me a copy of the O'Reilly Said an Orc book.
• 6:13
  SPEAKER_00: And so I spent two days learning how to do regular expressions.
• 6:18
  SPEAKER_00: And I wrote a refactoring using regular expressions,
• 6:21
  SPEAKER_00: accounting for line lengths and everything else.
• 6:23
  SPEAKER_00: And it's, I think, the closest I've ever come to feeling like a god.
• 6:27
  SPEAKER_00: I did it and I thought, oh, that's the thrill.
• 6:29
  SPEAKER_00: That's the absolute thrill.
• 6:30
  SPEAKER_01: I'm not sure if it's a joke or a story, but, you know, the two, the two, Yeah.
• 6:55
  SPEAKER_00: But sometimes, sometimes, of course, the wall falls down the moment you said the machine.
• 6:58
  SPEAKER_00: The thing is, I love that.
• 7:00
  SPEAKER_00: That story is all true.
• 7:02
  SPEAKER_00: It's one of the earliest times I can remember learning something and having such
• 7:05
  SPEAKER_00: a visceral reaction.
• 7:07
  SPEAKER_00: And I remember just towards one of the end of the thought works,
• 7:10
  SPEAKER_00: there's some grads in and they're asked to tell some stories
• 7:13
  SPEAKER_00: about like what had happened in previous career.
• 7:15
  SPEAKER_00: And I told the story.
• 7:17
  SPEAKER_00: I said, I added the nine, made the models bigger.
• 7:19
  SPEAKER_00: And then one of the,
• 7:22
  SPEAKER_00: one of the grads said, why didn't you just, Replace it all with a constant.
• 7:27
  SPEAKER_00: So you could just change the number once any time in the future.
• 7:29
  SPEAKER_00: And I was like...
• 7:33
  SPEAKER_00: I thought, I'm learning once again.
• 7:35
  SPEAKER_01: Yeah.
• 7:36
  SPEAKER_00: So that's also why I love it.
• 7:37
  SPEAKER_00: It's like you...
• 7:38
  SPEAKER_00: You get humbled all the time in this industry.
• 7:40
  SPEAKER_00: You get to learn stuff.
• 7:42
  SPEAKER_00: Yes, there's reinvention of the wheel going on,
• 7:43
  SPEAKER_00: but we're inventing better wheels the whole time.
• 7:46
  SPEAKER_01: And I think it's just nice to remember this every now and
• 7:48
  SPEAKER_01: then I'm sure everyone who's built code and programs,
• 7:52
  SPEAKER_01: like at some point you have the realization that you do feel like a god,
• 7:54
  SPEAKER_01: that you can do things with these things,
• 7:56
  SPEAKER_01: which is just so difficult to do by hand, right?
• 7:59
  SPEAKER_01: Like there's I think we forget about that when it's kind of day to day
• 8:02
  SPEAKER_01: and we're just like doing our stuff or even like right now we're using LLMs,
• 8:05
  SPEAKER_01: which also do incredible magical stuff that would have been impossible.
• 8:09
  SPEAKER_01: But like, it's kind of cool that we are in charge.
• 8:11
  SPEAKER_01: We can do amazing stuff.
• 8:13
  SPEAKER_00: You get that little fizzy dopamine here.
• 8:16
  SPEAKER_00: And I think that's something we're struggling with a bit now is
• 8:18
  SPEAKER_00: as people are getting further away from the code,
• 8:19
  SPEAKER_00: they're trying to work out like,
• 8:21
  SPEAKER_00: how do I still get that kind of dopamine in the same way?
• 8:25
  SPEAKER_00: It's not as much an issue for me because I've Later on,
• 8:29
  SPEAKER_00: you know,
• 8:29
  SPEAKER_00: I've been doing this for almost 30 years now and I've already been mostly working at
• 8:34
  SPEAKER_00: the area of architecture and sort of architecture and infrastructure for a while.
• 8:38
  SPEAKER_00: So I feel a little bit more removed,
• 8:39
  SPEAKER_00: but it's, yeah,
• 8:41
  SPEAKER_00: talking to my son who's a programmer,
• 8:42
  SPEAKER_00: he's got into it around an interesting time right now.
• 8:46
  SPEAKER_01: And then you eventually made your way to ThoughtWorks where you spent a bunch of time.
• 8:50
  SPEAKER_01: What kinds of things did you do at ThoughtWorks and what are things that you learned there?
• 8:55
  SPEAKER_00: I did anything.
• 8:56
  SPEAKER_00: I mean, it was, I joined when there were probably just under 400 people left with about 4,000.
• 9:02
  SPEAKER_00: And I worked and, you know, got to go all over the place.
• 9:04
  SPEAKER_00: I joined at a good time because a lot of the, I mean, we were very much an XP shop.
• 9:09
  SPEAKER_00: And I think that the core of the business still kind of primarily is when I speak
• 9:14
  SPEAKER_00: to people there.
• 9:14
  SPEAKER_00: By the way, XP, like, was that influenced by Kent Beck's XP?
• 9:19
  SPEAKER_00: Yeah, no, that extreme programming is very much, it's Kent's extreme programming.
• 9:23
  SPEAKER_00: There was a bit of a London flavour.
• 9:26
  SPEAKER_00: So the kind of London School of Test-Driven Development and
• 9:29
  SPEAKER_00: the Extreme Tuesday Club, XTC, around here.
• 9:32
  SPEAKER_01: So what's the London flavour of XTC?
• 9:35
  SPEAKER_00: It's sort of like,
• 9:35
  SPEAKER_00: if you want to know,
• 9:36
  SPEAKER_00: you probably need to read Growing Object-Oriented Software Guided by Tests.
• 9:39
  SPEAKER_00: One of the worst titled books in existence written by a couple of friends of mine,
• 9:44
  SPEAKER_00: Steve Freeman and Nat Price.
• 9:46
  SPEAKER_00: And I think that was a lot.
• 9:47
  SPEAKER_00: There's a lot of people in the community around here that knew that.
• 9:50
  SPEAKER_00: It was a rich meetup scene.
• 9:51
  SPEAKER_00: And so I kind of got introduced to people through there and came in.
• 9:55
  SPEAKER_00: And I was, same time I was offered a job by somebody else.
• 9:59
  SPEAKER_00: It was a bank.
• 10:00
  SPEAKER_00: The bank could tell me exactly what I was going to be doing.
• 10:03
  SPEAKER_00: And ThoughtWorks,
• 10:04
  SPEAKER_00: basically,
• 10:05
  SPEAKER_00: the recruiter at the time is now one of my very good friends,
• 10:08
  SPEAKER_00: one of my oldest friends now.
• 10:09
  SPEAKER_00: She said to me, I don't really know what you'll be doing.
• 10:12
  SPEAKER_00: And I'm like, that sounds like the job for me.
• 10:13
  SPEAKER_00: I went for that one.
• 10:15
  SPEAKER_00: And this is at a time when like,
• 10:17
  SPEAKER_00: you know,
• 10:17
  SPEAKER_00: we talk about continuous integration,
• 10:19
  SPEAKER_00: but the tooling wasn't there and talks about TDD and the tooling wasn't there.
• 10:24
  SPEAKER_00: It was in, you know, it was around that sort of time that we started work.
• 10:27
  SPEAKER_00: I started building the built pipeline stuff in about 2004.
• 10:30
  SPEAKER_00: And then I worked, Dave Farley was my first tech lead.
• 10:35
  SPEAKER_00: Dave Farley?
• 10:37
  SPEAKER_00: Yeah, yeah.
• 10:38
  SPEAKER_00: He's my first tech lead at ThoughtWorks.
• 10:40
  SPEAKER_00: I was one of the people that interviewed Jez.
• 10:43
  SPEAKER_00: Humble to backfill for me on a project at AOL of all places.
• 10:48
  SPEAKER_00: And we needed a backfill because I'm sort of half a CIS admin,
• 10:51
  SPEAKER_00: half dev really is how I see myself.
• 10:53
  SPEAKER_00: So we needed someone to take over my job.
• 10:56
  SPEAKER_00: And so I interviewed Jez.
• 10:59
  SPEAKER_00: And so I was around when a lot of interesting things were happening.
• 11:01
  SPEAKER_00: So I was fortunate enough to be there while we were sort of kicking those ideas around.
• 11:06
  SPEAKER_00: And obviously people like Dave and Jez put those ideas together really, really well.
• 11:12
  SPEAKER_00: You're sort of surrounded by everyone talking and thinking about these ideas.
• 11:16
  SPEAKER_00: I think you got that.
• 11:16
  SPEAKER_00: I got the idea very early on that this stuff's all up for grabs.
• 11:19
  SPEAKER_00: These things aren't set in stone that we can push.
• 11:23
  SPEAKER_01: And when you're saying these things, like you mentioned continuous delivery, you mentioned what?
• 11:28
  SPEAKER_00: Test automation.
• 11:30
  SPEAKER_00: Build pipeline optimization, infrastructure automation.
• 11:34
  SPEAKER_00: I went over to the US in 2007,
• 11:35
  SPEAKER_00: actually to work in a big,
• 11:38
  SPEAKER_00: I've actually worked at Google for like a year and a half,
• 11:41
  SPEAKER_00: as a ThoughtWorks employee.
• 11:43
  SPEAKER_00: Wow, that's
• 11:44
  SPEAKER_01: quite rare.
• 11:44
  SPEAKER_00: It is.
• 11:45
  SPEAKER_00: We were there doing a project.
• 11:47
  SPEAKER_00: It's all been talked about publicly.
• 11:48
  SPEAKER_00: It's a great write-up by Mike Bland all about our experiences.
• 11:51
  SPEAKER_00: It was me,
• 11:52
  SPEAKER_00: a few of my Thoris colleagues,
• 11:54
  SPEAKER_00: some people from Industrial Logic and some people from Pivotal,
• 11:57
  SPEAKER_00: the old Pivotal,
• 11:59
  SPEAKER_01: pre-acquisition.
• 12:00
  SPEAKER_01: They were known for their really excellent engineering culture.
• 12:03
  SPEAKER_00: Back when it was just Robby's company or anything else.
• 12:05
  SPEAKER_00: And we were there to teach Googlers primarily how to write automated tests and how
• 12:10
  SPEAKER_00: to write code that was testable.
• 12:12
  SPEAKER_00: And so I got out there and I was talking to people out in the Valley and stuff.
• 12:14
  SPEAKER_00: It's like we were going
• 12:15
  SPEAKER_00: into these enterprise companies doing test automation using stuff like Selenium.
• 12:19
  SPEAKER_00: And you go around talking to people and it's like people didn't.
• 12:22
  SPEAKER_00: I remember having conversations with people like Yahoo and even big chunks of Google
• 12:27
  SPEAKER_00: who did not think you could do functional automated testing of like websites
• 12:32
  SPEAKER_00: and stuff.
• 12:32
  SPEAKER_01: I guess because they just never did it, right?
• 12:35
  SPEAKER_00: Yeah, absolutely.
• 12:35
  SPEAKER_00: And for me, I hate it whenever I haven't pushed myself or found a better way of doing something.
• 12:44
  SPEAKER_00: And I was lucky enough that early days in ThoughtWorks,
• 12:47
  SPEAKER_00: that was just a constant churn of ideas and reinvention.
• 12:51
  SPEAKER_00: I mean, goodness knows how many mocking frameworks and inversion of control things I was...
• 12:56
  SPEAKER_00: I built two different replacements for cruise control myself, right?
• 13:00
  SPEAKER_00: And so...
• 13:01
  SPEAKER_00: But like, you know, it was at that time I created a thing called DB Deploy.
• 13:06
  SPEAKER_00: My friends Nick Ashley and Graham Tactic,
• 13:08
  SPEAKER_00: which was like the first database refactoring tool and all these sorts of things,
• 13:11
  SPEAKER_00: because they didn't exist.
• 13:12
  SPEAKER_00: We needed them.
• 13:13
  SPEAKER_00: And it was like, if you want to do it, just do it.
• 13:15
  SPEAKER_00: That was their mindset.
• 13:16
  SPEAKER_00: And so I think there was always this idea working at ThoughtWorks was always like, what's next?
• 13:21
  SPEAKER_00: And I was quite fortunate that because ThoughtWorks was growing so much,
• 13:24
  SPEAKER_00: there was always things needed to be done.
• 13:27
  SPEAKER_00: And my default was normally just to say yes to everything.
• 13:29
  SPEAKER_00: So I got to travel all over the world.
• 13:30
  SPEAKER_00: It was fantastic.
• 13:31
  SPEAKER_00: Met my wife through ThoughtWorks.
• 13:32
  SPEAKER_00: Most of my best friends will come from ThoughtWorks now still to stay.
• 13:35
  SPEAKER_00: So although it's not been there for a long time,
• 13:37
  SPEAKER_00: it was kind of really formative, I think, in terms of that.
• 13:41
  SPEAKER_01: And it is really interesting because ThoughtWorks has a...
• 13:45
  SPEAKER_01: Alumni pipeline,
• 13:46
  SPEAKER_01: people who used to work there,
• 13:47
  SPEAKER_01: who are now really well known in the industry for their ideas,
• 13:51
  SPEAKER_01: for thinking differently.
• 13:53
  SPEAKER_01: Some of them educate people, others have created businesses to build software in better ways.
• 14:00
  SPEAKER_01: And it's just interesting because like usually I wouldn't think of,
• 14:06
  SPEAKER_01: I'll just like put it in a box,
• 14:07
  SPEAKER_01: like you know,
• 14:08
  SPEAKER_01: you have product companies like Google,
• 14:10
  SPEAKER_01: like Meta,
• 14:10
  SPEAKER_01: like they build their own products and you have like different tiers
• 14:13
  SPEAKER_01: of these companies.
• 14:14
  SPEAKER_01: Some can afford the best people.
• 14:15
  SPEAKER_01: And they have an excellent engineering culture.
• 14:17
  SPEAKER_01: Again, Google was known for this.
• 14:18
  SPEAKER_01: Meta to some extent.
• 14:19
  SPEAKER_01: And there are some companies maybe who are like lower end tier.
• 14:22
  SPEAKER_01: And then you have the so-called consultancies who are,
• 14:24
  SPEAKER_01: you know, they don't build their own stuff or they don't do too much of it.
• 14:27
  SPEAKER_01: They come in and they help companies build or you have a problem when you Hire
• 14:30
  SPEAKER_01: a group of experts and ThoughtWorks technically is in this group.
• 14:34
  SPEAKER_01: And usually you would not think that there's too much innovation going on in these places.
• 14:38
  SPEAKER_01: But with ThoughtWorks proves to me,
• 14:40
  SPEAKER_01: yourself and other people,
• 14:42
  SPEAKER_01: it's actually you can have like when you are the best in what you do globally,
• 14:45
  SPEAKER_01: you can actually have a lot of innovation happening there as well, right?
• 14:49
  SPEAKER_00: Yeah, and it was innovation with a different set of constraints.
• 14:52
  SPEAKER_00: I mean, I would say there aren't that many people that have come from ThoughtWorks
• 14:55
  SPEAKER_00: that were really great startup product people.
• 14:57
  SPEAKER_00: I mean, a lot of ThoughtWorkers make X and scale up people once you're all together.
• 15:00
  SPEAKER_00: There are actually very few real entrepreneurs that came out of the ThoughtWorks machine.
• 15:03
  SPEAKER_00: I know a few of them.
• 15:04
  SPEAKER_00: There aren't many of them.
• 15:05
  SPEAKER_00: But I think there was always the opportunity...
• 15:12
  SPEAKER_00: So much of our working conditions as a consultancy,
• 15:15
  SPEAKER_00: who mostly did co-source delivery at that time,
• 15:17
  SPEAKER_00: we were doing kind of co-source delivery, a bit of enablement through delivery.
• 15:21
  SPEAKER_00: Your working conditions are often based on what the client does.
• 15:25
  SPEAKER_00: And unless you're there explicitly to change how the client builds software,
• 15:29
  SPEAKER_00: so what you worked, what you got.
• 15:31
  SPEAKER_00: But that was self-perpetuating.
• 15:33
  SPEAKER_00: Because we were already used to sort of challenging and pushing and trying
• 15:35
  SPEAKER_00: to do new things, that spirit carried on.
• 15:40
  SPEAKER_00: We grew quite aggressively,
• 15:41
  SPEAKER_00: but not that aggressively,
• 15:43
  SPEAKER_00: rotating people through and that spirit kind of carried on.
• 15:46
  SPEAKER_00: I mean, look,
• 15:46
  SPEAKER_00: we had to start most engagements with convincing people why testing was a good,
• 15:50
  SPEAKER_00: why pairing was a good idea, let alone TDD.
• 15:52
  SPEAKER_01: Yeah, and I imagine how the conversation went,
• 15:54
  SPEAKER_01: like a manager going like, well, what would you pair?
• 15:56
  SPEAKER_01: Then I have like,
• 15:57
  SPEAKER_01: you know,
• 15:57
  SPEAKER_01: like eight engineers and now you're making it four,
• 16:00
  SPEAKER_01: which sounds like a terrible thing to do, right?
• 16:02
  SPEAKER_00: Which, and this is, I mean,
• 16:03
  SPEAKER_00: I had good fortune because of the people, because of the work at ThoughtWorks.
• 16:05
  SPEAKER_00: I got to know Martin Fowler quite well.
• 16:07
  SPEAKER_00: And, you know, he supported me a lot when I wrote my first book.
• 16:10
  SPEAKER_00: I still remember when he said, well, programming is not typing.
• 16:13
  SPEAKER_00: And fundamentally, it made it click for me was having those conversations with clients.
• 16:18
  SPEAKER_00: If the client thinks that programming is typing,
• 16:22
  SPEAKER_00: and that's why they don't want you to pair program,
• 16:25
  SPEAKER_00: well, then you've got a fundamental problem that needs to be dealt with.
• 16:27
  SPEAKER_00: And it's not a conversation about pairing.
• 16:29
  SPEAKER_00: The conversation is, what do you think programming actually is?
• 16:32
  SPEAKER_01: Well, but let's bring this a little bit forward because right now we're having
• 16:35
  SPEAKER_01: the same conversation around, right?
• 16:36
  SPEAKER_01: For a lot of CEOs,
• 16:38
  SPEAKER_01: even founders are thinking who have been building tech companies,
• 16:43
  SPEAKER_01: you know,
• 16:43
  SPEAKER_01: they all look at what is the most time consuming part of what we do
• 16:47
  SPEAKER_01: in building software.
• 16:49
  SPEAKER_01: 50% of the time has historically been writing the code.
• 16:52
  SPEAKER_01: So a lot of people think,
• 16:53
  SPEAKER_01: you know, like programming is code and LMs are really good at churning out code.
• 16:57
  SPEAKER_01: So what is programming?
• 16:58
  SPEAKER_00: Okay, there's programming,
• 17:00
  SPEAKER_00: I think it's Peter Knorr's article,
• 17:02
  SPEAKER_00: the guy who came up with Barker's Knorr form,
• 17:04
  SPEAKER_00: well he's Knorr in Barker's Knorr form,
• 17:07
  SPEAKER_00: he wrote his paper years ago,
• 17:08
  SPEAKER_00: which he won the Turing Award for,
• 17:10
  SPEAKER_00: and he sort of describes the program as being a theory that lives in the mind of
• 17:16
  SPEAKER_00: the developers.
• 17:18
  SPEAKER_00: That it's actually us together, working together collectively, who builds a program.
• 17:23
  SPEAKER_00: The fact that we happen to make that program work through code,
• 17:28
  SPEAKER_00: The code is really a side effect of how we work together.
• 17:32
  SPEAKER_00: And I think product development is just that extended, right?
• 17:36
  SPEAKER_00: What we are building is collectively, we are building a system together.
• 17:41
  SPEAKER_00: And the code is one of the mechanisms we use to deliver the system.
• 17:44
  SPEAKER_00: It's not the only thing.
• 17:46
  SPEAKER_00: So I think holistically, that's what it's all about.
• 17:49
  SPEAKER_00: It's just that we focus on the code because that's historically been where we sort
• 17:53
  SPEAKER_00: of spend the time.
• 17:54
  SPEAKER_00: But I mean, this is also a classic theory of constraint stuff, though, right?
• 17:58
  SPEAKER_00: If you speed one bit up, you get constraints elsewhere.
• 18:01
  SPEAKER_00: And it exposes friction points elsewhere.
• 18:03
  SPEAKER_00: I would also argue that a lot of the time coding was not the problem or not the bottleneck.
• 18:07
  SPEAKER_00: It was how the coding was done.
• 18:09
  SPEAKER_00: Delayed feedback cycles and things.
• 18:10
  SPEAKER_00: But I do come back to that.
• 18:12
  SPEAKER_00: Peter Law piece again,
• 18:13
  SPEAKER_00: because it's like Margaret Ann Story referenced it recently in her term,
• 18:16
  SPEAKER_00: pay for uncalculative debt.
• 18:17
  SPEAKER_00: But it's like building software is a collective, it's a collective endeavor.
• 18:22
  SPEAKER_00: It's people working together who build software.
• 18:26
  SPEAKER_00: Kent put it in his tidy first book that software design is an exercise
• 18:30
  SPEAKER_00: in human relationships, right?
• 18:32
  SPEAKER_00: AI doesn't change that.
• 18:34
  SPEAKER_00: And I think all that's really happened over, say,
• 18:36
  SPEAKER_00: the last 15 years and my experience is that We've got a bit better at understanding
• 18:41
  SPEAKER_00: that we need better working relationships like product owners, for example.
• 18:44
  SPEAKER_00: But that's what the programming is.
• 18:46
  SPEAKER_00: The art of creating, it's the collective art of creating software.
• 18:49
  SPEAKER_00: And it's not just developers that do that.
• 18:51
  SPEAKER_00: Never has been.
• 18:51
  SPEAKER_00: And so after ThoughtWorks, you've You've gone independent, right?
• 18:55
  SPEAKER_00: I did three startups.
• 18:56
  SPEAKER_00: None of them worked out for different reasons.
• 18:58
  SPEAKER_00: So don't ask me for advice on how to make a successful startup.
• 19:02
  SPEAKER_00: And then I went independent.
• 19:03
  SPEAKER_00: Yeah, moved back.
• 19:03
  SPEAKER_00: I was living in Australia at the time and moved back from Australia.
• 19:06
  SPEAKER_00: And yeah, I've been independent ever since.
• 19:08
  SPEAKER_00: So doing kind of a mix of stuff really ever since.
• 19:10
  SPEAKER_00: I'd written the first book about microservices in my spare time at ThoughtWorks.
• 19:15
  SPEAKER_00: And then that book came out and I'd been at ThoughtWorks a long time.
• 19:20
  SPEAKER_00: And I realized that there's always a danger of becoming institutionalized.
• 19:24
  SPEAKER_00: You get to a point where the company is like, I think there's things as companies do to change.
• 19:29
  SPEAKER_00: And I've tried changing those things and they don't want me to change them.
• 19:33
  SPEAKER_00: So either I'm bad at changing them,
• 19:35
  SPEAKER_00: which is probably true,
• 19:36
  SPEAKER_00: or they don't think it needs to be changed, which is also probably true.
• 19:38
  SPEAKER_00: It's like, I could hang around and get bitter.
• 19:40
  SPEAKER_00: I'll just go now.
• 19:41
  SPEAKER_00: I'll just go now.
• 19:42
  SPEAKER_00: So I went off and I thought, I went independent.
• 19:44
  SPEAKER_00: It's been really interesting ever since because it's sort of,
• 19:46
  SPEAKER_00: on one level, the shackles are off.
• 19:50
  SPEAKER_00: The problem of working for yourself, though, is that the boss is frequently an arsehole.
• 19:54
  SPEAKER_00: So, current Sam frequently is cursing past Sam for the commitments he's made.
• 20:02
  SPEAKER_01: That's one lesson, I guess.
• 20:04
  SPEAKER_00: Yeah, yeah, yeah, yeah.
• 20:05
  SPEAKER_00: I mean, as I say, I'm almost 15.
• 20:07
  SPEAKER_00: I'm still not learning the lessons of that, right?
• 20:08
  SPEAKER_00: And I've made those mistakes for a few years now.
• 20:11
  SPEAKER_01: So I want to talk about microservices.
• 20:14
  SPEAKER_01: You have written the book Building Microservices in 2015.
• 20:17
  SPEAKER_01: You started to write it in 2014.
• 20:19
  SPEAKER_01: And as reference, Uber started to talk about how they have thousands of microservices in 2015.
• 20:24
  SPEAKER_01: So you were kind of writing this book right as the concept
• 20:27
  SPEAKER_01: of microservices was starting to go mainstream across startups.
• 20:31
  SPEAKER_01: Do you remember how and when microservices took off and where was it even coined?
• 20:35
  SPEAKER_01: Was it around Martin Fowler?
• 20:37
  SPEAKER_00: No, it was James Lewis.
• 20:38
  SPEAKER_00: So it was going back in the day.
• 20:40
  SPEAKER_00: So you got to remember, again, it's like something in the water.
• 20:43
  SPEAKER_00: It's like we have a bushfire.
• 20:44
  SPEAKER_00: There was a lot of buildup before the bushfire.
• 20:46
  SPEAKER_00: And so this would have been 2011 to 2012.
• 20:51
  SPEAKER_00: James was talking to a bunch of different companies
• 20:56
  SPEAKER_00: who were building what would have classed back then
• 20:58
  SPEAKER_00: as service-oriented architectures.
• 21:00
  SPEAKER_00: And earlier,
• 21:01
  SPEAKER_00: there'd been this kind of what we refer to as the guerrilla SOA movement,
• 21:04
  SPEAKER_00: which is an aim to simplify service-oriented architecture.
• 21:08
  SPEAKER_00: You know,
• 21:08
  SPEAKER_00: shift over to REST,
• 21:09
  SPEAKER_00: work done by people like...
• 21:10
  SPEAKER_01: Yeah, the original service-oriented architecture had like this XML format.
• 21:15
  SPEAKER_01: What was the name?
• 21:15
  SPEAKER_00: Oh, WSDL and Simple Object Access Protocol.
• 21:18
  SPEAKER_00: Never has the word simple been less well used.
• 21:21
  SPEAKER_01: Soap.
• 21:22
  SPEAKER_00: Soap.
• 21:23
  SPEAKER_01: So, yeah, let's start with simple.
• 21:25
  SPEAKER_00: It was the most complex thing ever.
• 21:26
  SPEAKER_00: Exactly.
• 21:27
  SPEAKER_00: So you had this whole heavyweight and then people like Ian Webber and Ian Robinson
• 21:32
  SPEAKER_00: and Jim Webber had like been pushing centralized.
• 21:34
  SPEAKER_00: They had mantras like make the endpoints smart and the pipes dumb
• 21:37
  SPEAKER_00: to keep things simple, put all the smarts in it.
• 21:39
  SPEAKER_00: So this is all the background.
• 21:40
  SPEAKER_00: We were also at this time doing a lot of work around like I was running
• 21:44
  SPEAKER_00: the continuous delivery practice for Europe at the time.
• 21:47
  SPEAKER_00: And so a lot of my engagements were like optimizing the final mile.
• 21:51
  SPEAKER_00: So you've written the code and it's like,
• 21:53
  SPEAKER_00: how do I get it here into production as quick and effectively as possible?
• 21:57
  SPEAKER_00: So basically taking the idea as a continuous delivery.
• 22:00
  SPEAKER_00: So I was around when Jez and Dave started working on those in 2006.
• 22:03
  SPEAKER_00: I'd had the conversations with them.
• 22:05
  SPEAKER_00: So that was all happening.
• 22:07
  SPEAKER_00: James was talking to companies that were creating service-oriented architectures
• 22:11
  SPEAKER_00: where they could replace the services quite quickly.
• 22:14
  SPEAKER_00: And so basically you'd look at it and say it's a finer grained form of SOA.
• 22:17
  SPEAKER_00: So that was all happening.
• 22:18
  SPEAKER_00: So James went,
• 22:19
  SPEAKER_00: we had this architectural symposium up in the Lake District
• 22:23
  SPEAKER_00: and James pitched this idea.
• 22:25
  SPEAKER_00: He was called the micro apps at the time in this room.
• 22:28
  SPEAKER_00: And somebody else in the room said, they're more like services.
• 22:31
  SPEAKER_00: Don't remember who said it.
• 22:32
  SPEAKER_00: There was about 10 of us in the room.
• 22:34
  SPEAKER_00: Most of us were hungover.
• 22:35
  SPEAKER_00: And then James and Ian, so James and Martin wrote the first post.
• 22:40
  SPEAKER_00: So the first paper on Martin's website was written by,
• 22:43
  SPEAKER_00: it was sort of James's original kind of discovery of this pattern
• 22:46
  SPEAKER_00: that he called microservices.
• 22:48
  SPEAKER_00: At the same time, I was doing a lot of stuff around infrastructure automation.
• 22:51
  SPEAKER_00: And so I was coming at it from a different lens.
• 22:53
  SPEAKER_00: So I think James was coming at it more from an architectural lens and I was coming
• 22:56
  SPEAKER_00: into it from a continuous delivery lens.
• 22:59
  SPEAKER_00: I spent at this point almost 10 years working Would that be my focus?
• 23:04
  SPEAKER_00: And so I was starting to see the architectural impediment.
• 23:09
  SPEAKER_00: So if I want to change things frequently,
• 23:12
  SPEAKER_00: but the things I want to change are integrated with other things.
• 23:15
  SPEAKER_01: Like a monolith.
• 23:17
  SPEAKER_01: Now that's how we would describe it.
• 23:18
  SPEAKER_00: Exactly right.
• 23:19
  SPEAKER_00: I mean, I did a talk called Designing for Rapid Release in 2012, which is sort of touching.
• 23:23
  SPEAKER_00: So basically James had this sort of, had put a name to this stuff and put some patterns on it.
• 23:27
  SPEAKER_00: And I sort of thought,
• 23:28
  SPEAKER_00: Hang on a minute,
• 23:29
  SPEAKER_00: that stuff I was talking about over here seems, these things seem quite sensible.
• 23:32
  SPEAKER_00: And I'd always wanted to write a book.
• 23:34
  SPEAKER_00: I mean, honestly, I wanted to write an animal book.
• 23:36
  SPEAKER_00: I wanted to write an O'Reilly book because that said an orc book I was given.
• 23:40
  SPEAKER_00: And so I just started writing.
• 23:42
  SPEAKER_00: I've always found writing quite, I'm quite lucky that I find writing enjoyable and quite easy.
• 23:47
  SPEAKER_00: So I just, yeah, sat down and wrote the book and that's it.
• 23:50
  SPEAKER_00: But for me, it was an extension of continuous delivery.
• 23:54
  SPEAKER_00: And it was a style of architecture that was only possible Because
• 23:59
  SPEAKER_00: of continuous living, because of DevOps.
• 24:01
  SPEAKER_00: We'd had these ideas before.
• 24:03
  SPEAKER_00: I mean, Erlang is a representation of these ideas,
• 24:06
  SPEAKER_00: but making it real, practical and attractable thing.
• 24:12
  SPEAKER_00: But I mean,
• 24:12
  SPEAKER_00: really fundamentally,
• 24:13
  SPEAKER_00: it's just the whole thing has been an exercise in smuggling through concepts of
• 24:18
  SPEAKER_00: the 1970s, which is fine by me, right?
• 24:21
  SPEAKER_00: If I get people to pay attention to information,
• 24:22
  SPEAKER_00: hiding and coupling and cohesion, that's all right.
• 24:26
  SPEAKER_01: But then as you said,
• 24:27
  SPEAKER_01: like this was the spark before the bushfire and then
• 24:29
  SPEAKER_01: the bushfire did start like suddenly again,
• 24:32
  SPEAKER_01: I think Uber was a very notable company starting to talk in public
• 24:35
  SPEAKER_01: about having thousands of microservices on conferences like InfoQ
• 24:39
  SPEAKER_01: and it got people's attention.
• 24:40
  SPEAKER_01: And then it kind of seemed that overnight,
• 24:42
  SPEAKER_01: a lot of startups started to say like, you know what, let's just do microservices.
• 24:46
  SPEAKER_00: There was a bit of rebadging going on.
• 24:47
  SPEAKER_00: I mean,
• 24:47
  SPEAKER_00: I was lucky when I wrote the first book that I met Ben Christensen fairly early on
• 24:52
  SPEAKER_00: through the Yale conference.
• 24:54
  SPEAKER_00: Ben was looking after performance and sort of reliability at Netflix.
• 24:59
  SPEAKER_00: And I also knew people at REA.
• 25:01
  SPEAKER_00: And these were two early REA, realestate.com.au in Australia.
• 25:05
  SPEAKER_00: And these were companies that were very much pushing the envelope in different ways.
• 25:09
  SPEAKER_00: And so I had two very good real world case studies to draw upon for my book.
• 25:13
  SPEAKER_00: But most other people weren't talking as publicly around that.
• 25:17
  SPEAKER_00: And also,
• 25:18
  SPEAKER_00: before Netflix referred to their architecture as microservices,
• 25:21
  SPEAKER_00: they just referred to it as fine-grained SOA.
• 25:23
  SPEAKER_00: And to an extent, you know, that's what happened at Uber and elsewhere.
• 25:27
  SPEAKER_00: We didn't have a different word for it.
• 25:29
  SPEAKER_00: Then, of course,
• 25:29
  SPEAKER_00: what happened was that everyone just called their SOA architecture as microservices,
• 25:33
  SPEAKER_00: which is sort of, you know, semantic diffusion is a thing, is to be expected.
• 25:40
  SPEAKER_00: So I just kept fighting the good fight for like,
• 25:43
  SPEAKER_00: It's just independent deployability and,
• 25:45
  SPEAKER_00: you know, I've always positioned it as it's an opinionated style of SOA.
• 25:51
  SPEAKER_00: You may not agree with the opinions, that's fine, do something else then.
• 25:53
  SPEAKER_01: But there are some opinions.
• 25:55
  SPEAKER_01: So then what is the difference between microservices and, what was it, our service?
• 25:58
  SPEAKER_01: Service-oriented architecture.
• 25:59
  SPEAKER_01: Service-oriented architecture.
• 26:00
  SPEAKER_00: What are the differences?
• 26:03
  SPEAKER_01: In your very strict... Now,
• 26:05
  SPEAKER_00: to be clear, right,
• 26:06
  SPEAKER_00: I have my definition,
• 26:07
  SPEAKER_00: you might have your own definition,
• 26:08
  SPEAKER_00: but I'm The first and most important is that your microservices need
• 26:10
  SPEAKER_00: to be independently deployable.
• 26:10
  SPEAKER_00: So your unit of deployment, unit of change needs to be a single microservice boundary.
• 26:30
  SPEAKER_00: And, you know, rather than I'm deploying everything together, that's a monolithic application.
• 26:34
  SPEAKER_00: Three services at the same time, et cetera.
• 26:36
  SPEAKER_00: Yeah, even a mini monolith, right?
• 26:37
  SPEAKER_00: And it's a single service at a time.
• 26:39
  SPEAKER_00: That's clear.
• 26:39
  SPEAKER_00: That's it.
• 26:40
  SPEAKER_00: And the second one,
• 26:41
  SPEAKER_00: which I think there are going to be exceptions to,
• 26:43
  SPEAKER_00: is that your kind of default way
• 26:45
  SPEAKER_00: for defining your service boundaries is around your business domain.
• 26:48
  SPEAKER_00: This is taking the concepts of,
• 26:50
  SPEAKER_00: you can use the management design,
• 26:52
  SPEAKER_00: you don't have to,
• 26:53
  SPEAKER_00: but rather than a lot of SOA architectures were dominated to an extent
• 26:57
  SPEAKER_00: by technical decomposition.
• 26:59
  SPEAKER_00: The classical N-tier architectures,
• 27:02
  SPEAKER_00: layered architectures,
• 27:03
  SPEAKER_00: where you've picked your service decomposition based on the nature of tech,
• 27:05
  SPEAKER_00: rather than thinking about the business functionality.
• 27:08
  SPEAKER_00: That's a bit softer.
• 27:09
  SPEAKER_00: Independent deployability is the main thing though.
• 27:12
  SPEAKER_01: Okay, well, I'm not thinking back at Uber.
• 27:13
  SPEAKER_01: I was at Uber at the time when your book was starting to become really popular.
• 27:18
  SPEAKER_01: I was there from 2016, and we had independently deployable services.
• 27:23
  SPEAKER_01: We didn't really think too much about the businesses or domain.
• 27:26
  SPEAKER_01: They had obviously their interfaces.
• 27:28
  SPEAKER_01: They had contracts that they offered,
• 27:30
  SPEAKER_01: increasingly strongly typed, and we used the ways to do it.
• 27:35
  SPEAKER_01: But it was just more like, I mean,
• 27:37
  SPEAKER_01: as you can imagine,
• 27:37
  SPEAKER_01: there was just There's devs,
• 27:38
  SPEAKER_01: oftentimes not very experienced devs,
• 27:40
  SPEAKER_01: who are like, all right, let's build a new service.
• 27:42
  SPEAKER_01: Here's the interface.
• 27:43
  SPEAKER_01: And then anyone can use it,
• 27:45
  SPEAKER_01: which actually,
• 27:46
  SPEAKER_01: there were these things where a dev or even a not very experienced dev deployed
• 27:52
  SPEAKER_01: a microservice.
• 27:52
  SPEAKER_01: I mean, there was always, I mean,
• 27:53
  SPEAKER_01: also in those sorts of environments,
• 27:55
  SPEAKER_01: when I was talking to people at SoundCloud, it's similar about that.
• 28:12
  SPEAKER_00: And this is also like about corporate culture and like what was acceptable.
• 28:16
  SPEAKER_00: Like in that kind of environment, just go and do stuff.
• 28:18
  SPEAKER_00: It's absolutely fine.
• 28:19
  SPEAKER_01: And I think that's why Uber,
• 28:21
  SPEAKER_01: for example,
• 28:22
  SPEAKER_01: when did I just talk with Twan Palm,
• 28:25
  SPEAKER_01: the C2 at the time, about like what was the real deal about microservices?
• 28:28
  SPEAKER_01: And he said Uber,
• 28:29
  SPEAKER_01: like he never really wanted that many services,
• 28:31
  SPEAKER_01: but it was just more of a constraint because Uber had a monolith.
• 28:34
  SPEAKER_01: The story was like,
• 28:35
  SPEAKER_01: which he told in the podcast,
• 28:37
  SPEAKER_01: we'll link it in the show notes below,
• 28:38
  SPEAKER_01: that Uber had this big monolith and it was really slow.
• 28:41
  SPEAKER_01: It was slowing delivery down and they needed to move fast because all
• 28:45
  SPEAKER_01: the other competitors were out there.
• 28:47
  SPEAKER_01: So they said,
• 28:47
  SPEAKER_01: all right,
• 28:48
  SPEAKER_01: like they wanted to get rid of the monolith and put together smaller services,
• 28:51
  SPEAKER_01: but they knew unless they had a strong mandate that you cannot add new stuff to it,
• 28:55
  SPEAKER_01: this will never happen.
• 28:56
  SPEAKER_01: So they said the mandate was everything new needs to be your own service.
• 28:59
  SPEAKER_01: You take care of it.
• 29:00
  SPEAKER_01: We don't care.
• 29:01
  SPEAKER_01: And suddenly so many independent services were born, which was not the goal.
• 29:05
  SPEAKER_01: And years after the fact that when Uber decomposed the monolith,
• 29:08
  SPEAKER_01: they're now looking at maybe sensibly merging based on business domain, etc.
• 29:13
  SPEAKER_01: But this story wasn't really told at the time.
• 29:15
  SPEAKER_01: It was just, oh, look,
• 29:16
  SPEAKER_01: like Uber has thousands of microservices, almost one or two per engineer, right?
• 29:19
  SPEAKER_01: And this is what the world saw.
• 29:21
  SPEAKER_00: I mean, it's also though,
• 29:22
  SPEAKER_00: but this is the reality of microservice adoption is like,
• 29:25
  SPEAKER_00: it's just a taller toolbox.
• 29:27
  SPEAKER_00: I mean, I've described microservices as being an architecture of last resort.
• 29:31
  SPEAKER_00: Really?
• 29:31
  SPEAKER_00: Yeah, absolutely.
• 29:32
  SPEAKER_00: Because it pushes you quite naturally to a more aggressive distribution
• 29:36
  SPEAKER_00: of your state and everything else.
• 29:40
  SPEAKER_00: And you have to have a good reason.
• 29:42
  SPEAKER_00: One of the best reasons to use microservices is if what you actually want is
• 29:46
  SPEAKER_00: an organization with a high degree of autonomy.
• 29:49
  SPEAKER_00: If what you want is teams who are able to act independently from each other,
• 29:55
  SPEAKER_00: The microservice architecture is a fantastically good fit,
• 29:58
  SPEAKER_00: but then really it's a means to an end, right?
• 30:00
  SPEAKER_00: So that's often what a lot of people get wrong,
• 30:02
  SPEAKER_00: I think,
• 30:03
  SPEAKER_00: is they adopt microservices and they think this other magic stuff's going to happen.
• 30:06
  SPEAKER_00: If you want organizational autonomy, that's what you're working towards.
• 30:10
  SPEAKER_00: And this is just one of the mechanisms to get you there.
• 30:13
  SPEAKER_00: And most of the enterprise organizations,
• 30:15
  SPEAKER_00: the vast majority of ones I've worked with in this space, that's what they're after.
• 30:19
  SPEAKER_00: They've got huge numbers of developers, lots of middle management.
• 30:23
  SPEAKER_00: They're not getting much work done.
• 30:24
  SPEAKER_00: They're thinking, what do we do?
• 30:26
  SPEAKER_00: Organizational autonomy becomes, you know, the solution to all problems.
• 30:30
  SPEAKER_00: Again, I don't always get to have to say whether or not it's a good idea.
• 30:33
  SPEAKER_00: And I think that's where microservices can be especially useful.
• 30:36
  SPEAKER_00: I mean, there are other exceptions as well.
• 30:38
  SPEAKER_00: And so an organization like, you know, Uber...
• 30:41
  SPEAKER_00: I mean, I know at Netflix as well,
• 30:43
  SPEAKER_00: things like domain-driven design weren't talked about at Netflix,
• 30:46
  SPEAKER_00: but the teams were already fairly well scoped around there as a product already.
• 30:50
  SPEAKER_00: And so you had those natural high-level boundaries around the business domain.
• 30:53
  SPEAKER_00: So those things flowed naturally.
• 30:55
  SPEAKER_00: And so that for me was a logical kind of stepping stone.
• 30:59
  SPEAKER_00: You also have the difference,
• 31:00
  SPEAKER_00: and this is something I realized quite early on,
• 31:02
  SPEAKER_00: When you go and do the work in the dot-coms versus the enterprise organizations.
• 31:06
  SPEAKER_01: When you say dot-coms, you mean the digital companies?
• 31:09
  SPEAKER_01: Tech companies.
• 31:09
  SPEAKER_00: Yeah, tech companies.
• 31:11
  SPEAKER_00: I'm old enough that I still talk about dot-coms.
• 31:13
  SPEAKER_00: But the tech companies, I mean,
• 31:14
  SPEAKER_00: fundamentally,
• 31:15
  SPEAKER_00: algorithm,
• 31:15
  SPEAKER_00: you know,
• 31:16
  SPEAKER_00: from a domain company,
• 31:18
  SPEAKER_00: The domains are very, very simple compared to enterprise organizations by and large.
• 31:22
  SPEAKER_00: The domains in there are much,
• 31:23
  SPEAKER_00: much more complicated because they're based on business processes that have existed
• 31:26
  SPEAKER_00: for 10, 20, 30, 40, 50 years.
• 31:28
  SPEAKER_01: So what you mean in a lot of tech companies,
• 31:30
  SPEAKER_01: a product is pretty much like, here's our product, here's how Oh, it does.
• 31:47
  SPEAKER_01: And it's clear where one ends versus where the other begins.
• 31:52
  SPEAKER_01: And then you do some hacks when you need to launch Uber Eats really quickly
• 31:54
  SPEAKER_01: and you build it on the right infrastructure
• 31:56
  SPEAKER_01: and Uber Eats order has like two trips attached to it because anyway,
• 32:01
  SPEAKER_01: but as you say, like early on, it starts very simple.
• 32:04
  SPEAKER_00: But then also in those environments,
• 32:05
  SPEAKER_00: your business processes and the digital implementation
• 32:08
  SPEAKER_00: of those business processes is evolving together.
• 32:10
  SPEAKER_00: And a lot of enterprise organizations didn't have that.
• 32:13
  SPEAKER_00: I mean, their first generation of digital products were codifications of paper-based processes.
• 32:18
  SPEAKER_01: Oh, so you're talking about These companies before digital,
• 32:21
  SPEAKER_01: let's say, Adidas or let's say a shoe company.
• 32:24
  SPEAKER_01: Virtually all banks.
• 32:26
  SPEAKER_01: Virtually all banks.
• 32:27
  SPEAKER_01: And now they're like,
• 32:28
  SPEAKER_01: we have all these processes and now we need to build a software
• 32:30
  SPEAKER_01: that mimics what we do, which of course is... That's how you
• 32:33
  SPEAKER_00: start off.
• 32:33
  SPEAKER_00: I mean, that's what the whole Domain Design book came out in 2004.
• 32:39
  SPEAKER_00: And it was a reflection of the fact that the vast majority
• 32:41
  SPEAKER_00: of software development was the process of us trying
• 32:44
  SPEAKER_00: to model existing non-digital processes and how you do that.
• 32:49
  SPEAKER_01: I wonder if we will have a different but similar problem of the same thing now
• 32:54
  SPEAKER_01: that we know AI tools are here.
• 32:57
  SPEAKER_01: We know that, for example, they're great at generating code.
• 33:00
  SPEAKER_01: And we have two types of companies.
• 33:02
  SPEAKER_01: We have a company, a digital company that has existed pre-AI.
• 33:04
  SPEAKER_01: AI, they're used to writing product programming documents,
• 33:07
  SPEAKER_01: deploying developers,
• 33:09
  SPEAKER_01: product managers,
• 33:09
  SPEAKER_01: designers,
• 33:10
  SPEAKER_01: QA, they all have their roles and it works like a really well-oiled machine.
• 33:14
  SPEAKER_01: And now you have these so-called AI native startups who just, again,
• 33:17
  SPEAKER_01: they have a few people,
• 33:18
  SPEAKER_01: they're just, from the start, they're using agents for everything.
• 33:21
  SPEAKER_01: They have a very different process.
• 33:23
  SPEAKER_01: And at some point when this whole thing will mature and we will have best practices,
• 33:26
  SPEAKER_01: we'll have the existing companies who,
• 33:27
  SPEAKER_01: like the previous one,
• 33:29
  SPEAKER_01: they will try to retrofit this tooling to It's already happening.
• 33:54
  SPEAKER_00: Software is all about abstractions, right?
• 33:55
  SPEAKER_00: But, you know,
• 33:56
  SPEAKER_00: we went from doing machine code to doing assembly,
• 33:59
  SPEAKER_00: went from assembly to general purpose programming languages.
• 34:02
  SPEAKER_00: And you're coming back to the idea of the program being the mental model.
• 34:05
  SPEAKER_00: We would evolve that mental model at the code point of view.
• 34:09
  SPEAKER_00: That was the most effective place was us working at Java,
• 34:11
  SPEAKER_00: you know,
• 34:13
  SPEAKER_00: Java, C Sharp, whatever it is, rather than working at assembly level, right?
• 34:18
  SPEAKER_00: And it's like, okay, well...
• 34:20
  SPEAKER_00: It's actually what we're talking about, really.
• 34:21
  SPEAKER_00: We're stepping that level up.
• 34:23
  SPEAKER_00: We're still actually building a product together.
• 34:25
  SPEAKER_00: We're still thinking about what the software is.
• 34:27
  SPEAKER_00: We're still building that mental model,
• 34:29
  SPEAKER_00: maybe now more collaboratively involving,
• 34:32
  SPEAKER_00: you know,
• 34:33
  SPEAKER_00: more product people, for example, in that conversation when we're writing our specs.
• 34:37
  SPEAKER_00: And so the question for me is, are we ready to make that step?
• 34:41
  SPEAKER_00: And the answer is no, not everywhere.
• 34:43
  SPEAKER_01: And this is a really good point,
• 34:43
  SPEAKER_01: because if you think about it, like,
• 34:45
  SPEAKER_01: you know, like one of the mantras inside of Facebook used to be that I think...
• 34:50
  SPEAKER_01: Something along the line that code is truth or something like that.
• 34:53
  SPEAKER_01: But in general,
• 34:54
  SPEAKER_01: in a lot of tech companies,
• 34:56
  SPEAKER_01: if you want to know how things work,
• 34:59
  SPEAKER_01: like where the truth lives, is it in documents or is it in the coded business logic?
• 35:03
  SPEAKER_01: It's in the code in the business logic.
• 35:04
  SPEAKER_01: And if it's well-maintained code,
• 35:07
  SPEAKER_01: often well-commented or structured,
• 35:09
  SPEAKER_01: you can actually see the business logic and the rules for,
• 35:11
  SPEAKER_01: I don't know, for example.
• 35:12
  SPEAKER_00: See, I feel like I've got to channel my friend Charity at this point and say
• 35:16
  SPEAKER_00: the truth is production.
• 35:18
  SPEAKER_00: Everything else is the lies we tell ourselves.
• 35:21
  SPEAKER_00: Production is truth.
• 35:23
  SPEAKER_00: But I think in terms of like a definition of what the system should be doing,
• 35:28
  SPEAKER_00: there is this ongoing debate about where is that definition?
• 35:32
  SPEAKER_00: Is it the code or is it the spec?
• 35:33
  SPEAKER_00: Brigitte Brockley did a nice article on Martin Faller's website recently
• 35:37
  SPEAKER_00: about spec-driven development.
• 35:39
  SPEAKER_00: And she talks about the idea of some companies being spec-first or spec-anchored.
• 35:44
  SPEAKER_00: And a lot of it comes down to where they see the source of truth being.
• 35:48
  SPEAKER_00: So a lot of companies are using things like Kiro or SpecKit to define a feature and
• 35:56
  SPEAKER_00: then the LLM executes that spec and you can iterate on that process, right?
• 36:02
  SPEAKER_00: And so a lot of people who start off,
• 36:04
  SPEAKER_00: they use the spec to write the code and then they throw the spec away.
• 36:07
  SPEAKER_00: And then you get other companies that keep the spec and then when
• 36:10
  SPEAKER_00: the feature changes, they just update the spec.
• 36:12
  SPEAKER_00: And so they're sort of seeing a bit the code and a bit the spec as a source of truth
• 36:16
  SPEAKER_00: in terms of what the system should do.
• 36:18
  SPEAKER_00: And then you're now starting to get companies where they didn't even ever look at
• 36:21
  SPEAKER_00: the code and they just look at the spec.
• 36:23
  SPEAKER_00: And so we're kind of going on this spectrum.
• 36:25
  SPEAKER_00: She had some great terms for these.
• 36:27
  SPEAKER_00: And I think for me, it's like, is that the right path?
• 36:29
  SPEAKER_00: I don't know.
• 36:30
  SPEAKER_00: I think there's a lot of questions about what we would need to do to get
• 36:34
  SPEAKER_00: that We have a degree of confidence about what happens when I execute a line
• 36:40
  SPEAKER_00: of Java code.
• 36:41
  SPEAKER_00: Beyond the safety of the compiler and the runtime and all that stuff,
• 36:45
  SPEAKER_00: there's a lot more questions to answer,
• 36:47
  SPEAKER_00: I think,
• 36:47
  SPEAKER_00: before the whole world is ready to say the spec is the That's a source of truth
• 36:51
  SPEAKER_00: of what our system is supposed to do.
• 36:52
  SPEAKER_00: Production is always going to be the real truth, so just to be clear.
• 36:55
  SPEAKER_01: But I think it's a good question to just ask yourself,
• 36:57
  SPEAKER_01: however you're building,
• 36:58
  SPEAKER_01: however you're using agents, where is the truth of your system right now?
• 37:02
  SPEAKER_01: Is it more in the code?
• 37:04
  SPEAKER_01: Do you have a spec?
• 37:05
  SPEAKER_01: Is it scattered between the minds of people?
• 37:09
  SPEAKER_01: Which goes back to the collaboration,
• 37:11
  SPEAKER_01: because again,
• 37:12
  SPEAKER_01: a lot of the reality of a lot of startups are is a few people hold these things
• 37:16
  SPEAKER_01: in their head, right?
• 37:16
  SPEAKER_01: Which is okay in the beginning.
• 37:18
  SPEAKER_01: But of course,
• 37:18
  SPEAKER_01: as you scale,
• 37:19
  SPEAKER_01: one of the reasons we used to love code review,
• 37:21
  SPEAKER_01: which is now going out of fashion,
• 37:22
  SPEAKER_01: but it We used to share information and when someone left the company or got ill or,
• 37:28
  SPEAKER_01: you know,
• 37:29
  SPEAKER_01: the famous bus factor,
• 37:30
  SPEAKER_01: it was okay because other people understood because they reviewed,
• 37:33
  SPEAKER_01: they were taking part in that part of it.
• 37:35
  SPEAKER_01: But now with AI code reviewers,
• 37:38
  SPEAKER_01: some companies are like, well, we're not going to do it because it's too much code.
• 37:41
  SPEAKER_00: That was always one of the superpowers of pairing.
• 37:44
  SPEAKER_00: Pairing.
• 37:44
  SPEAKER_00: If you do promiscuous pairing, you rotate every day.
• 37:47
  SPEAKER_00: When someone leaves, everyone's got the same shared context.
• 37:49
  SPEAKER_00: That was something people missed around one of the benefits of pairing.
• 37:53
  SPEAKER_00: I think there's a lot to be said around this.
• 37:54
  SPEAKER_00: I was chatting to Chris Ford.
• 37:56
  SPEAKER_00: He's sort of head of software, head of technology for ThoughtWorks in Spain.
• 38:01
  SPEAKER_00: And he said
• 38:02
  SPEAKER_00: the biggest problem we've got around AI-driven software delivery is tacit knowledge.
• 38:09
  SPEAKER_00: There's a lot of knowledge in our heads that we've built up
• 38:14
  SPEAKER_00: that we don't necessarily know that we have.
• 38:17
  SPEAKER_00: When you do something like a code review,
• 38:19
  SPEAKER_00: you probably had looked at that and said, that doesn't really feel right.
• 38:21
  SPEAKER_00: I wouldn't do it like that.
• 38:22
  SPEAKER_00: And sometimes it's subjectivity,
• 38:24
  SPEAKER_00: but sometimes there's something about that that you can't quite pinpoint.
• 38:27
  SPEAKER_00: Maybe you can unpack it, maybe you can't.
• 38:30
  SPEAKER_00: But it's also like you have bunches of things like that in your head
• 38:33
  SPEAKER_00: about how software delivery should be done,
• 38:35
  SPEAKER_00: about how the business works and everything else.
• 38:37
  SPEAKER_00: If we want to step up that level of abstraction to work at the spec level,
• 38:42
  SPEAKER_00: we actually need as human beings to be able to unpack that tacit knowledge
• 38:47
  SPEAKER_00: that we have and make sure it's in the spec.
• 38:49
  SPEAKER_00: If we really want to be hands off.
• 38:52
  SPEAKER_00: I also think there's a fundamental problem that we've got in the current way a lot
• 38:56
  SPEAKER_00: of people are doing software deliveries.
• 38:57
  SPEAKER_00: You've got,
• 38:57
  SPEAKER_00: and this comes back to the cognitive debt stuff,
• 38:59
  SPEAKER_00: which is you've got a lot of people running off in different directions.
• 39:03
  SPEAKER_00: I've seen a few people talking about pairing with their AI assistant.
• 39:06
  SPEAKER_00: Not a human.
• 39:07
  SPEAKER_00: Stop pretending it is.
• 39:08
  SPEAKER_00: And so you're all going off, you're doing more in a different direction.
• 39:11
  SPEAKER_00: That shared mental model of the program is breaking down and there's gaps in between.
• 39:16
  SPEAKER_00: AI was supposed to free us from drudgery.
• 39:20
  SPEAKER_00: It isn't for most software developers because we're doing more work,
• 39:23
  SPEAKER_00: we've got more context switching, we're losing that big picture.
• 39:25
  SPEAKER_00: So I think if we can work collectively on the specs,
• 39:29
  SPEAKER_00: we can rebuild that model, then that absolutely could be the source of truth.
• 39:32
  SPEAKER_00: I still think it's going to be parts our brains and part the spec, right?
• 39:37
  SPEAKER_00: But for that to work, there does have to be the collaboration piece.
• 39:42
  SPEAKER_00: Whether or not that's our dev teams go from being 15 people all in to
• 39:46
  SPEAKER_00: about five people or six people all in, don't know.
• 39:49
  SPEAKER_00: But collaboration is really important.
• 39:52
  SPEAKER_00: You know, Chain Guard are a proper company, right?
• 39:54
  SPEAKER_00: They do proper stuff.
• 39:54
  SPEAKER_00: They're excellent for them.
• 39:56
  SPEAKER_00: They recently announced they were getting rid of any human-based code reviews,
• 39:59
  SPEAKER_00: and they said,
• 39:59
  SPEAKER_00: we're going to spend more time working on things like,
• 40:02
  SPEAKER_00: effectively working at design doc level.
• 40:03
  SPEAKER_00: And actually, I'm okay with that, because firstly, I trust that they know what they're doing.
• 40:06
  SPEAKER_00: But also,
• 40:08
  SPEAKER_00: there's something about I mean,
• 40:10
  SPEAKER_00: I think a lot of companies are making the mistake of not doing the thinking,
• 40:13
  SPEAKER_00: whereas Chain Guard is saying we're going to spend
• 40:15
  SPEAKER_00: that time doing more important thinking, right?
• 40:18
  SPEAKER_00: And I think that's sort of how we should view this.
• 40:20
  SPEAKER_00: I'm okay,
• 40:21
  SPEAKER_00: you know,
• 40:21
  SPEAKER_00: not looking at the code as long as you're doing other things to make sure
• 40:24
  SPEAKER_00: the system is operating correctly and that you're thinking critically about how
• 40:31
  SPEAKER_00: the system needs to change and evolve.
• 40:33
  SPEAKER_00: If you're just not looking at the code and you're not caring
• 40:35
  SPEAKER_00: about those other things,
• 40:36
  SPEAKER_00: well, then you deserve everything that's going to happen to you.
• 40:38
  SPEAKER_01: Which also goes back a little bit to what Charity was talking about on how you want
• 40:42
  SPEAKER_01: to have systems that value, that works properly.
• 40:44
  SPEAKER_01: About having empathy,
• 40:45
  SPEAKER_01: for example,
• 40:46
  SPEAKER_01: for the work that QAs and SRE folks have had to own code in production
• 40:52
  SPEAKER_01: that they did not write.
• 40:54
  SPEAKER_01: They don't necessarily read or even understand, but they know how to operate.
• 40:59
  SPEAKER_01: They know what good looks like in production.
• 41:01
  SPEAKER_01: They know what it looks like when it goes poorly.
• 41:04
  SPEAKER_00: I mean,
• 41:04
  SPEAKER_00: that's also a bigger thing that now more developers are having to struggle with
• 41:08
  SPEAKER_00: because if you want to do spectrum development,
• 41:10
  SPEAKER_00: you have to know what good looks like.
• 41:11
  SPEAKER_00: I did a, I run these like sort of half day conferences for O'Reilly online.
• 41:15
  SPEAKER_00: We did one recently on software delivery.
• 41:19
  SPEAKER_00: A load of speakers and about six speakers and four
• 41:23
  SPEAKER_00: of them were talking specifically around different varieties
• 41:26
  SPEAKER_00: of spectrum development effectively.
• 41:29
  SPEAKER_00: And all of them came back and said,
• 41:30
  SPEAKER_00: you can't do this unless you can define what good looks like.
• 41:33
  SPEAKER_00: And if you can't, don't bother.
• 41:35
  SPEAKER_00: And, you know,
• 41:35
  SPEAKER_00: some of those people were much more bullish,
• 41:37
  SPEAKER_00: like Addy, also very bullish on all of this stuff.
• 41:40
  SPEAKER_00: But one of the presenters who was doing,
• 41:41
  SPEAKER_00: they've got like a small software fracture,
• 41:43
  SPEAKER_00: only 15 or three people,
• 41:44
  SPEAKER_00: but like they're actively pulling in their observability data,
• 41:47
  SPEAKER_00: their hotel stuff, right, into their agent to understand how it's changing.
• 41:54
  SPEAKER_00: And I think Sam made a really good point that you cannot hand your work over
• 41:58
  SPEAKER_00: to agents unless it does these two things.
• 42:00
  SPEAKER_00: One, define what good looks like.
• 42:21
  SPEAKER_01: Two, prove that your system does what this good definition looks like.
• 42:25
  SPEAKER_01: You already know who I need to mention when it comes to proving
• 42:28
  SPEAKER_01: that your system works as expected.
• 42:29
  SPEAKER_01: Antithesis, our season sponsor.
• 42:32
  SPEAKER_01: Agents are writing more of the code,
• 42:33
  SPEAKER_01: so how can we trust that our system works correctly without bugs?
• 42:36
  SPEAKER_01: Code review is an increasingly naive and ineffective approach to do so,
• 42:40
  SPEAKER_01: which is why Antithesis goes well beyond this practice.
• 42:43
  SPEAKER_01: Antithesis runs your whole system in a hostile simulation.
• 42:46
  SPEAKER_01: This simulation This simulation includes both targeted testing and fast testing.
• 42:49
  SPEAKER_01: By running this simulation, it finds every bug before your users do.
• 42:53
  SPEAKER_01: And because the simulation is fully deterministic,
• 42:55
  SPEAKER_01: it doesn't only find bugs,
• 42:57
  SPEAKER_01: it gives you a perfect reproduction of every issue,
• 43:00
  SPEAKER_01: which makes it a lot easier to fix issues.
• 43:02
  SPEAKER_01: The first thing I thought when I heard about Antithesis and
• 43:04
  SPEAKER_01: that it's automated bug discovery and fully deterministic testing,
• 43:08
  SPEAKER_01: that it sounds like science fiction,
• 43:09
  SPEAKER_01: but actually it's hardcore engineering when you look under the hood.
• 43:13
  SPEAKER_01: JaneStreet, Fly.io,
• 43:14
  SPEAKER_01: and the etcd community ship agent origin code with full confidence
• 43:18
  SPEAKER_01: because they know it's been verified by Antithesis.
• 43:20
  SPEAKER_01: To see more case studies and details, head to antithesis.com slash pragmatic.
• 43:25
  SPEAKER_01: I'd also like to mention our presenting sponsor, TurboPuffer.
• 43:27
  SPEAKER_01: We know it's a database that's ridiculously fast and extremely cheap.
• 43:31
  SPEAKER_01: And they are doing something amazing.
• 43:33
  SPEAKER_01: They are completely redesigning their storage architecture from first principles
• 43:36
  SPEAKER_01: to make search faster, cheaper, and more reliable at scale.
• 43:40
  SPEAKER_01: If you follow TurboBuffer,
• 43:41
  SPEAKER_01: you know that their storage architecture was a massive part of their early success.
• 43:45
  SPEAKER_01: Redesigning a winning architecture is a big deal.
• 43:48
  SPEAKER_01: It's one thing for query plans to pass unit tests which is already happening.
• 43:52
  SPEAKER_01: It's another to bring each query plan to performance parity or better
• 43:56
  SPEAKER_01: while maintaining correctness and reliability in production.
• 43:59
  SPEAKER_01: And the team is sharing the messy reality of the rewrite and how
• 44:01
  SPEAKER_01: the v3 version is currently A lot slower than the current production one.
• 44:05
  SPEAKER_01: For example,
• 44:06
  SPEAKER_01: we can see how cold full-text search is 11 times slower than prod
• 44:10
  SPEAKER_01: and hot full-text search is 126 times slower.
• 44:13
  SPEAKER_01: But we can already see some wins like hot and cold vector search is at parity or better.
• 44:18
  SPEAKER_01: I really appreciate this work in progress view and the real-time logs
• 44:22
  SPEAKER_01: and what it takes to do a successful storage architecture rewrite.
• 44:24
  SPEAKER_01: The TurboPuffer team is keeping on the work log of their journey
• 44:27
  SPEAKER_01: and you can follow along at turbopuffer.com slash v3.
• 44:31
  SPEAKER_01: I definitely will do.
• 44:32
  SPEAKER_01: And with this,
• 44:32
  SPEAKER_01: let's get back to Sam and what QA and S3 folks already know about running code
• 44:36
  SPEAKER_01: that they didn't write.
• 44:38
  SPEAKER_01: So I want to talk to you about your latest book that is just coming out,
• 44:42
  SPEAKER_01: Building Resilient Distributed Systems.
• 44:44
  SPEAKER_00: Yeah.
• 44:45
  SPEAKER_01: What is the book about and why did you feel this is a good time to write it?
• 44:49
  SPEAKER_00: Years and years and years ago, I went to Velocity, which was quite a well-known conference.
• 44:55
  SPEAKER_01: It's a conference.
• 44:55
  SPEAKER_01: Is it still on?
• 44:56
  SPEAKER_00: No, no.
• 44:56
  SPEAKER_00: It was the O'Reilly Conference.
• 44:58
  SPEAKER_00: And it was like the conference where a lot of the early dot-com people went to talk
• 45:01
  SPEAKER_00: about how they made their systems scalable and stuff.
• 45:03
  SPEAKER_00: I got to go.
• 45:05
  SPEAKER_00: It had been like one of those career ambitions to go to.
• 45:07
  SPEAKER_00: I got to go.
• 45:08
  SPEAKER_00: Because after I'd written the first book,
• 45:10
  SPEAKER_00: and I got chatting to John Allsport there, who I still chat to for now and then.
• 45:13
  SPEAKER_00: He now works in his resilience engineering space.
• 45:15
  SPEAKER_00: Pointed me at, I don't know if it was him or someone else, but it pointed me at this paper.
• 45:19
  SPEAKER_00: By David Woods, it was all about resilience engineering.
• 45:22
  SPEAKER_00: And the paper was utterly baffling to me, in absolutely no sense to me.
• 45:25
  SPEAKER_00: But it was all,
• 45:26
  SPEAKER_00: because it's not,
• 45:26
  SPEAKER_00: yeah, because it's just, it's not written for computer scientists to read.
• 45:29
  SPEAKER_00: And I'm not even a computer scientist, so I was really in trouble.
• 45:32
  SPEAKER_00: But it was really fascinating seeing this whole other world about how we make,
• 45:37
  SPEAKER_00: how people view resiliency, and then what we do in the software world.
• 45:41
  SPEAKER_00: And people like John were trying to take some of those ideas in a little bit,
• 45:44
  SPEAKER_00: but I found it very difficult to take some of those ideas from Sort
• 45:47
  SPEAKER_00: of safety critical systems and that type of thinking and make in any way,
• 45:52
  SPEAKER_00: shape or form palatable for normal people like me.
• 45:55
  SPEAKER_00: And so I kind of This would have been back, what, 2016 maybe, 2015?
• 46:01
  SPEAKER_00: And so I'd go back and try and reread that paper every six months or so,
• 46:04
  SPEAKER_00: and I'd read bits around the edges and see other smarter people than me
• 46:07
  SPEAKER_00: that summarized it data better.
• 46:09
  SPEAKER_00: And then I was thinking, okay, these ideas now are starting to make sense.
• 46:12
  SPEAKER_00: And then also I kind of realized that despite my best efforts,
• 46:15
  SPEAKER_00: lots of people seem to be building microservice architectures,
• 46:18
  SPEAKER_00: even though If you've read my books,
• 46:20
  SPEAKER_00: most of my book is about telling you not to do it.
• 46:22
  SPEAKER_01: But you have to read the book, which is called microservices.
• 46:26
  SPEAKER_00: Let's be clear, as long as they buy the book, that's the most important thing.
• 46:28
  SPEAKER_00: Whether they read it or not, that's just, you know.
• 46:30
  SPEAKER_00: And so then I realized that there were more people exposed to the horror,
• 46:35
  SPEAKER_00: pain and suffering of a distributed system.
• 46:37
  SPEAKER_00: And so it was like, okay, so I've got this stuff happening.
• 46:40
  SPEAKER_00: I've got this, like my, you know, my son's developing now.
• 46:43
  SPEAKER_00: And I almost thought that if somebody found themselves building
• 46:47
  SPEAKER_00: a distributed system,
• 46:49
  SPEAKER_00: Working as part of building a distributed system,
• 46:51
  SPEAKER_00: what was like the kind of equipment of a rubber ring I could give them,
• 46:54
  SPEAKER_00: say, stay afloat and don't drown.
• 46:56
  SPEAKER_00: And so the goal of the book is to start you off with really easy,
• 46:59
  SPEAKER_00: foundational,
• 47:01
  SPEAKER_00: simple kind of engineering concepts about how you can start making,
• 47:05
  SPEAKER_00: dealing with the problems of the disability system,
• 47:07
  SPEAKER_00: but to also take you on a journey into kind of the whole concept
• 47:10
  SPEAKER_00: of resilience engineering.
• 47:11
  SPEAKER_00: And so it starts off with really easy ideas and then you slowly get more and more complicated.
• 47:16
  SPEAKER_00: And, you know,
• 47:16
  SPEAKER_00: so I introduced kind of concepts like what a socio-technical system is,
• 47:20
  SPEAKER_00: the difference between safety management and resilience engineering.
• 47:23
  SPEAKER_00: These are all like big concepts.
• 47:25
  SPEAKER_00: But the second chapter after the introduction is what's observability, let's talk about OTEL.
• 47:30
  SPEAKER_00: The third chapter is what's a timeout, right?
• 47:33
  SPEAKER_00: And these are all, I think, quite tractable things to talk to people about.
• 47:36
  SPEAKER_01: But they're building blocks, right?
• 47:37
  SPEAKER_01: Yeah.
• 47:38
  SPEAKER_01: The primitives for these systems and the vocabulary as well.
• 47:41
  SPEAKER_00: Yeah, absolutely.
• 47:42
  SPEAKER_00: And to give,
• 47:43
  SPEAKER_00: you know,
• 47:43
  SPEAKER_00: to go through and sort of distill some of that knowledge down and also to be a bit
• 47:48
  SPEAKER_00: of,
• 47:48
  SPEAKER_00: I'm quite happy being a bit opinionated on these things,
• 47:51
  SPEAKER_00: but I'm also lucky now,
• 47:53
  SPEAKER_00: partly because of my early work,
• 47:54
  SPEAKER_00: I get to chat to more people so I can actually talk to people
• 47:56
  SPEAKER_00: who this is their full-time job and try and distill down some of that knowledge
• 48:00
  SPEAKER_00: a bit.
• 48:00
  SPEAKER_00: And so I wanted that book,
• 48:02
  SPEAKER_00: partly it's about being evergreen,
• 48:04
  SPEAKER_00: but it was also about saying there are lots of people building a distributed system.
• 48:07
  SPEAKER_00: Let's put what type of distributed system you've got off to one side for a moment.
• 48:10
  SPEAKER_00: There are lots of people building them,
• 48:12
  SPEAKER_00: whether they should or not, and they're really important to us now.
• 48:15
  SPEAKER_00: So how do I help people start and grow?
• 48:20
  SPEAKER_00: And that's what I wanted to do.
• 48:21
  SPEAKER_00: I mean, I enjoy the process of writing anyway,
• 48:24
  SPEAKER_00: but this is a book I wanted to be able to give to people.
• 48:26
  SPEAKER_00: That's always been why I wrote a book.
• 48:27
  SPEAKER_00: I want that book.
• 48:28
  SPEAKER_00: Every now and then I've gone sat down to write a book
• 48:31
  SPEAKER_00: and I found someone else has done it already.
• 48:33
  SPEAKER_00: It's like, well, that exists now.
• 48:35
  SPEAKER_00: So this for me was a book that I wanted to exist, but didn't.
• 48:37
  SPEAKER_00: That's kind of why I wrote it.
• 48:39
  SPEAKER_01: Yeah, and I think it's a book I certainly could have used.
• 48:42
  SPEAKER_01: It got dropped into a place like Uber
• 48:44
  SPEAKER_01: where we did have what I realized were distributed systems.
• 48:48
  SPEAKER_01: Just large systems handling a bunch of stuff,
• 48:50
  SPEAKER_01: lots of services depending on one another because even a simple thing
• 48:54
  SPEAKER_01: of knowing like, oh, you should pay attention to observability.
• 48:57
  SPEAKER_01: I mean, you figure it out after a while, but it takes a while.
• 49:00
  SPEAKER_00: Yeah, and I think that's, I mean, again, I mean,
• 49:03
  SPEAKER_00: maybe this is my attempt to unpack some of that tacit knowledge to an extent.
• 49:06
  SPEAKER_00: I've made so many mistakes and I keep making mistakes.
• 49:09
  SPEAKER_00: I think making mistakes is useful,
• 49:11
  SPEAKER_00: but I think also creating opportunities where people can learn
• 49:13
  SPEAKER_00: from other people's mistakes first is a good idea too.
• 49:16
  SPEAKER_00: In the book,
• 49:16
  SPEAKER_00: I also try and pull in real world examples and sometimes from big companies,
• 49:22
  SPEAKER_00: sometimes from my mistakes to kind of make this stuff seem real.
• 49:25
  SPEAKER_00: And I think the other thing I think I really try and get across is that things go wrong, right?
• 49:31
  SPEAKER_00: I mean, engineering is all about making sure as many things go right as possible.
• 49:34
  SPEAKER_00: It's not about making sure everything goes right.
• 49:38
  SPEAKER_00: And there's also the only way you can build resilient systems in general is also
• 49:42
  SPEAKER_00: by being honest about mistakes that have been made.
• 49:46
  SPEAKER_00: And talking about them and sharing them because that's how everyone gets better.
• 49:50
  SPEAKER_00: And if you don't do that, you can't build a resilient system in your company.
• 49:54
  SPEAKER_00: And so this, again,
• 49:55
  SPEAKER_00: is my attempt to sort of highlight all the awesome work that's gone before,
• 49:59
  SPEAKER_00: the fantastic post-incident reports of people that some companies share.
• 50:03
  SPEAKER_01: Like when there's a system that we all use or many of us use has an outage.
• 50:07
  SPEAKER_01: I mean, it's it's annoying, especially when you're using it.
• 50:10
  SPEAKER_01: But I always greatly appreciate when the companies because they don't have
• 50:13
  SPEAKER_01: to do this,
• 50:13
  SPEAKER_01: but when they share a public postmortem or incident recap,
• 50:17
  SPEAKER_01: whatever That is, they go through what happened.
• 50:19
  SPEAKER_01: For example,
• 50:19
  SPEAKER_01: GitHub had an outage and they shared a really interesting one on what all the things
• 50:24
  SPEAKER_01: that went wrong and in what order.
• 50:26
  SPEAKER_01: And then you also have companies where something like this happens and they don't do anything.
• 50:30
  SPEAKER_01: So, of course, you're not obliged to And anyway, but I always have a bit of a lesser of...
• 50:37
  SPEAKER_00: They are obliged to now, a lot of them.
• 50:39
  SPEAKER_00: Regulations are now making them.
• 50:41
  SPEAKER_01: Which I think is great.
• 50:42
  SPEAKER_00: Yeah, things like DORA and stuff in the UK, in Europe rather.
• 50:45
  SPEAKER_00: And I think there's also,
• 50:46
  SPEAKER_00: we should also separate around that post-instant report stuff that there's marketing
• 50:49
  SPEAKER_00: and PR versus have you done a proper post-instant review?
• 50:53
  SPEAKER_00: Now, I do think you've a duty of care really to let users know it's part of
• 50:56
  SPEAKER_00: a proper post-instant review, but those companies are maybe saying nothing.
• 50:59
  SPEAKER_00: They might do a fantastic process internally.
• 51:01
  SPEAKER_00: There's also the fact that some companies that put those things out,
• 51:04
  SPEAKER_00: you read them and you're like, this does feel more marketing.
• 51:07
  SPEAKER_00: I think a great,
• 51:08
  SPEAKER_00: if you want a good,
• 51:09
  SPEAKER_00: a company that puts out some of
• 51:11
  SPEAKER_00: the best self-flagellating post-incident review write-ups,
• 51:15
  SPEAKER_00: go for Cloudflare stuff, they absolutely go for themselves in those.
• 51:19
  SPEAKER_00: And it's like, I feel like, guys...
• 51:21
  SPEAKER_00: Go easy on yourselves, right?
• 51:23
  SPEAKER_00: Everyone makes some mistakes, but I quite
• 51:24
  SPEAKER_01: like it.
• 51:25
  SPEAKER_01: And they did it rapidly within 24 hours and they're a publicly traded company.
• 51:29
  SPEAKER_01: Yeah.
• 51:30
  SPEAKER_01: Which I think just goes back to the myth,
• 51:31
  SPEAKER_01: like some of them are like,
• 51:32
  SPEAKER_01: we're publicly traded,
• 51:33
  SPEAKER_01: we don't want to do it because it will give away information for shareholders.
• 51:36
  SPEAKER_01: No, like Cloudflare does it and they're fine.
• 51:39
  SPEAKER_00: Exactly.
• 51:40
  SPEAKER_00: And also it's... And it builds up trust.
• 51:42
  SPEAKER_00: It is.
• 51:43
  SPEAKER_00: I mean,
• 51:43
  SPEAKER_00: a lot of how an organization is perceived is down to how you deal with mistakes
• 51:48
  SPEAKER_00: when they occur, right?
• 51:49
  SPEAKER_00: I mean...
• 51:50
  SPEAKER_00: Nothing ever goes perfectly for every company.
• 51:52
  SPEAKER_01: So in your book, you talk about the three rules of distributed systems.
• 51:56
  SPEAKER_01: What are these three rules?
• 51:57
  SPEAKER_00: So I really tried to distill this down because I could never remember the eight,
• 52:02
  SPEAKER_00: the rule of eight that came out of Sun, the fallacies distributed computing.
• 52:06
  SPEAKER_00: I always forget what those eight were.
• 52:07
  SPEAKER_00: There's loads of complicated stuff,
• 52:08
  SPEAKER_00: but I realized when you strip it all down,
• 52:10
  SPEAKER_00: there's really only three things that really cause all the other problems.
• 52:14
  SPEAKER_00: The first is that you can't send information instantaneously from point A to point B.
• 52:19
  SPEAKER_00: It's physics.
• 52:20
  SPEAKER_00: I mean, now it turns out,
• 52:21
  SPEAKER_00: when you get into it and start talking to people about quantum physics,
• 52:24
  SPEAKER_00: they say, aha.
• 52:25
  SPEAKER_00: And there's a thing called quantum entanglement,
• 52:28
  SPEAKER_00: which does allow for the theoretical instantaneous read of electron spin states
• 52:32
  SPEAKER_00: across arbitrary distances.
• 52:34
  SPEAKER_00: At which point I point out that A,
• 52:36
  SPEAKER_00: that only works for reads, as far as I'm aware, and B, not a networking protocol.
• 52:40
  SPEAKER_00: So like...
• 52:41
  SPEAKER_00: But it takes time, right?
• 52:42
  SPEAKER_00: So I'm going to get from A to B.
• 52:44
  SPEAKER_00: And that, I can't control all of that.
• 52:47
  SPEAKER_00: I can control bits of that as a developer.
• 52:49
  SPEAKER_00: I could control how much data I send.
• 52:51
  SPEAKER_00: I could control the payload size, technology I use to serialize and deserialize.
• 52:56
  SPEAKER_00: I'm unlikely to be specifying the network cabling,
• 52:59
  SPEAKER_00: the equipment we use, or the distances between them.
• 53:01
  SPEAKER_00: So I'm not really under my control.
• 53:03
  SPEAKER_00: And the moment you go over the public internet,
• 53:06
  SPEAKER_00: BGP has a bad day and your packet has gone down the wrong trouser leg of
• 53:09
  SPEAKER_00: the internet, right?
• 53:10
  SPEAKER_00: So that's the rule one.
• 53:11
  SPEAKER_00: It takes time.
• 53:13
  SPEAKER_00: And that's important to recognize because sometimes it looks instantaneous.
• 53:16
  SPEAKER_00: It's not.
• 53:17
  SPEAKER_00: Second thing, sometimes the thing you want to talk to isn't there.
• 53:20
  SPEAKER_00: Sometimes that's under.
• 53:21
  SPEAKER_00: And we do things to make it under our control, right?
• 53:23
  SPEAKER_00: Like we would run, say, multiple copies of a service instance behind a load balancer.
• 53:28
  SPEAKER_00: What happens is the load balancer fails.
• 53:30
  SPEAKER_00: But, I mean, like, you can't control for everything, right?
• 53:33
  SPEAKER_00: I've had services taken out by a fire in a data center,
• 53:37
  SPEAKER_00: a SAN caught fire, and specifically one bit of a SAN.
• 53:40
  SPEAKER_00: The SAN had a caching storage layer with SSDs, these two pairs of SSDs.
• 53:46
  SPEAKER_00: One of the pairs of SSDs caught fire.
• 53:48
  SPEAKER_00: The other pair, the SSDs, were working fine.
• 53:50
  SPEAKER_00: And so we had these weird latency graphs coming out of the SAN,
• 53:53
  SPEAKER_00: like, perfectly fine if your load went through the ones that weren't on fire.
• 53:58
  SPEAKER_00: But the failovers in the other one were routing,
• 54:00
  SPEAKER_00: to be fair, it was routing around the bits while on fire.
• 54:02
  SPEAKER_00: I had a system where we had ducting between the office buildings and rabbits moved
• 54:07
  SPEAKER_00: into the ducts and ate their networking cable.
• 54:10
  SPEAKER_00: I can't control for rabbits, right?
• 54:12
  SPEAKER_00: And sure, these things are like, they're real.
• 54:15
  SPEAKER_00: These are true things, right?
• 54:16
  SPEAKER_00: Yes, you can control that and you can reduce the chances,
• 54:19
  SPEAKER_00: but you can't eliminate those chances, right?
• 54:20
  SPEAKER_00: So sometimes, rule number two is sometimes the thing you want to talk to isn't there.
• 54:24
  SPEAKER_00: And the third rule,
• 54:25
  SPEAKER_00: I'm very grateful to an old colleague of mine,
• 54:26
  SPEAKER_00: Ben Butler-Cole, for saying I'm missing, is that resource pools are not infinite.
• 54:32
  SPEAKER_01: Meaning?
• 54:32
  SPEAKER_00: You can run out of computing resources.
• 54:34
  SPEAKER_00: CPU, memory, IO, whatever.
• 54:37
  SPEAKER_01: Basically,
• 54:38
  SPEAKER_01: your load balancer could be overloaded and then just not respond even
• 54:41
  SPEAKER_01: though it's there and we know the latency, but maybe its queue fills up.
• 54:46
  SPEAKER_00: And look, the vast majority of the time when you have a system outage, it's down to that.
• 54:50
  SPEAKER_00: Some resource has been totally saturated somewhere.
• 54:53
  SPEAKER_00: And it might have been that saturation might in turn have been caused by the fact
• 54:56
  SPEAKER_00: that something is down and you're in a retry storm,
• 55:00
  SPEAKER_00: which is in turn causing,
• 55:01
  SPEAKER_00: you know,
• 55:02
  SPEAKER_00: connection starvation,
• 55:03
  SPEAKER_00: which is in course because resource saturation, which is in turn yada, yada, yada.
• 55:07
  SPEAKER_00: But that's the killer.
• 55:09
  SPEAKER_00: And again, I think we understand that, you know, sort of intellectually, all of these things.
• 55:14
  SPEAKER_00: But You can go to all the complicated concepts in distributed systems and talk
• 55:18
  SPEAKER_00: about why is Byzantine consensus protocol a thing?
• 55:20
  SPEAKER_00: Well, let's just go all the way down to this bit.
• 55:22
  SPEAKER_00: What about CAP and PAC-HELC?
• 55:25
  SPEAKER_00: Let's put ourselves on the side.
• 55:27
  SPEAKER_00: Remember these three rules first and then most problems end up becoming a little bit
• 55:32
  SPEAKER_00: A little bit easier to deal with.
• 55:34
  SPEAKER_01: And in the book,
• 55:35
  SPEAKER_01: after you introduce these rules,
• 55:36
  SPEAKER_01: you start, as mentioned, the second chapter is observability.
• 55:39
  SPEAKER_01: To me, that kind of suggests that this is a really,
• 55:42
  SPEAKER_01: really important part for distributed systems.
• 55:44
  SPEAKER_01: Why is observability so important?
• 55:46
  SPEAKER_00: I mean, the way to think about observability is it's a property of a system, right?
• 55:49
  SPEAKER_00: So the more observable something is,
• 55:52
  SPEAKER_00: the easier it is for me to understand how that thing is behaving based on
• 55:55
  SPEAKER_00: the external signals.
• 55:57
  SPEAKER_00: When you run a system,
• 55:58
  SPEAKER_00: which is a single program,
• 56:00
  SPEAKER_00: You actually get some pretty high quality signals coming out of that program,
• 56:05
  SPEAKER_00: therefore give you a high degree of observability.
• 56:08
  SPEAKER_00: I can get a lot of information out of that.
• 56:10
  SPEAKER_00: I could attach a debugger to a process.
• 56:11
  SPEAKER_00: I could generate a core dump and run that in.
• 56:14
  SPEAKER_00: I could just thread dump on the JVM and work out where my semaphores are blocked.
• 56:18
  SPEAKER_00: If this is my whole system,
• 56:19
  SPEAKER_00: like knowing the CPU is stuck at 100%,
• 56:22
  SPEAKER_00: it's actually really useful information because that's probably a bad,
• 56:24
  SPEAKER_00: that's a problem, right?
• 56:26
  SPEAKER_00: So this basic information that we pretty much get for free is So, you have to do more work.
• 56:56
  SPEAKER_00: To make the system observable.
• 56:58
  SPEAKER_00: And as annoying with these sorts of things,
• 57:01
  SPEAKER_00: it's much easier to do that from the beginning than it is to retrofit it.
• 57:03
  SPEAKER_00: And then,
• 57:04
  SPEAKER_00: you know,
• 57:04
  SPEAKER_00: luckily we've got things like OTEL now,
• 57:06
  SPEAKER_00: the Open Telemetry Protocol and specification,
• 57:09
  SPEAKER_00: because that at least gives us an open framework, a vendor agnostic.
• 57:14
  SPEAKER_00: Although Honeycomb are the best.
• 57:15
  SPEAKER_00: Charity pays me in stickers.
• 57:17
  SPEAKER_00: Just to be really clear.
• 57:18
  SPEAKER_00: I always think I work independently.
• 57:20
  SPEAKER_00: I have to be very clear.
• 57:21
  SPEAKER_00: Charity pays me in stickers.
• 57:22
  SPEAKER_00: But like we get a vendor agnostic open standard for this now.
• 57:25
  SPEAKER_00: So it makes it so much easier for me to have the conversations about this.
• 57:28
  SPEAKER_00: But if I don't get the signals coming out of the system,
• 57:30
  SPEAKER_00: if I don't get that stream of structured events telling me what the hell's going on
• 57:34
  SPEAKER_00: with my system, I'm flying blind.
• 57:36
  SPEAKER_00: How do I know what appropriate timeout is from a client point of view talking to a service?
• 57:41
  SPEAKER_00: Well, I'm only going to know that if I have a sense of,
• 57:44
  SPEAKER_00: well, what is its normal response time distribution?
• 57:48
  SPEAKER_00: So I'm going to need latencies,
• 57:50
  SPEAKER_00: know the success or failure of those latencies,
• 57:52
  SPEAKER_00: probably want to be outplotting a histogram maybe to take a look at it.
• 57:55
  SPEAKER_00: If I don't have that information coming out,
• 57:57
  SPEAKER_00: I'm not going to know how to specify timeout correctly.
• 57:59
  SPEAKER_01: And then we're talking about a low-level thing like latency and things
• 58:02
  SPEAKER_01: that we can measure at the machine level.
• 58:03
  SPEAKER_01: But of course,
• 58:04
  SPEAKER_01: is it observably when I'm talking about,
• 58:06
  SPEAKER_01: like, does it work for the user or the customer?
• 58:08
  SPEAKER_01: Does it do what they do when they click the button to submit?
• 58:11
  SPEAKER_01: Does it submit?
• 58:12
  SPEAKER_01: Do you look at that?
• 58:13
  SPEAKER_01: Because we know observability is usually we talk about these kind of latencies,
• 58:17
  SPEAKER_01: the statuses, the 500 responses.
• 58:19
  SPEAKER_01: That's kind of bad.
• 58:20
  SPEAKER_01: That means it's down.
• 58:22
  SPEAKER_01: But there's always a player that I always wondered,
• 58:24
  SPEAKER_01: and I found it hard even when we're working in teams,
• 58:27
  SPEAKER_01: to Okay, like we have the easy observability.
• 58:30
  SPEAKER_01: Like this is kind of text, not textbook.
• 58:32
  SPEAKER_01: It's the metrics.
• 58:33
  SPEAKER_01: It's the metrics.
• 58:34
  SPEAKER_01: And the logs, maybe.
• 58:36
  SPEAKER_01: But then there's a stuff of, does it work?
• 58:38
  SPEAKER_01: Does it do what people care about?
• 58:40
  SPEAKER_01: Like, again, in the case of like an Uber...
• 58:42
  SPEAKER_01: Like when people hit a button to request your ride,
• 58:45
  SPEAKER_01: like does it actually request within usually five seconds or something like that?
• 58:48
  SPEAKER_01: Or is it stuck in an infinite spinner, which is bad?
• 58:51
  SPEAKER_01: Your locks can be as healthy as it can be, but people are upset, right?
• 58:55
  SPEAKER_00: You know,
• 58:55
  SPEAKER_00: you treat the raw data of making a system observable as a stream
• 59:00
  SPEAKER_00: of structured events.
• 59:01
  SPEAKER_00: And then from that, you build up the higher level abstractions and of use, right?
• 59:05
  SPEAKER_00: So uses as primitives.
• 59:06
  SPEAKER_00: So a trace, right, where we get a sense of for a given...
• 59:10
  SPEAKER_00: Click of a button, what happened in the production system?
• 59:13
  SPEAKER_00: And this is a big blind spot, right?
• 59:14
  SPEAKER_00: Most organizations had to build their own tracing systems.
• 59:17
  SPEAKER_00: So this is where things like the open tracing stuff came from and things like Zipkin
• 59:21
  SPEAKER_00: and things like that were built as an attempt to recreate.
• 59:24
  SPEAKER_00: Like, I mean,
• 59:25
  SPEAKER_00: most people were just normally trying
• 59:26
  SPEAKER_00: to recreate what Google had already built there.
• 59:28
  SPEAKER_00: And this was an attempt to say, okay, can we at least look at a flow of calls?
• 59:31
  SPEAKER_00: What happened during that flow of calls?
• 59:33
  SPEAKER_00: Where did these things go?
• 59:33
  SPEAKER_00: Well, that requires me to get a bunch of spans and All these different machines.
• 59:39
  SPEAKER_00: And a span meaning?
• 59:40
  SPEAKER_00: A span meaning an activity.
• 59:42
  SPEAKER_00: It's something I did.
• 59:44
  SPEAKER_00: And then a span can begat more spans.
• 59:47
  SPEAKER_00: So I can say my first span might be a call has hit my server.
• 59:51
  SPEAKER_00: So I've now got a span that starts.
• 59:53
  SPEAKER_00: Within that span, I might talk to my database and then do some other operations.
• 59:56
  SPEAKER_00: And I can put additional information on those spans.
• 59:58
  SPEAKER_00: And then I can go up to another service.
• 1:00:00
  SPEAKER_00: And then you get a trace ID that's shared across those spans and you've got timing.
• 1:00:03
  SPEAKER_00: So now you get effectively a view of what's happened across those calls.
• 1:00:07
  SPEAKER_00: But that's all just based on individual events that have been stitched together.
• 1:00:10
  SPEAKER_00: And then you also start getting into a situation which is, is my system behaving correctly?
• 1:00:16
  SPEAKER_00: And, you know,
• 1:00:16
  SPEAKER_00: this is going away from,
• 1:00:18
  SPEAKER_00: again, it's I think half the time I'm just saying what charity would say.
• 1:00:22
  SPEAKER_00: You know, again,
• 1:00:22
  SPEAKER_00: going away from the vanity dashboards,
• 1:00:24
  SPEAKER_00: this is moving into sort of the world we call the SLOs, service level objectives.
• 1:00:28
  SPEAKER_01: And SLIs.
• 1:00:29
  SPEAKER_00: And SLIs.
• 1:00:29
  SPEAKER_00: So an SLI,
• 1:00:30
  SPEAKER_00: service level indicator,
• 1:00:31
  SPEAKER_00: that's our way of saying,
• 1:00:33
  SPEAKER_00: at this given point in time for this thing, is this thing good or is this thing bad?
• 1:00:38
  SPEAKER_00: And we use a collection of those.
• 1:00:39
  SPEAKER_00: So one example of that might be a successful percentile response time, right?
• 1:00:44
  SPEAKER_00: So that would be, is it a good one or a bad one?
• 1:00:46
  SPEAKER_00: It could be a bad one because it's an error.
• 1:00:48
  SPEAKER_00: It could be a bad one because it's slow, good or bad.
• 1:00:51
  SPEAKER_00: And I collect those together and that helps me know if I'm meeting my SLO.
• 1:00:54
  SPEAKER_00: So I might have an overarching SLO,
• 1:00:56
  SPEAKER_00: which is I should be able to request a ride,
• 1:01:00
  SPEAKER_00: the 99th percentile successful ride request within 300 minutes.
• 1:01:07
  SPEAKER_00: And that would be your statement and that would be a commitment.
• 1:01:09
  SPEAKER_00: The SLO is supposed to be a commitment.
• 1:01:10
  SPEAKER_00: That's something our team is working towards.
• 1:01:13
  SPEAKER_00: You then get this raw data,
• 1:01:15
  SPEAKER_00: you calculate your SLIs on the individual things,
• 1:01:18
  SPEAKER_00: and then you use that to work out what your SLO is looking like.
• 1:01:21
  SPEAKER_00: If it's green, everything at a good time.
• 1:01:22
  SPEAKER_00: If it's red, people are unhappy.
• 1:01:24
  SPEAKER_00: I mean, you get a bit of red.
• 1:01:26
  SPEAKER_00: There's way more nuance to it.
• 1:01:27
  SPEAKER_00: I'm taking a very bad introduction to this stuff, I think.
• 1:01:29
  SPEAKER_00: But without those building blocks,
• 1:01:31
  SPEAKER_00: because I talk about the first chapter of the book,
• 1:01:33
  SPEAKER_00: nobody says, I don't want my system to be resilient.
• 1:01:37
  SPEAKER_01: No, no one.
• 1:01:38
  SPEAKER_00: But at the same time, resiliency is not a binary property.
• 1:01:41
  SPEAKER_00: It's not like off or on.
• 1:01:42
  SPEAKER_00: It is a bit like observability.
• 1:01:43
  SPEAKER_00: It's a degree.
• 1:01:44
  SPEAKER_00: So then the question is, how much do you need?
• 1:01:46
  SPEAKER_00: Because am I having a conversation with you about multi-cloud and,
• 1:01:50
  SPEAKER_00: you know,
• 1:01:51
  SPEAKER_00: doing complete rewrites of systems for critical standings like Monzo do,
• 1:01:55
  SPEAKER_00: for example?
• 1:01:55
  SPEAKER_01: Yeah, they have a stand-in bank.
• 1:01:57
  SPEAKER_01: They have a whole stand-in bank.
• 1:01:59
  SPEAKER_01: So it's sitting there.
• 1:02:00
  SPEAKER_01: It's costing them money for them.
• 1:02:01
  SPEAKER_01: It's not even a slight sidebar,
• 1:02:03
  SPEAKER_01: but firstly, their standby bank has a real stripped down list of features.
• 1:02:08
  SPEAKER_01: It's much more bare bones, but it's all the core ones.
• 1:02:10
  SPEAKER_01: You can freeze cards, you can take money
• 1:02:13
  SPEAKER_00: out, you can check your balance.
• 1:02:14
  SPEAKER_00: It's a complete rewrite.
• 1:02:26
  SPEAKER_00: So there's no shared code because they're worried about a bug.
• 1:02:29
  SPEAKER_00: Different cloud vendor.
• 1:02:30
  SPEAKER_00: It's also, by design, significantly more cheap to run.
• 1:02:34
  SPEAKER_00: So they've got thousands of microservices on the full stack.
• 1:02:36
  SPEAKER_00: This is about a tenth the cost of the cloud costs.
• 1:02:39
  SPEAKER_00: They also run it constantly.
• 1:02:42
  SPEAKER_00: So it's always there.
• 1:02:43
  SPEAKER_00: But like Monzo, no, they're Monzo, right?
• 1:02:45
  SPEAKER_00: That's how that works.
• 1:02:46
  SPEAKER_00: But like, there's no point you starting there.
• 1:02:47
  SPEAKER_00: So actually,
• 1:02:48
  SPEAKER_00: the reason I put the observatory stuff in is like,
• 1:02:50
  SPEAKER_00: you need the raw data,
• 1:02:51
  SPEAKER_00: but also you need to make decisions about, this comes back to what good looks like.
• 1:02:56
  SPEAKER_00: Then you need to know, okay,
• 1:02:57
  SPEAKER_00: do I need to worry about multi-cloud,
• 1:03:01
  SPEAKER_00: vendor, DR strategies, because that's where I need to be going?
• 1:03:06
  SPEAKER_00: Or am I somewhere else?
• 1:03:07
  SPEAKER_00: And so I think without that kind of sense of your North Star of how resilient do we need to be?
• 1:03:12
  SPEAKER_00: Without that defined, you don't know how deep you need to get.
• 1:03:16
  SPEAKER_00: This is why I'm trying to say how secure does my system need to be?
• 1:03:19
  SPEAKER_00: Have you done a threat model?
• 1:03:20
  SPEAKER_00: If you haven't done a threat model, I don't know.
• 1:03:22
  SPEAKER_01: And I guess to answer this question of how resilient do I need to be, it greatly helps.
• 1:03:27
  SPEAKER_01: And maybe it's actually a requirement to do that for
• 1:03:29
  SPEAKER_01: a decision is how much will it cost me in terms of operating costs,
• 1:03:33
  SPEAKER_01: in terms of time to build it,
• 1:03:34
  SPEAKER_01: in terms of,
• 1:03:35
  SPEAKER_01: for example,
• 1:03:35
  SPEAKER_01: latency,
• 1:03:36
  SPEAKER_01: just a simple example of,
• 1:03:37
  SPEAKER_01: Because if you say like,
• 1:03:38
  SPEAKER_01: okay, you know, the business is like, oh, would you like multi-cloud?
• 1:03:41
  SPEAKER_01: And business is like, yeah, we'd love multi-cloud.
• 1:03:43
  SPEAKER_01: Like if Amazon goes down because we heard they had an outage last week or
• 1:03:46
  SPEAKER_01: a few months ago, I actually want to be able to run on Google Cloud.
• 1:03:49
  SPEAKER_01: But then you'll be like, okay, well, first of all, this will mean we're paying two cloud bills.
• 1:03:53
  SPEAKER_01: And if we want full replication,
• 1:03:54
  SPEAKER_01: it might increase our latency because we need to wait for it to write for both
• 1:03:59
  SPEAKER_01: of them,
• 1:03:59
  SPEAKER_01: which might mean slower,
• 1:04:01
  SPEAKER_01: you know, customers might see Anyway, there's all of these trade-offs.
• 1:04:05
  SPEAKER_01: And in my experience, oftentimes, you might retrofit that.
• 1:04:09
  SPEAKER_01: But if you can have a discussion up front, that's great.
• 1:04:12
  SPEAKER_01: But to have this discussion, you need to know what the trade-offs are.
• 1:04:16
  SPEAKER_01: You need to know what's happening in all these dimensions that you care about.
• 1:04:21
  SPEAKER_01: How long will it take to build?
• 1:04:22
  SPEAKER_01: How complex will it be?
• 1:04:24
  SPEAKER_01: How much money will it cost to operate?
• 1:04:26
  SPEAKER_00: But also as technologists...
• 1:04:28
  SPEAKER_00: We can't just say, oh,
• 1:04:31
  SPEAKER_00: we have also got to get good at communicating with our business stakeholders,
• 1:04:35
  SPEAKER_00: with our end users.
• 1:04:36
  SPEAKER_00: What percentile latency do you want?
• 1:04:38
  SPEAKER_00: Like, some developers know what I say when I say that.
• 1:04:41
  SPEAKER_00: Not all, but some.
• 1:04:43
  SPEAKER_00: How many, you know...
• 1:04:45
  SPEAKER_00: Experts in their field who are great product owners or subject matter experts
• 1:04:49
  SPEAKER_00: but are not software developers would know what I meant by that.
• 1:04:52
  SPEAKER_00: And so we also have to understand the business domain,
• 1:04:56
  SPEAKER_00: the product domain,
• 1:04:57
  SPEAKER_00: well enough to do the interpretation and help and to bridge that gap.
• 1:05:01
  SPEAKER_00: And it is a dynamic thing as well.
• 1:05:04
  SPEAKER_00: I mean, this is the whole thing about resiliency in general, right?
• 1:05:06
  SPEAKER_00: A resilient system is one that changes.
• 1:05:08
  SPEAKER_00: It changes and it evolves.
• 1:05:09
  SPEAKER_00: If you don't change, it evolves through a brittle system.
• 1:05:11
  SPEAKER_00: You're not a resilient system.
• 1:05:13
  SPEAKER_00: And so even those targets,
• 1:05:15
  SPEAKER_00: that conversation we have to have about what is good enough does change.
• 1:05:19
  SPEAKER_00: But think about something like in the case of,
• 1:05:21
  SPEAKER_00: say, the US East One outage towards the end of last year.
• 1:05:24
  SPEAKER_01: There were lots of... On AWS.
• 1:05:26
  SPEAKER_00: On AWS.
• 1:05:26
  SPEAKER_00: So AWS, US East One region went down.
• 1:05:29
  SPEAKER_00: It was the oldest cloud region.
• 1:05:30
  SPEAKER_00: Big region.
• 1:05:31
  SPEAKER_00: Lots of people were there first because that's often
• 1:05:33
  SPEAKER_00: where new services got launched historically.
• 1:05:36
  SPEAKER_00: When that went down, it caused a lot of people to say, do we need to be multi-region?
• 1:05:42
  SPEAKER_00: And some of those companies would have had the conversation before and decided, no, we don't.
• 1:05:46
  SPEAKER_00: And then they'd seen the fallout and they may have changed their minds.
• 1:05:48
  SPEAKER_00: And that is fine.
• 1:05:50
  SPEAKER_01: This is what you mean by a resilient system evolves.
• 1:05:53
  SPEAKER_00: Yeah.
• 1:05:53
  SPEAKER_00: Yeah.
• 1:05:55
  SPEAKER_00: But also like it is about what problems you're not going to try and solve.
• 1:05:59
  SPEAKER_00: There were some big name companies that got taken out when US East 1 went down
• 1:06:03
  SPEAKER_00: and got severely, had severe degradation in terms of what they could offer.
• 1:06:09
  SPEAKER_00: Companies like Zoom,
• 1:06:09
  SPEAKER_00: for example, they weren't taken off entirely, but they were severely affected.
• 1:06:12
  SPEAKER_00: Do we think that people working at Zoom are idiots?
• 1:06:14
  SPEAKER_00: No.
• 1:06:15
  SPEAKER_00: Do we think they knew that if a cloud region went down, they'd be impacted?
• 1:06:18
  SPEAKER_00: Yes.
• 1:06:19
  SPEAKER_00: They probably looked at that and said, well,
• 1:06:21
  SPEAKER_00: for us to come out with a multi-region approach, It's going to have this impact.
• 1:06:25
  SPEAKER_00: It's going to cost us this much money.
• 1:06:27
  SPEAKER_00: It's not worth it.
• 1:06:28
  SPEAKER_00: Let alone talking multi-cloud.
• 1:06:31
  SPEAKER_00: Because multi-cloud versus multi-reach is a whole different conversation.
• 1:06:35
  SPEAKER_00: And that's okay.
• 1:06:35
  SPEAKER_00: And I think as long as you've had the conversation,
• 1:06:37
  SPEAKER_00: and as long as,
• 1:06:39
  SPEAKER_00: this is the other secret bit of it,
• 1:06:40
  SPEAKER_00: of course,
• 1:06:40
  SPEAKER_00: is as long as what you think you need to do is in tune
• 1:06:44
  SPEAKER_00: with what your users think you need to be doing.
• 1:06:46
  SPEAKER_00: This is why I make the point in the book when talking about the difference
• 1:06:49
  SPEAKER_00: between SLAs and SLOs.
• 1:06:50
  SPEAKER_00: The SLA is what you're going to get sued for if you breach.
• 1:06:53
  SPEAKER_00: If you only ever deliver to the SLA, your customers are rarely ever happy.
• 1:06:57
  SPEAKER_00: You've got to do more than that, right?
• 1:06:58
  SPEAKER_00: So there is the social contract there as well.
• 1:07:01
  SPEAKER_01: Now, with building distributed systems,
• 1:07:03
  SPEAKER_01: building resilient systems,
• 1:07:05
  SPEAKER_01: it really helps to understand concepts that are not as straightforward.
• 1:07:09
  SPEAKER_01: And I'm not going to go through all the things.
• 1:07:10
  SPEAKER_01: You explain a lot of things in the book.
• 1:07:11
  SPEAKER_01: But I do want to touch on a concept that I found a bit tricky to understand and then appreciate.
• 1:07:17
  SPEAKER_01: And you really appreciate in distributed systems, which is item potency.
• 1:07:20
  SPEAKER_01: What is idempotency?
• 1:07:22
  SPEAKER_00: So there's a mathematical concept behind idempotency that I don't understand.
• 1:07:25
  SPEAKER_00: But basically,
• 1:07:27
  SPEAKER_00: an idempotent operation is one that you can apply that operation multiple times
• 1:07:32
  SPEAKER_00: without it actually having an effect multiple times.
• 1:07:35
  SPEAKER_01: So it has an effect once, first time.
• 1:07:38
  SPEAKER_00: A really good example of an idempotent operation would be pressing a lift button.
• 1:07:44
  SPEAKER_00: So you want to go down in the lift.
• 1:07:46
  SPEAKER_00: You press the down button, it lights up.
• 1:07:49
  SPEAKER_00: If I press that down button again, nothing's changed.
• 1:07:51
  SPEAKER_00: The lift is still coming.
• 1:07:52
  SPEAKER_00: Well, we hope the lift is still coming, right?
• 1:07:55
  SPEAKER_00: That's an idempotent operation.
• 1:07:57
  SPEAKER_00: Some light switches are actually toggles.
• 1:07:59
  SPEAKER_00: I toggle it, light goes on.
• 1:08:00
  SPEAKER_00: Toggle it, light goes off.
• 1:08:01
  SPEAKER_00: That's not an idempotent operation.
• 1:08:04
  SPEAKER_00: In computing, we have some operations which are naturally idempotent reads.
• 1:08:08
  SPEAKER_00: A delete is typically idempotent.
• 1:08:11
  SPEAKER_00: But the example I think I use when I introduce this concept is one of payment.
• 1:08:16
  SPEAKER_00: I was about to say that I have a lot better item potent example for you.
• 1:08:20
  SPEAKER_00: Yeah, payment is a good one.
• 1:08:21
  SPEAKER_00: So I'm going to pay you money.
• 1:08:22
  SPEAKER_00: So I'm a service that wants to issue a payment and I send you, please pay Sam £100.
• 1:08:30
  SPEAKER_00: I'm NLPing people back at home.
• 1:08:32
  SPEAKER_00: Please say pay Sam £100.
• 1:08:34
  SPEAKER_00: And I don't get a confirmation back.
• 1:08:36
  SPEAKER_00: I don't get back a response.
• 1:08:38
  SPEAKER_00: The problem is from my point of view, there are two possibilities.
• 1:08:41
  SPEAKER_00: The first possibility is that you receive that payment And you've done the payment
• 1:08:46
  SPEAKER_00: and it's been completed and the result has got lost to me.
• 1:08:49
  SPEAKER_00: For some reason, I didn't get the response.
• 1:08:51
  SPEAKER_00: That's one possibility.
• 1:08:53
  SPEAKER_00: The other possibility is that you never got that.
• 1:08:55
  SPEAKER_00: You never carried out the operation.
• 1:08:57
  SPEAKER_00: Maybe it didn't get to you.
• 1:08:58
  SPEAKER_00: Maybe you crashed halfway through.
• 1:08:59
  SPEAKER_01: I think like a good example that I always,
• 1:09:03
  SPEAKER_01: I've seen it in the past,
• 1:09:04
  SPEAKER_01: maybe it's over,
• 1:09:05
  SPEAKER_01: but airlines, you're about to buy this airline ticket for $1,000, $2,000.
• 1:09:09
  SPEAKER_01: You've checked all of your details and it says like play or reserve.
• 1:09:13
  SPEAKER_01: And then often some airlines would take you to an intermediate page which says like,
• 1:09:17
  SPEAKER_01: please do not refresh this page or submit it again
• 1:09:21
  SPEAKER_01: because you might be double charged.
• 1:09:22
  SPEAKER_01: And what you really want is you pay,
• 1:09:24
  SPEAKER_01: you know,
• 1:09:24
  SPEAKER_01: like pay,
• 1:09:26
  SPEAKER_01: And if you get the spinner and it's not idempotent if you do it again
• 1:09:30
  SPEAKER_01: and you get charged twice.
• 1:09:32
  SPEAKER_01: Yeah.
• 1:09:32
  SPEAKER_01: But it is idempotent if you refresh, you copy the same thing, you submit the exact same form.
• 1:09:39
  SPEAKER_00: Oh, that's interesting.
• 1:09:40
  SPEAKER_00: So then the question then becomes, right?
• 1:09:42
  SPEAKER_00: So I sent you that payment,
• 1:09:43
  SPEAKER_00: book my airline ticket, whatever it is, and I haven't had that response back.
• 1:09:46
  SPEAKER_00: But I really want to make sure that the airline ticket booking goes through.
• 1:09:50
  SPEAKER_00: This is you with the browser or...
• 1:09:52
  SPEAKER_00: The people at home sending Sam 100 pounds, right?
• 1:09:54
  SPEAKER_00: You really want to make sure Sam gets his 100 pounds, right?
• 1:09:57
  SPEAKER_00: Now, when something doesn't work in a distributed system because of rule number two.
• 1:10:02
  SPEAKER_01: Rule number two meaning the thing might not be there.
• 1:10:04
  SPEAKER_00: Rule number one, sometimes things take ages.
• 1:10:07
  SPEAKER_00: And accepting this thing is, you just try it again.
• 1:10:08
  SPEAKER_00: You retry.
• 1:10:09
  SPEAKER_00: Because you really want it to happen.
• 1:10:11
  SPEAKER_00: And in retrying,
• 1:10:13
  SPEAKER_00: there's the danger that If the first operation had been processed successfully
• 1:10:18
  SPEAKER_00: and you didn't know about it, well,
• 1:10:19
  SPEAKER_00: the first operation had been applied successfully and you do it again,
• 1:10:21
  SPEAKER_00: you're not getting paid twice and it were two tickets.
• 1:10:24
  SPEAKER_00: So that's the problem with an idempotency.
• 1:10:26
  SPEAKER_00: You want to make it safe to retry important operations.
• 1:10:31
  SPEAKER_00: So to do that, we need to make an idempotent.
• 1:10:33
  SPEAKER_00: There's two ways to do it, really.
• 1:10:34
  SPEAKER_00: One that's good and that is the one you should do.
• 1:10:38
  SPEAKER_00: But you kind of have to retrofitting is paint.
• 1:10:40
  SPEAKER_00: And one, which is one you just explained, which is useful, but has problems, right?
• 1:10:44
  SPEAKER_00: The first one is idempotency keys.
• 1:10:47
  SPEAKER_00: So you generate, you have a unique ID.
• 1:10:49
  SPEAKER_00: You'd normally use a UUID for this.
• 1:10:52
  SPEAKER_00: And so when you say...
• 1:10:53
  SPEAKER_01: The unique ID, which you generate.
• 1:10:54
  SPEAKER_01: And when you generate two of them, they should be different.
• 1:10:58
  SPEAKER_01: There's like this tiny, tiny,
• 1:10:59
  SPEAKER_01: tiny, tiny, tiny, tiny, Likelihood of clash, which again, what happened, I think.
• 1:11:05
  SPEAKER_00: I know, but I think with UUID 7, I think it's like a 50 nanosecond chance.
• 1:11:09
  SPEAKER_00: But it's something you can generate locally without an additional roundtrip.
• 1:11:12
  SPEAKER_00: Your collision chance is extremely low.
• 1:11:14
  SPEAKER_00: And there's ways to modify that.
• 1:11:16
  SPEAKER_01: You can work with that for the most part.
• 1:11:18
  SPEAKER_00: Unless you're Amazon, probably.
• 1:11:20
  SPEAKER_00: But this is, I mean,
• 1:11:21
  SPEAKER_00: the AWS APIs, you see Stripe, Adium, PayPal, they all use the same key schemes.
• 1:11:28
  SPEAKER_00: So you send this to your UID.
• 1:11:29
  SPEAKER_00: So then if you send that to me, you say, pay Sam, and here's the idempotency key.
• 1:11:34
  SPEAKER_00: I don't send you back a response for whatever reason.
• 1:11:37
  SPEAKER_00: You retry it.
• 1:11:38
  SPEAKER_00: You pass the idempotency key.
• 1:11:39
  SPEAKER_00: If I had carried out the processing, I could look at it and say, oh, have I processed this?
• 1:11:42
  SPEAKER_00: Yes, I have.
• 1:11:43
  SPEAKER_00: Okay, great.
• 1:11:44
  SPEAKER_00: If I have processed it, even if I get a second or a third or a fourth, have you processed this?
• 1:11:48
  SPEAKER_00: I just keep going, yeah, I've processed this.
• 1:11:50
  SPEAKER_00: It's processed.
• 1:11:50
  SPEAKER_00: It's fine.
• 1:11:51
  SPEAKER_00: That's a nice, simple thing to do.
• 1:11:53
  SPEAKER_00: The issue with making things with the idempotency key is it requires both the client
• 1:11:57
  SPEAKER_00: and server to support it.
• 1:11:58
  SPEAKER_00: So it has to be a required field to go through, and it has to be sent to server support.
• 1:12:03
  SPEAKER_00: So if you're trying to retrofit this in, it's a pain.
• 1:12:06
  SPEAKER_01: But if you're building it, use an idempotency key.
• 1:12:09
  SPEAKER_00: Use it, exactly.
• 1:12:09
  SPEAKER_00: If anything needs to be idempotent, do that from the beginning.
• 1:12:12
  SPEAKER_00: The one you sort of mentioned was,
• 1:12:13
  SPEAKER_00: I go back,
• 1:12:14
  SPEAKER_00: I hit refresh,
• 1:12:15
  SPEAKER_00: I fill in all the same fields again, and I submit the airline booking again.
• 1:12:20
  SPEAKER_00: You should know it's a duplicate.
• 1:12:21
  SPEAKER_00: That's another thing you can do.
• 1:12:22
  SPEAKER_00: And this was as good if you want to retrofit item potency in.
• 1:12:26
  SPEAKER_00: So if I've sent the same fields, so the same person, Sam, the same amount, £100.
• 1:12:33
  SPEAKER_00: And I'm sending this through again.
• 1:12:34
  SPEAKER_00: What you can do on the server side is generate with a fingerprint.
• 1:12:37
  SPEAKER_00: So I take these fields.
• 1:12:39
  SPEAKER_00: I then generate a hash that represents those fields.
• 1:12:43
  SPEAKER_00: So if you do a second booking, It looks the same.
• 1:12:46
  SPEAKER_01: And you're using that as a retrofitted item potency key.
• 1:12:49
  SPEAKER_00: Effectively, yeah.
• 1:12:50
  SPEAKER_00: There's a couple of downsides with that.
• 1:12:52
  SPEAKER_00: You do get false negatives with that.
• 1:12:54
  SPEAKER_01: The false negative meaning that?
• 1:12:56
  SPEAKER_00: It might get legitimate second requests and get rejected.
• 1:13:01
  SPEAKER_01: Got it.
• 1:13:02
  SPEAKER_00: So a real world example, I know this happens in Denmark.
• 1:13:05
  SPEAKER_00: It might happen elsewhere.
• 1:13:06
  SPEAKER_00: I'm speaking to people in Denmark who works on a payment system.
• 1:13:09
  SPEAKER_00: You go in,
• 1:13:09
  SPEAKER_00: you buy something for 40 kroner with this particular payment system and give them
• 1:13:13
  SPEAKER_00: the card.
• 1:13:13
  SPEAKER_00: Scan the card, 40 krona, whatever.
• 1:13:14
  SPEAKER_00: If within the next sort of five minutes,
• 1:13:16
  SPEAKER_00: you buy something else of 40 krona in the same shop,
• 1:13:19
  SPEAKER_00: the tilt, the handset rejects it.
• 1:13:21
  SPEAKER_00: If it was 41 krona, it'd be fine.
• 1:13:22
  SPEAKER_00: If it was 39 krona, it'd be fine.
• 1:13:24
  SPEAKER_00: But that's because the chances of you buying something for the exact same money is so remote.
• 1:13:29
  SPEAKER_00: They're doing some basic idempotency to top double taps.
• 1:13:33
  SPEAKER_00: And so when you get situations where operations can legitimately look the same,
• 1:13:37
  SPEAKER_00: but actually be different and carried out at the same time,
• 1:13:40
  SPEAKER_00: So a really good example would be the AWS APIs.
• 1:13:44
  SPEAKER_00: It is completely legitimate for me to, in a for loop, to provision, say, 20 EC2 instances.
• 1:13:51
  SPEAKER_00: Now, there's better ways of doing it than sticking in a for loop, but I could.
• 1:13:54
  SPEAKER_00: So I could iterate through the for loop and spin up 20 identical EC2 instances.
• 1:13:59
  SPEAKER_00: If I just relied on there being a fingerprint, they'd all look the same.
• 1:14:03
  SPEAKER_00: So that's where this stuff breaks down.
• 1:14:06
  SPEAKER_00: And so if you've got situations where legitimately you have operations that look
• 1:14:11
  SPEAKER_00: the same,
• 1:14:12
  SPEAKER_00: but they're actually different,
• 1:14:13
  SPEAKER_00: they can be done close to each other, you have an issue.
• 1:14:15
  SPEAKER_00: If you've got a bit of space around them,
• 1:14:17
  SPEAKER_00: what you can do is hold on to those fingerprints for a period of time,
• 1:14:20
  SPEAKER_00: five minutes, 10 minutes, an hour.
• 1:14:22
  SPEAKER_00: So there's still some potentials you can still let duplicates stick through.
• 1:14:25
  SPEAKER_00: So it's not ideal.
• 1:14:27
  SPEAKER_00: Item potency key is the way to go.
• 1:14:28
  SPEAKER_00: Fingerprints,
• 1:14:29
  SPEAKER_00: when you have to retrofit, if you use them both together, That can be really useful.
• 1:14:34
  SPEAKER_00: So you can pick up situations where people change the parameters between
• 1:14:38
  SPEAKER_00: the retry attempts, which leads systems into confusing.
• 1:14:41
  SPEAKER_00: It's a bit more of a complex topic,
• 1:14:42
  SPEAKER_00: but a lot of public-facing APIs will still generate a fingerprint.
• 1:14:47
  SPEAKER_00: And so if you send,
• 1:14:48
  SPEAKER_00: carry out this operation,
• 1:14:50
  SPEAKER_00: and you give the same idempotency keys you gave before,
• 1:14:52
  SPEAKER_00: but you've changed the parameters,
• 1:14:54
  SPEAKER_00: the server can reject it as being effectively an invalid request.
• 1:14:58
  SPEAKER_00: So that's how those things work together.
• 1:14:59
  SPEAKER_00: They're actually, the ideas are really simple.
• 1:15:00
  SPEAKER_00: I think it's, but it's, it's, And there's some nuances about implementing them.
• 1:15:04
  SPEAKER_00: There's some interesting stuff that crops up in terms of how you store them on
• 1:15:08
  SPEAKER_00: the server side and things.
• 1:15:09
  SPEAKER_00: But the fundamental ideas are pretty straightforward, I think.
• 1:15:12
  SPEAKER_01: Another interesting topic that comes up oftentimes these days,
• 1:15:15
  SPEAKER_01: it feels more often a thundering urge.
• 1:15:17
  SPEAKER_01: Yes.
• 1:15:18
  SPEAKER_01: It's when a system restarts and suddenly it faces this big load.
• 1:15:22
  SPEAKER_01: Specifically,
• 1:15:23
  SPEAKER_01: GitHub had an outage now a few weeks ago where it was networking related
• 1:15:28
  SPEAKER_01: and recovery took a lot longer time because there were retries going on
• 1:15:32
  SPEAKER_01: and it created the thundering herd.
• 1:15:34
  SPEAKER_01: What is a thundering herd?
• 1:15:36
  SPEAKER_00: Genetically, I talk about it in any situation where you've got mismatched capacities.
• 1:15:39
  SPEAKER_00: You've got too many requests for too few resources, right?
• 1:15:42
  SPEAKER_00: Resource pool's not infinite.
• 1:15:44
  SPEAKER_00: Coming back to rule number three.
• 1:15:45
  SPEAKER_01: Yeah, rule number three.
• 1:15:45
  SPEAKER_00: Yeah, so in the chapter,
• 1:15:47
  SPEAKER_00: I talk about a whole bunch of different ones from different types of scenarios,
• 1:15:50
  SPEAKER_00: right?
• 1:15:51
  SPEAKER_00: So you've got everything from the malicious thundering herd,
• 1:15:53
  SPEAKER_00: like a denial of service attack, right?
• 1:15:55
  SPEAKER_00: A digital denial of service attack, malicious attack.
• 1:15:58
  SPEAKER_00: Often short notice, and there's not that much you can do about them.
• 1:16:02
  SPEAKER_00: The answer is use Fastit or Cloudflare, but I go into details about how that happens.
• 1:16:06
  SPEAKER_00: You get things like a kind of internal self-inflicted ones.
• 1:16:09
  SPEAKER_00: So a good example would be something like a cache collapse, right?
• 1:16:14
  SPEAKER_00: A system relies on the cache not only for speed of operation,
• 1:16:19
  SPEAKER_00: but also for reducing load on other components.
• 1:16:22
  SPEAKER_00: Cache gets restarted.
• 1:16:24
  SPEAKER_00: You then have a lot of cash misses.
• 1:16:26
  SPEAKER_00: You're hitting the origin and the origin goes down.
• 1:16:29
  SPEAKER_01: The origin was not expecting such load.
• 1:16:31
  SPEAKER_00: Yeah.
• 1:16:31
  SPEAKER_00: So a common thing there would be you first,
• 1:16:34
  SPEAKER_00: you'd flush those cash snapshots to disk,
• 1:16:36
  SPEAKER_00: reduces the chance of you being out or having high cash misses over a restart.
• 1:16:40
  SPEAKER_00: But also a lot of companies I know will just turn their services off until the cash refills.
• 1:16:43
  SPEAKER_00: And so that's kind of another example of why RetryStorm is another one.
• 1:16:48
  SPEAKER_00: Back in 2017, Square had a big outage.
• 1:16:52
  SPEAKER_00: They'd restarted Multipass, which was their multi-factor authentication system.
• 1:16:55
  SPEAKER_00: Multipass,
• 1:16:56
  SPEAKER_00: in turn,
• 1:16:56
  SPEAKER_00: pulled its data from Redis that they were using as a database,
• 1:16:59
  SPEAKER_00: because you can do that, apparently.
• 1:17:01
  SPEAKER_00: I don't think it's a database.
• 1:17:02
  SPEAKER_01: They were using it as a cache, basically, right?
• 1:17:04
  SPEAKER_00: Well, no, I think they were using it as a database.
• 1:17:06
  SPEAKER_00: There are some people that think Redis is a database.
• 1:17:08
  SPEAKER_00: This is a different podcast.
• 1:17:10
  SPEAKER_00: I'm thinking of starting a stupid technology podcast where we talk
• 1:17:12
  SPEAKER_00: about using technology in stupid ways.
• 1:17:13
  SPEAKER_00: That's a separate conversation.
• 1:17:14
  SPEAKER_00: Anyway, so they were putting this data back from Redis,
• 1:17:17
  SPEAKER_00: and they had a problem that the clients in multipass were going,
• 1:17:21
  SPEAKER_00: were basically,
• 1:17:22
  SPEAKER_00: if the Redis calls failed,
• 1:17:24
  SPEAKER_00: they'd try again,
• 1:17:25
  SPEAKER_00: but without any delay between the retries,
• 1:17:27
  SPEAKER_00: and they had a hard-coded limit of 500 retries.
• 1:17:30
  SPEAKER_00: So they just basically smashed Redis and they kept restarting Redis and
• 1:17:33
  SPEAKER_00: then they'd restart Multipass.
• 1:17:35
  SPEAKER_00: And again, this is a long time ago.
• 1:17:36
  SPEAKER_00: And they fixed it, right?
• 1:17:37
  SPEAKER_00: They fixed it by having a more sensible retry policy and externalizing those values.
• 1:17:41
  SPEAKER_00: That's another thundering herd.
• 1:17:42
  SPEAKER_00: The last kind of sections I talked about were ones caused
• 1:17:45
  SPEAKER_00: by products being too successful, like Twitter and healthcare.gov.
• 1:17:52
  SPEAKER_00: This is where you have an external, real, legitimate load that takes you out.
• 1:17:57
  SPEAKER_00: And the thing is,
• 1:17:58
  SPEAKER_00: when you look at all of these as solutions,
• 1:18:00
  SPEAKER_00: although what's happened is the same, the kind of solutions are a bit different.
• 1:18:03
  SPEAKER_00: So like,
• 1:18:04
  SPEAKER_00: when I've got a thundering herd that's coming from externally that's legitimate,
• 1:18:07
  SPEAKER_00: well, I've got different things I'll look at.
• 1:18:09
  SPEAKER_00: Like, I'll have a more constrained sign-up.
• 1:18:11
  SPEAKER_00: I'll have fewer invite codes.
• 1:18:14
  SPEAKER_00: So with HealthcareGov,
• 1:18:15
  SPEAKER_00: one of the issues,
• 1:18:15
  SPEAKER_00: there was many others, but was that they went live in like 36 states at once.
• 1:18:20
  SPEAKER_00: And they were massively overloaded.
• 1:18:22
  SPEAKER_00: So in hindsight, maybe going live in one state would have been a good idea first.
• 1:18:26
  SPEAKER_00: For example, with Twitter, they were going down all the time.
• 1:18:30
  SPEAKER_00: So they could have constrained signups or put tweet limits on you and things like that.
• 1:18:34
  SPEAKER_00: Those are things.
• 1:18:35
  SPEAKER_00: And they did those things as
• 1:18:36
  SPEAKER_01: well, right?
• 1:18:36
  SPEAKER_00: And so although they look the same, The mitigations for Eton was a little bit different.
• 1:18:42
  SPEAKER_00: So I kind of quite like talking about that because they were all great stories I found.
• 1:18:46
  SPEAKER_00: And of course, I think we're talking about
• 1:18:47
  SPEAKER_01: solving it with
• 1:18:51
  SPEAKER_01: When you have same resources, but then we get into things like scaling.
• 1:18:54
  SPEAKER_01: Like, can you scale horizontally, add more resources?
• 1:18:58
  SPEAKER_01: Because there's two sides of this, right?
• 1:19:00
  SPEAKER_01: Like whenever you have it.
• 1:19:01
  SPEAKER_01: And so what point can you do it?
• 1:19:02
  SPEAKER_01: Or are you going to face a bottleneck because you still have a central whatever?
• 1:19:06
  SPEAKER_00: But like, that's a great example of where sometimes mitigations work and sometimes they don't.
• 1:19:10
  SPEAKER_00: So an example of something like your product is successful,
• 1:19:13
  SPEAKER_00: if you're a startup or whatever,
• 1:19:14
  SPEAKER_00: and you suddenly got successful, that success comes like a wave, right?
• 1:19:18
  SPEAKER_00: The wave of success hits you.
• 1:19:20
  SPEAKER_00: And then a lot of it recedes.
• 1:19:22
  SPEAKER_00: You'll get a thousand customers in and maybe you'll keep a hundred.
• 1:19:27
  SPEAKER_00: Maybe, right?
• 1:19:27
  SPEAKER_00: That's just how things work.
• 1:19:29
  SPEAKER_00: So you need to be up during the peak,
• 1:19:32
  SPEAKER_00: but then you probably can shrink down again to keep the cost low.
• 1:19:36
  SPEAKER_00: So if you've built your application to have some dynamic scalability in it,
• 1:19:39
  SPEAKER_00: and if you're on the public cloud or a similar place where you can spin up
• 1:19:42
  SPEAKER_00: and spin down, that's a viable mechanism, right?
• 1:19:45
  SPEAKER_00: Okay, we can scale up to handle this and that's fine.
• 1:19:50
  SPEAKER_00: Would I use that as defense against a distributed denial of service attack?
• 1:19:53
  SPEAKER_00: No,
• 1:19:54
  SPEAKER_00: because then your distributed denial of service attack becomes a distributed denial
• 1:19:57
  SPEAKER_00: of cash attack, right?
• 1:19:59
  SPEAKER_00: You go from to not only having no service,
• 1:20:01
  SPEAKER_00: but also now no money because you hit spinning up,
• 1:20:03
  SPEAKER_00: you know, nodes on your Kubernetes cluster and Amazon just keep taking the money.
• 1:20:08
  SPEAKER_00: It's great.
• 1:20:09
  SPEAKER_00: And you still can't keep up.
• 1:20:11
  SPEAKER_00: So that's a perfectly valid mechanism for For some things,
• 1:20:16
  SPEAKER_00: but you kind of have to know the context in which
• 1:20:18
  SPEAKER_00: to say you can't just get yourself out of trouble.
• 1:20:21
  SPEAKER_01: I think we're going back to a lot of times as like as an engineer,
• 1:20:25
  SPEAKER_01: You want to know about the business.
• 1:20:27
  SPEAKER_01: You want to know what creates a value,
• 1:20:29
  SPEAKER_01: what creates revenue, what is important for the company versus what is not.
• 1:20:34
  SPEAKER_01: And then where we want to maybe invest to with our infrastructure,
• 1:20:38
  SPEAKER_01: with our reliability, versus where it doesn't matter and it's not worth doing.
• 1:20:44
  SPEAKER_00: It's all about trade-offs.
• 1:20:45
  SPEAKER_00: And I think it's about,
• 1:20:46
  SPEAKER_00: you don't know,
• 1:20:47
  SPEAKER_00: unless you get information out of your production systems,
• 1:20:49
  SPEAKER_00: you don't know how close you are to the red lines.
• 1:20:51
  SPEAKER_00: That's why the observability is key, but it is all about trade-offs.
• 1:20:53
  SPEAKER_00: And it And that's why I do a whole chapter on like formal methods.
• 1:20:57
  SPEAKER_00: I talk about cap and pack elk and I say, look,
• 1:20:59
  SPEAKER_00: these things are complicated,
• 1:21:00
  SPEAKER_00: but actually really all comes down to a conversation with understanding
• 1:21:03
  SPEAKER_00: the business context.
• 1:21:04
  SPEAKER_00: So the example I give there is like if an e-commerce company,
• 1:21:09
  SPEAKER_00: if I want to buy an item online, right,
• 1:21:11
  SPEAKER_00: physical item online,
• 1:21:12
  SPEAKER_00: and if at that moment I want to purchase it,
• 1:21:14
  SPEAKER_00: if you're not sure if you've got it or not, most e-commerce companies...
• 1:21:18
  SPEAKER_00: We'll say, yeah, sure, you can have it.
• 1:21:20
  SPEAKER_00: Because they know there's a good chance they do actually have it in stock.
• 1:21:23
  SPEAKER_00: And if they don't have it in stock, they can get it in and back order it.
• 1:21:26
  SPEAKER_00: And the worst case scenario, they can email and say, sorry, refund you the money, right?
• 1:21:30
  SPEAKER_00: So that's not great.
• 1:21:31
  SPEAKER_00: It's not ideal.
• 1:21:32
  SPEAKER_00: Now, if they said,
• 1:21:33
  SPEAKER_00: we don't know if we've got this item,
• 1:21:34
  SPEAKER_00: therefore we're not going to sell it to you, they're missing out on a sale.
• 1:21:38
  SPEAKER_00: And I could go buy that somewhere else probably.
• 1:21:40
  SPEAKER_00: And they might have had it.
• 1:21:42
  SPEAKER_00: So e-commerce companies in that situation,
• 1:21:44
  SPEAKER_00: because of that sort of understanding of the relationship with the customer and
• 1:21:47
  SPEAKER_00: the impacts if they get it wrong, they're going to try selling the item.
• 1:21:51
  SPEAKER_00: Keep that problem exactly the same, but now change the business context.
• 1:21:54
  SPEAKER_00: Now we're selling concert tickets.
• 1:21:55
  SPEAKER_00: So you're getting tickets to go see Tay-Tay, right?
• 1:21:59
  SPEAKER_00: You're going to go see Taylor Swift.
• 1:22:01
  SPEAKER_00: And we're not sure if you've got the Taylor Swift ticket on it because our link to
• 1:22:04
  SPEAKER_00: that system is down.
• 1:22:05
  SPEAKER_00: But those Taylor Swift tickets are yours.
• 1:22:07
  SPEAKER_00: You're taking your kids, taking your wife.
• 1:22:09
  SPEAKER_00: It's all going to be awesome.
• 1:22:10
  SPEAKER_00: We're going to Portugal.
• 1:22:11
  SPEAKER_00: We're going to go see Taylor Swift.
• 1:22:12
  SPEAKER_00: It's going to be amazing.
• 1:22:13
  SPEAKER_00: And then we email you guys.
• 1:22:14
  SPEAKER_00: Sorry.
• 1:22:15
  SPEAKER_00: We don't actually have those tickets.
• 1:22:16
  SPEAKER_00: Now, the problem with that is,
• 1:22:18
  SPEAKER_00: firstly, The devastation in your organization feels a bit different.
• 1:22:21
  SPEAKER_00: They can't backorder a ticket.
• 1:22:24
  SPEAKER_00: And so in that situation, it's like the impact of the customer is so much greater.
• 1:22:28
  SPEAKER_00: And then they'll book the flight.
• 1:22:29
  SPEAKER_00: Well, you might book the flight.
• 1:22:30
  SPEAKER_00: So what they do is they make a different decision.
• 1:22:32
  SPEAKER_00: So in that situation, if they don't know if they've got the ticket, they don't sell the ticket.
• 1:22:35
  SPEAKER_01: I'm not sure if you cover it in the book, but this is the fail open versus fail close, right?
• 1:22:39
  SPEAKER_01: When you have an error or you're unsure, are you going to go through or not?
• 1:22:43
  SPEAKER_01: And as you say, this can be the decision, the same problem.
• 1:22:46
  SPEAKER_01: Two companies can take a decision.
• 1:22:47
  SPEAKER_01: A company years down the road could take a decision.
• 1:22:50
  SPEAKER_01: Uber had this classic example of when payment changes We weren't sure if
• 1:22:55
  SPEAKER_01: the payment went through.
• 1:22:56
  SPEAKER_01: Do we allow the ride to continue or not?
• 1:22:59
  SPEAKER_01: And early in the company's years, we always failed open.
• 1:23:02
  SPEAKER_01: Yeah, just have the ride.
• 1:23:04
  SPEAKER_01: We might have the money, and if not, we'll deal with it.
• 1:23:07
  SPEAKER_01: As the company starts to go for profitability, We feel closed.
• 1:23:11
  SPEAKER_01: We're not sure we have the money.
• 1:23:12
  SPEAKER_01: If the charge went through, you're not going to get the ride.
• 1:23:14
  SPEAKER_01: And again, same company, different focus.
• 1:23:17
  SPEAKER_01: And if you're a developer,
• 1:23:19
  SPEAKER_01: you needed to pay attention to what is important right now for the business.
• 1:23:22
  SPEAKER_01: It was growth initially, and it's okay if we lose money.
• 1:23:24
  SPEAKER_01: And then now it's, I don't know, unit economics, profitability.
• 1:23:27
  SPEAKER_00: But also that open close versus open,
• 1:23:30
  SPEAKER_00: that's a failed close versus a failed open discussion isn't one size fits all for
• 1:23:34
  SPEAKER_00: the whole system.
• 1:23:36
  SPEAKER_00: It's on a feature by feature basis.
• 1:23:37
  SPEAKER_00: And even then, you could say this taxi ride is 10 euros or it's a thousand euros.
• 1:23:44
  SPEAKER_00: You can make different decisions.
• 1:23:45
  SPEAKER_00: Now, if you approach this from a mathematical and from a computer science viewpoint,
• 1:23:50
  SPEAKER_00: we just talked about cat theory.
• 1:23:52
  SPEAKER_00: That was a conversation about cat theory.
• 1:23:54
  SPEAKER_00: Right, so in the chapter I talk about,
• 1:23:55
  SPEAKER_00: here's CAP, here's Pac-Elk, which is an extension of CAP.
• 1:23:58
  SPEAKER_00: I explain what those models are because when you get deep in distribution,
• 1:24:00
  SPEAKER_00: you'll hear about them.
• 1:24:02
  SPEAKER_00: But fundamentally, that's what this is about.
• 1:24:04
  SPEAKER_00: Right, when you have a partition, that's what the P in CAP really defines.
• 1:24:08
  SPEAKER_00: I can't talk to one part of my system.
• 1:24:10
  SPEAKER_00: Remember, it's rule number two.
• 1:24:12
  SPEAKER_00: When I can't talk to another part of my system, what do I do?
• 1:24:16
  SPEAKER_00: And all you need to know as a developer, I can't talk to that.
• 1:24:19
  SPEAKER_00: What do I do?
• 1:24:20
  SPEAKER_00: Sometimes we can make technical decisions.
• 1:24:23
  SPEAKER_00: I'll try again.
• 1:24:25
  SPEAKER_00: I hope I don't bring my systems down for the retry storm, but I'll try again.
• 1:24:28
  SPEAKER_00: If that doesn't work, what do I do?
• 1:24:30
  SPEAKER_00: Well, actually,
• 1:24:30
  SPEAKER_00: often the answer is, yeah, sure,
• 1:24:32
  SPEAKER_00: you can look at CAP and PAC-ELC or whatever else,
• 1:24:34
  SPEAKER_00: but go have a conversation with a product owner.
• 1:24:36
  SPEAKER_00: What should we do?
• 1:24:38
  SPEAKER_00: The resolution of that often ends up being a decision
• 1:24:40
  SPEAKER_00: about are we trading off consistency versus availability in pack health or harvest
• 1:24:44
  SPEAKER_00: and yield if you're going into that version of Brewer's Conjecture.
• 1:24:47
  SPEAKER_00: But don't worry about that.
• 1:24:48
  SPEAKER_00: That's in the book.
• 1:24:49
  SPEAKER_00: That's why it's a later chapter.
• 1:24:50
  SPEAKER_00: That's chapter like 12, I think, or 11.
• 1:24:53
  SPEAKER_00: Don't start there.
• 1:24:54
  SPEAKER_00: Start with having the conversation.
• 1:24:56
  SPEAKER_00: Rule number two, I can't talk to this part of the system right now.
• 1:24:58
  SPEAKER_00: What should we do?
• 1:25:00
  SPEAKER_00: And that starts with having the chat.
• 1:25:03
  SPEAKER_00: That starts having the chat.
• 1:25:04
  SPEAKER_01: When we're talking about resilience, you talk about the dimensions to think about resilience.
• 1:25:10
  SPEAKER_01: What are these dimensions?
• 1:25:11
  SPEAKER_00: I start in the first chapter by coming back to that paper by David Woods I mentioned
• 1:25:15
  SPEAKER_00: that was impenetrable to me at the start.
• 1:25:17
  SPEAKER_00: And he comes up with...
• 1:25:19
  SPEAKER_00: Four concepts for resilience engineering.
• 1:25:22
  SPEAKER_00: And I like the four concepts he comes up with because he helps us think broadly, right?
• 1:25:26
  SPEAKER_00: So it starts off with robustness.
• 1:25:27
  SPEAKER_00: This is where we can basically, I think in his words, absorb known perturbation.
• 1:25:32
  SPEAKER_00: So we model a system and something goes wrong and we can kind of just deal with it
• 1:25:37
  SPEAKER_00: in a way that none cares about, doesn't even know.
• 1:25:39
  SPEAKER_00: So a very simple example would be in Kubernetes, a pod fails and Kubernetes means a replacement.
• 1:25:46
  SPEAKER_00: Now, the pod failing, that's the perturbation.
• 1:25:48
  SPEAKER_00: It's gone a bit wrong, but we've dealt with it.
• 1:25:51
  SPEAKER_00: Great.
• 1:25:51
  SPEAKER_00: So that's the start.
• 1:25:52
  SPEAKER_00: And the thing is, most engineers I speak to, they focus entirely on this.
• 1:25:55
  SPEAKER_00: They don't look at the other three.
• 1:25:57
  SPEAKER_00: They focus entirely on this.
• 1:25:59
  SPEAKER_00: One of the really nasty things about robustness is the things we often do
• 1:26:02
  SPEAKER_00: to make our systems more robust in turn increases the complexity of our system,
• 1:26:07
  SPEAKER_00: which in turn increases the surface area for new types of failure.
• 1:26:11
  SPEAKER_00: I bring Kubernetes in to deal with the failure of a container.
• 1:26:14
  SPEAKER_00: Now I've got Kubernetes and that becomes... And that's not solvable, by the way.
• 1:26:17
  SPEAKER_00: That's just something to be aware of.
• 1:26:19
  SPEAKER_00: That is also why sometimes people don't go multi-region or multi-cloud
• 1:26:23
  SPEAKER_00: because it's increasing the surface area of other failure types.
• 1:26:26
  SPEAKER_00: So that's number one, it's robustness.
• 1:26:28
  SPEAKER_00: And for things to be robust, we have to know about them ahead of time.
• 1:26:32
  SPEAKER_00: This is predictable, we deal with it.
• 1:26:34
  SPEAKER_00: Then there's rebound,
• 1:26:36
  SPEAKER_00: which is when something does happen and goes wrong, how quickly can we recover?
• 1:26:39
  SPEAKER_01: So this is typically the time to recovery, well, time to detection and time to recovery.
• 1:26:45
  SPEAKER_01: Yeah, and there's different ways of
• 1:26:46
  SPEAKER_00: measuring that.
• 1:26:47
  SPEAKER_00: And I talk about in the DR chapter, I talk about, you know, disaster recovery.
• 1:26:51
  SPEAKER_00: Exactly, yeah.
• 1:26:52
  SPEAKER_00: But that's like, how quickly can we get back up and running?
• 1:26:54
  SPEAKER_00: So we've gone into a degraded functionality, how quickly can we get up and running?
• 1:26:57
  SPEAKER_00: So that's thinking about the rebound.
• 1:26:59
  SPEAKER_00: And again,
• 1:27:00
  SPEAKER_00: for that,
• 1:27:00
  SPEAKER_00: you have to be very humble as a developer,
• 1:27:02
  SPEAKER_00: because the robustness is, the system will never fail.
• 1:27:04
  SPEAKER_00: And it's like, but what if we got it wrong?
• 1:27:06
  SPEAKER_00: Okay, well, we have to admit things go outside our control.
• 1:27:08
  SPEAKER_01: And I guess for recovery on how you can recover,
• 1:27:11
  SPEAKER_01: this is where,
• 1:27:11
  SPEAKER_01: I mean, if you want to do it well, teams that are serious about this, they do...
• 1:27:17
  SPEAKER_01: Simulations of the system going down.
• 1:27:18
  SPEAKER_01: This is where Netflix's very famous chaos monkey comes in,
• 1:27:21
  SPEAKER_01: where it goes and shuts off parts of the system to test this live.
• 1:27:24
  SPEAKER_01: And there's drills that are being done on how our on-call process works,
• 1:27:29
  SPEAKER_01: how our decision-making, etc.
• 1:27:31
  SPEAKER_01: This is actually a hard problem.
• 1:27:32
  SPEAKER_01: And then you did it once,
• 1:27:34
  SPEAKER_01: that's not enough because you need to do it regularly because
• 1:27:36
  SPEAKER_01: the system keeps changing.
• 1:27:37
  SPEAKER_00: I'd like to give a shout out to Uptime Labs, who are based here partly in London.
• 1:27:40
  SPEAKER_00: They do excellent, instant training that you can do online in a web browser.
• 1:27:43
  SPEAKER_00: It's fantastic.
• 1:27:44
  SPEAKER_00: We'll add them to
• 1:27:45
  SPEAKER_01: the show notes below.
• 1:27:47
  SPEAKER_00: They're nice people.
• 1:27:48
  SPEAKER_00: So you've got that recovery process, like how do we get up and running?
• 1:27:52
  SPEAKER_00: And I think kind of techies are with me.
• 1:27:54
  SPEAKER_00: And then we go into the next two, which the third one is graceful extensibility.
• 1:27:59
  SPEAKER_00: And so graceful extensibility...
• 1:28:01
  SPEAKER_00: Again, it's a slightly weird name, but it's basically dealing with surprise.
• 1:28:04
  SPEAKER_00: When we make our system robust, we're dealing with the known problems.
• 1:28:10
  SPEAKER_00: What about things we don't expect?
• 1:28:12
  SPEAKER_00: If we don't expect it, this is now not about computers, it's about humans.
• 1:28:16
  SPEAKER_00: This is why distributed systems are fundamentally associated with technical systems,
• 1:28:19
  SPEAKER_00: people and technology are working together.
• 1:28:20
  SPEAKER_00: We couldn't expect the rabbits.
• 1:28:22
  SPEAKER_00: How do we deal with the rabbits?
• 1:28:24
  SPEAKER_00: How do we deal with a farm in the data center?
• 1:28:25
  SPEAKER_00: Turns out badly is the answer.
• 1:28:27
  SPEAKER_00: But if you're in a corporate environment,
• 1:28:29
  SPEAKER_00: which is very command and control,
• 1:28:32
  SPEAKER_00: quite hierarchical,
• 1:28:33
  SPEAKER_00: where you've got a very narrow job description,
• 1:28:35
  SPEAKER_00: you're often constrained in what you're able to do.
• 1:28:38
  SPEAKER_00: Those organizations tend to deal with surprise badly because
• 1:28:42
  SPEAKER_00: the corporate structure is structured around the known.
• 1:28:45
  SPEAKER_00: And so often structurally, you'll find that difficult.
• 1:28:48
  SPEAKER_00: You mentioned drills earlier.
• 1:28:49
  SPEAKER_00: Drills are a great way of helping with surprise.
• 1:28:52
  SPEAKER_00: Coming in and having people often from outside saying,
• 1:28:54
  SPEAKER_00: this has happened, how will you deal with it?
• 1:28:56
  SPEAKER_00: Google had a thing,
• 1:28:56
  SPEAKER_00: I don't know if they still do it,
• 1:28:57
  SPEAKER_00: SREs would pop around and they'd do a thing called the Wheel of Misfortune.
• 1:29:00
  SPEAKER_00: They had a physical wheel.
• 1:29:01
  SPEAKER_00: Today, your system has suffered a spin the wheel.
• 1:29:05
  SPEAKER_00: DNS poisoning attack by a state, insert name of rogue state here.
• 1:29:09
  SPEAKER_00: Here's what's happening.
• 1:29:10
  SPEAKER_00: How do you deal with it?
• 1:29:12
  SPEAKER_00: That is all about Like people and experiences and trying new things
• 1:29:17
  SPEAKER_00: and constantly pushing yourselves.
• 1:29:19
  SPEAKER_01: I guess learning also from others.
• 1:29:20
  SPEAKER_01: I remember this like very famous outage happened a few years ago at Meta
• 1:29:24
  SPEAKER_01: where their DNS system through BGP, it just all went down.
• 1:29:29
  SPEAKER_01: WhatsApp, Meta, everything.
• 1:29:31
  SPEAKER_01: To fix this,
• 1:29:32
  SPEAKER_01: because they pushed out an update to their BGP table,
• 1:29:35
  SPEAKER_01: which is inside DNS to their servers.
• 1:29:38
  SPEAKER_01: Now, to fix this,
• 1:29:39
  SPEAKER_01: they needed to get access to the computer that physically had to get access
• 1:29:43
  SPEAKER_01: for whatever, because they couldn't reach it remotely.
• 1:29:45
  SPEAKER_01: So they needed to go into the building.
• 1:29:47
  SPEAKER_01: Now, in going to the building, they had to go to a door that protected this computer.
• 1:29:52
  SPEAKER_01: System, which was running on their own domain, which was down.
• 1:29:55
  SPEAKER_01: So they had to break physically into.
• 1:29:58
  SPEAKER_01: But again,
• 1:29:58
  SPEAKER_01: this is a thing where they share the details,
• 1:30:00
  SPEAKER_01: which was great because now everyone else could think about like,
• 1:30:03
  SPEAKER_01: do we have any of our physical security facilities?
• 1:30:06
  SPEAKER_01: For example, tied to our own domain.
• 1:30:08
  SPEAKER_01: Because if you have it, you want to change that learning from what meta have learned about.
• 1:30:13
  SPEAKER_00: And this learning takes us from that grace of extensibility, which is dealing with surprise.
• 1:30:17
  SPEAKER_00: That's like sort of almost in the moment.
• 1:30:18
  SPEAKER_00: Are we ready to deal with surprise?
• 1:30:19
  SPEAKER_00: Into the fourth one, which is sustained adaptability.
• 1:30:22
  SPEAKER_00: Which is, how can we learn from what's happened?
• 1:30:25
  SPEAKER_00: Not just the learning.
• 1:30:26
  SPEAKER_00: So do we have the environment in which we can learn and share and grow,
• 1:30:30
  SPEAKER_00: but then also adapt the system based on that learning?
• 1:30:33
  SPEAKER_00: And it's called sustained adaptability because that is a completely continual process.
• 1:30:38
  SPEAKER_00: That's not just today.
• 1:30:39
  SPEAKER_00: That's not just when you have an incident.
• 1:30:41
  SPEAKER_00: That's when you don't have an incident.
• 1:30:42
  SPEAKER_00: What are we doing to learn?
• 1:30:44
  SPEAKER_00: And I think your point you raised there was excellent,
• 1:30:45
  SPEAKER_00: which is I like reading other people's incident reports.
• 1:30:48
  SPEAKER_00: There's a little bit of schadenfreude with companies I don't like,
• 1:30:50
  SPEAKER_00: but there's also a little bit of it is like they've learned something.
• 1:30:55
  SPEAKER_00: Can I learn from them and do something that resolves my system?
• 1:30:59
  SPEAKER_00: So when you don't have an incident, read other people's incident reports.
• 1:31:01
  SPEAKER_00: There's a place called The Void.
• 1:31:03
  SPEAKER_00: It hasn't been updated for a bit.
• 1:31:05
  SPEAKER_00: There's hundreds and hundreds of incident reports there, which is worth going through.
• 1:31:08
  SPEAKER_00: So that idea of sustained adaptability is all about us adapting and changing
• 1:31:13
  SPEAKER_00: over longer timescales.
• 1:31:15
  SPEAKER_00: For those last two things,
• 1:31:16
  SPEAKER_00: sustained adaptability and graceful extensibility,
• 1:31:19
  SPEAKER_00: this all comes down to people and culture.
• 1:31:21
  SPEAKER_00: Having proper environment psychological safety is just Absolutely required.
• 1:31:27
  SPEAKER_00: If you can't raise issues,
• 1:31:29
  SPEAKER_00: if you can't challenge in a safe way,
• 1:31:33
  SPEAKER_00: you will deal with surprise badly because things won't get raised
• 1:31:35
  SPEAKER_00: and you won't learn because things will be hidden.
• 1:31:37
  SPEAKER_00: And so that all comes down to behavior stuff.
• 1:31:40
  SPEAKER_00: And so the last three chapters of the book where you focus on that is looking
• 1:31:43
  SPEAKER_00: at some of the people and the social aspects of that.
• 1:31:46
  SPEAKER_00: So I've got like 14, 13 chapters, or 12 chapters, sorry, about the other technical.
• 1:31:49
  SPEAKER_00: And then the last three chapters are all around that side of things.
• 1:31:52
  SPEAKER_00: So like, What can we do to make sure that we have that environment of psychological safety?
• 1:31:57
  SPEAKER_00: Because without it you can't actually be resilient.
• 1:31:59
  SPEAKER_01: It's pretty amazing because so far throughout this whole conversation,
• 1:32:04
  SPEAKER_01: we talked about building resilient systems.
• 1:32:06
  SPEAKER_01: We talked a lot about people, about how to think about resiliency.
• 1:32:10
  SPEAKER_01: One thing that we didn't really talk about just passively is AI.
• 1:32:13
  SPEAKER_01: Even though this book is coming out now,
• 1:32:16
  SPEAKER_01: and I know it has a section on AI as well,
• 1:32:19
  SPEAKER_01: but it's just interesting to reflect that even
• 1:32:21
  SPEAKER_01: though AI is changing how we're building software,
• 1:32:24
  SPEAKER_01: it seems like some of these ground truths of how to operate reliable software are
• 1:32:28
  SPEAKER_01: a little bit just...
• 1:32:29
  SPEAKER_01: Outside of this realm, right?
• 1:32:30
  SPEAKER_01: Well, I mean... Or is it?
• 1:32:32
  SPEAKER_00: No, it's not.
• 1:32:33
  SPEAKER_00: I mean, firstly, there's some stuff, there's some AI stuff in there that can help.
• 1:32:37
  SPEAKER_00: I detail quite a few examples where around sort of correlation analysis and,
• 1:32:42
  SPEAKER_00: you know,
• 1:32:42
  SPEAKER_00: help support around post-instant,
• 1:32:44
  SPEAKER_00: there's some instant.io doing some amazing stuff in this area.
• 1:32:47
  SPEAKER_01: So you're saying we can use tools to do a lot, bits and bits.
• 1:32:50
  SPEAKER_00: But really, there's more fundamental kind of things here.
• 1:32:52
  SPEAKER_00: And the first thing is the macro and the micro.
• 1:32:53
  SPEAKER_00: At the micro level, there's some useful stuff we can do here.
• 1:32:56
  SPEAKER_00: And I talk about in the book, the macro level is that None of it stacks up financially, right?
• 1:33:01
  SPEAKER_00: So if you're taking a longer view about resilience,
• 1:33:03
  SPEAKER_00: if I'm building my products based on A market sector that is fundamentally flawed.
• 1:33:10
  SPEAKER_00: These companies don't have a viable path to sustainability.
• 1:33:14
  SPEAKER_00: When you say which companies are we talking about?
• 1:33:16
  SPEAKER_00: OpenAI and Anthropik and the hypervisors, right?
• 1:33:19
  SPEAKER_00: Conservative analysis by The Economist are showing that they need
• 1:33:22
  SPEAKER_00: to be generating 2.6 trillion in revenue to justify their capex expenditure by 2030.
• 1:33:28
  SPEAKER_00: The entire software market is 1.4 trillion.
• 1:33:31
  SPEAKER_00: So macro level, I don't know how that gets sorted.
• 1:33:34
  SPEAKER_00: I'm not smart enough.
• 1:33:36
  SPEAKER_00: I also don't have any shares in those companies.
• 1:33:38
  SPEAKER_00: But, you know, that's a separate conversation.
• 1:33:40
  SPEAKER_00: I can't solve that.
• 1:33:41
  SPEAKER_00: What I can start thinking about is,
• 1:33:42
  SPEAKER_00: okay, if that's what I'm operating in, how do I deal with that?
• 1:33:45
  SPEAKER_00: I can't deal with all that.
• 1:33:46
  SPEAKER_00: What I can do if I'm taking a view to resiliency is being very careful about how I use AI.
• 1:33:51
  SPEAKER_00: For example,
• 1:33:52
  SPEAKER_00: you know,
• 1:33:52
  SPEAKER_00: if I'm thinking taking a longer view,
• 1:33:55
  SPEAKER_00: being more flexible about model choices,
• 1:33:57
  SPEAKER_00: that's going to help me because I've got the opportunity to use open-weight models
• 1:34:00
  SPEAKER_00: from different vendors.
• 1:34:01
  SPEAKER_00: Maybe even just basic things like OpenAI and Anthropic are pretty terrible
• 1:34:05
  SPEAKER_00: when it comes to availability of their core models from their services,
• 1:34:08
  SPEAKER_00: which is why everyone I know also will use like,
• 1:34:10
  SPEAKER_00: they'll use Google or Amazon or Microsoft as either their primary place they go
• 1:34:15
  SPEAKER_00: to those models or their fallbacks.
• 1:34:17
  SPEAKER_00: They have better uptimes and things.
• 1:34:19
  SPEAKER_00: So there's all that stuff we've got to deal with at that level.
• 1:34:22
  SPEAKER_00: There's also the conversation about non-determinism.
• 1:34:24
  SPEAKER_00: And Charity pulled me up on this a bit.
• 1:34:26
  SPEAKER_00: And I said,
• 1:34:27
  SPEAKER_00: the thing is that,
• 1:34:27
  SPEAKER_00: you know, we're going from deterministic software to non-determinism.
• 1:34:31
  SPEAKER_00: You know, that's a challenge for us.
• 1:34:32
  SPEAKER_00: We don't know if things work.
• 1:34:33
  SPEAKER_00: And then she said to me, our distributed systems were never deterministic.
• 1:34:37
  SPEAKER_00: And now they're really not deterministic.
• 1:34:39
  SPEAKER_00: And of course, she's right.
• 1:34:39
  SPEAKER_00: Because when we take our distributed system and we add the people, Beyond the rabbits, right?
• 1:34:46
  SPEAKER_00: And the fires and the people.
• 1:34:48
  SPEAKER_00: We always had the degree of non-determinism.
• 1:34:50
  SPEAKER_00: We're just amping that up a bit.
• 1:34:51
  SPEAKER_00: So I think that's a continuation.
• 1:34:53
  SPEAKER_00: So for me, I think it's a confusing landscape.
• 1:34:55
  SPEAKER_00: It's not good or bad.
• 1:34:56
  SPEAKER_00: I think at the micro level, there's some really great things that we can do.
• 1:34:59
  SPEAKER_01: When we say micro level, that's the tactical things that we can do, right?
• 1:35:02
  SPEAKER_00: I think really what I mean by that is I'm taking the macroeconomic policy situation out of it.
• 1:35:06
  SPEAKER_00: I'm saying as developers, there's things that we can make use of, right?
• 1:35:09
  SPEAKER_00: There's some good stuff in there that we can make use of.
• 1:35:11
  SPEAKER_00: But I also think there's some hedging we need to do.
• 1:35:13
  SPEAKER_00: Longer term, if you're building AI into your product...
• 1:35:17
  SPEAKER_00: There's some simple hedging things.
• 1:35:18
  SPEAKER_00: So being multi-vendor,
• 1:35:20
  SPEAKER_00: being multi-model,
• 1:35:22
  SPEAKER_00: for example,
• 1:35:23
  SPEAKER_00: breaking rather than having a single agent,
• 1:35:26
  SPEAKER_00: breaking that down into more workflow models.
• 1:35:28
  SPEAKER_00: Therefore,
• 1:35:28
  SPEAKER_00: you can take bits of your workflow and realize actually this can now be swapped out
• 1:35:32
  SPEAKER_00: for a bit of deterministic code that I write that could run more efficiently.
• 1:35:35
  SPEAKER_00: I think these are all sensible things that you can do.
• 1:35:37
  SPEAKER_00: We can't fix this.
• 1:35:39
  SPEAKER_00: We can at least deal with our world.
• 1:35:40
  SPEAKER_00: And I think there's some sensible things that we can do in that space.
• 1:35:43
  SPEAKER_01: What are the things that we know that we do not know about AI?
• 1:35:48
  SPEAKER_00: I think the tech world in general is quite naive about AI
• 1:35:54
  SPEAKER_00: and fundamentally misunderstands what an LLM is.
• 1:35:58
  SPEAKER_00: Let's be real.
• 1:35:59
  SPEAKER_00: Why did the LLM delete my database?
• 1:36:03
  SPEAKER_00: Well, because it has no concept of causality.
• 1:36:06
  SPEAKER_00: That's what it comes down to.
• 1:36:07
  SPEAKER_00: There's no concept of cause.
• 1:36:09
  SPEAKER_00: There's a glass here.
• 1:36:11
  SPEAKER_00: If I push that glass onto the floor, you'll be working as a carpet down there.
• 1:36:16
  SPEAKER_00: If I ask you what's going to happen, you'll say the glass is going to be fine.
• 1:36:18
  SPEAKER_00: If I push it onto that floor, you're looking at it saying it's a hard floor, that might break.
• 1:36:22
  SPEAKER_00: You're carrying out a bunch of calculations in your head.
• 1:36:25
  SPEAKER_00: You've got an idea about the physical properties of the universe.
• 1:36:27
  SPEAKER_00: You've got a sense if I push the glass, this is going to happen.
• 1:36:29
  SPEAKER_00: LMs don't do any of that.
• 1:36:30
  SPEAKER_00: They have no concept that if I do, A, B happens.
• 1:36:34
  SPEAKER_01: Well, there's a push for world models, but These LLMs are not world models.
• 1:36:38
  SPEAKER_00: They're not world models.
• 1:36:38
  SPEAKER_00: And world models have been around for a long time.
• 1:36:41
  SPEAKER_00: They arguably predate LLMs.
• 1:36:43
  SPEAKER_00: They can do stuff like that.
• 1:36:44
  SPEAKER_00: So they could make decisions like if you have the appropriate model and
• 1:36:47
  SPEAKER_00: the appropriate inputs,
• 1:36:48
  SPEAKER_00: they can at least get to the point of if I do X,
• 1:36:51
  SPEAKER_00: then Y happens and I've been told not to do Y.
• 1:36:53
  SPEAKER_00: This is why all the guardrails around LLMs are really not long-term going to
• 1:36:57
  SPEAKER_00: the right solution.
• 1:36:58
  SPEAKER_00: And so I think we expect LLMs to do more than they can actually do because they seem so smart.
• 1:37:03
  SPEAKER_01: Yeah.
• 1:37:04
  SPEAKER_00: And so I think that's the problem and that leads us
• 1:37:07
  SPEAKER_00: to overestimate their capabilities a bit too much, I think.
• 1:37:12
  SPEAKER_01: Or maybe over rely on them without knowing and we might have some incidents later.
• 1:37:17
  SPEAKER_00: Absolutely, absolutely.
• 1:37:18
  SPEAKER_00: And I think we're knowing where and when to use those things.
• 1:37:23
  SPEAKER_00: And I think what we don't know right now is where should I be having,
• 1:37:26
  SPEAKER_00: and again, this is wider, where should I have the LLM do it?
• 1:37:29
  SPEAKER_00: And where should I just write some deterministic code?
• 1:37:31
  SPEAKER_00: There's a lot to be said for just having your agent call a script and you know
• 1:37:34
  SPEAKER_00: that script works,
• 1:37:35
  SPEAKER_00: rather than having it implement the script in its magical brain for you.
• 1:37:39
  SPEAKER_01: Well, there's something to be said that it seems both labs,
• 1:37:42
  SPEAKER_01: both leading labs, Antropic and OpenAI have realized that Coding is so important.
• 1:37:47
  SPEAKER_01: So oftentimes when you ask the alum to do something,
• 1:37:51
  SPEAKER_01: it will start to generate code because it writes a deterministic program,
• 1:37:55
  SPEAKER_01: because that part will be deterministic.
• 1:37:57
  SPEAKER_01: And it's very interesting when you ask it,
• 1:37:59
  SPEAKER_01: it will often just pump out a Python or TypeScript or whatever,
• 1:38:02
  SPEAKER_01: especially when it has to do with involving compare this or that.
• 1:38:07
  SPEAKER_01: And they have trained these models to like default to coding.
• 1:38:11
  SPEAKER_01: I'm not sure how that this came out.
• 1:38:13
  SPEAKER_01: But it seems to be working to some extent.
• 1:38:16
  SPEAKER_00: It's also because it's very tractable, right?
• 1:38:18
  SPEAKER_00: They are there to generate plausible code.
• 1:38:21
  SPEAKER_00: They're there to generate plausible words, right?
• 1:38:23
  SPEAKER_00: And in the context of code,
• 1:38:25
  SPEAKER_00: they've generated some plausible code that we can now validate, does it execute?
• 1:38:29
  SPEAKER_00: And so that's a really nice, tractable aspect of it.
• 1:38:33
  SPEAKER_00: And there are some places where it works really, really well.
• 1:38:36
  SPEAKER_00: But there are other areas where it's terrible.
• 1:38:37
  SPEAKER_00: Like when I write my book, I use very little AI at all.
• 1:38:42
  SPEAKER_01: Well, you didn't use it for the writing, you used it for research.
• 1:38:44
  SPEAKER_00: I don't use it for the writing at all.
• 1:38:46
  SPEAKER_00: I used it for the research.
• 1:38:48
  SPEAKER_00: But in the context of the research, I used Notebook LM, basically.
• 1:38:51
  SPEAKER_00: But in that context, it's plausible.
• 1:38:53
  SPEAKER_00: But then I just go and click all the links it gives me.
• 1:38:55
  SPEAKER_00: And I say, was that there or not?
• 1:38:56
  SPEAKER_00: I am checking the work.
• 1:38:58
  SPEAKER_00: It's done way more work than I could have done going through all those papers.
• 1:39:01
  SPEAKER_00: And God knows there's only so many NIST papers you can read through, right?
• 1:39:04
  SPEAKER_00: But in that context, I feel completely fine and able to do that.
• 1:39:09
  SPEAKER_00: In other settings,
• 1:39:10
  SPEAKER_00: you're getting now this problem called cognitive surrender,
• 1:39:13
  SPEAKER_00: which is where you're using AI to write documents that you're never going to read.
• 1:39:17
  SPEAKER_00: That's not really around a problem with the LLM technology.
• 1:39:21
  SPEAKER_00: That's the problem with us.
• 1:39:23
  SPEAKER_01: Can we talk about this cognitive surrender and cognitive depth?
• 1:39:27
  SPEAKER_00: Yeah, cognitive depth is a kind of term that's been around for a while.
• 1:39:30
  SPEAKER_00: And that sort of explains the erosion in our ability to Thank you.
• 1:39:44
  SPEAKER_00: Is impacted the more they use chatbots and things.
• 1:39:47
  SPEAKER_00: And I think we've sort of seen other examples.
• 1:39:50
  SPEAKER_00: Margaret Ann Storey's got a great article about this,
• 1:39:52
  SPEAKER_00: and she cited some of the research around it as well.
• 1:39:55
  SPEAKER_00: And that's sort of explaining that, I mean,
• 1:39:57
  SPEAKER_00: her thing is,
• 1:39:57
  SPEAKER_00: you know, she's talking, we talk about technical debt building up in a system.
• 1:40:00
  SPEAKER_00: We have to be careful that our use of AI needs that cognitive debt.
• 1:40:03
  SPEAKER_00: We need to have a shared mental model of what this system is supposed to do,
• 1:40:07
  SPEAKER_00: thinking about it narrowly for the lens of a program that we're building.
• 1:40:11
  SPEAKER_00: And if our use of AI As an example, we all go off in different directions.
• 1:40:16
  SPEAKER_00: That shared review breaks down.
• 1:40:18
  SPEAKER_00: That becomes a debt cognitively.
• 1:40:21
  SPEAKER_00: Cognitive surrender is a different thing, which is like, oh, yeah, it's fine.
• 1:40:26
  SPEAKER_00: It's basically LGTM, right?
• 1:40:29
  SPEAKER_00: But for whatever the AI tells me.
• 1:40:30
  SPEAKER_01: Yeah, LGTM was typical,
• 1:40:32
  SPEAKER_01: looks good to me,
• 1:40:33
  SPEAKER_01: approving large pull requests where users couldn't be bothered or you tried to look
• 1:40:38
  SPEAKER_01: at it and for whatever reason you didn't want to go through or it came from someone
• 1:40:42
  SPEAKER_01: who you
• 1:40:43
  SPEAKER_00: trusted.
• 1:40:43
  SPEAKER_00: I worked in the Google tool team on all that stuff, right?
• 1:40:45
  SPEAKER_00: And so LGTM was literally a sign with pattern match on and takes that as an approval.
• 1:40:50
  SPEAKER_00: So if you saw LGTM, that person's approved, right?
• 1:40:53
  SPEAKER_00: And I think that may even have been where the acronym originally came from, I'm not sure.
• 1:40:56
  SPEAKER_01: I think it came from Google, yeah.
• 1:40:58
  SPEAKER_00: But it is this idea, and it's become shorthand for, I didn't really look at it.
• 1:41:02
  SPEAKER_01: Yeah.
• 1:41:02
  SPEAKER_00: And I think that's that cognitive surrender thing is coming out in lots of ways.
• 1:41:06
  SPEAKER_00: You know, we're getting AI to write documents and no one's going to read.
• 1:41:10
  SPEAKER_00: And we've got AI that's reading documents that weren't written by other people.
• 1:41:15
  SPEAKER_00: And so the cognitive surrender piece is really that the humans,
• 1:41:19
  SPEAKER_00: it comes back to the human being in the loop.
• 1:41:21
  SPEAKER_00: There's very subtle differences in those workflows and to the kinds of differences it changes.
• 1:41:27
  SPEAKER_00: Like there's There's a big difference.
• 1:41:29
  SPEAKER_00: Cory Doctorow talks about it.
• 1:41:30
  SPEAKER_00: It's a big difference.
• 1:41:31
  SPEAKER_00: Like if you're, say,
• 1:41:31
  SPEAKER_00: a cancer,
• 1:41:32
  SPEAKER_00: an oncologist versus cancer specialist,
• 1:41:34
  SPEAKER_00: there's a big difference to you as an oncologist working and an LLM doing sort
• 1:41:39
  SPEAKER_00: of detection of some of the scans that you're looking at going,
• 1:41:41
  SPEAKER_00: oh, you should look over there, by the way, you might have missed that.
• 1:41:44
  SPEAKER_00: That's actually quite useful, right?
• 1:41:45
  SPEAKER_00: Versus the LLM powered AI going, here's all the scans I've looked through.
• 1:41:50
  SPEAKER_00: Here's the ones I need you to sign off.
• 1:41:52
  SPEAKER_00: The last one, the second one, takes the expert out of the loop and makes them an approver.
• 1:41:58
  SPEAKER_00: And often in that context, they've got rid of a bunch of oncologists.
• 1:42:01
  SPEAKER_00: You've got one person in the funnel that's going to lead into issues.
• 1:42:04
  SPEAKER_00: The other model is we're making people better.
• 1:42:07
  SPEAKER_00: I think that's to come back to the original idea of AI or some of the pitch,
• 1:42:11
  SPEAKER_00: right, was it was going to free us from drudgery.
• 1:42:14
  SPEAKER_00: Just like the Industrial Revolution was going to free us from drudgery.
• 1:42:16
  SPEAKER_00: But it would elevate us.
• 1:42:18
  SPEAKER_00: We'd have less toil.
• 1:42:19
  SPEAKER_00: We can focus more on critical thinking and smarts.
• 1:42:22
  SPEAKER_00: And so much of our current use of AI is not freeing us up from critical thinking.
• 1:42:27
  SPEAKER_00: It's causing more context switching.
• 1:42:30
  SPEAKER_00: It's causing less time for thinking.
• 1:42:32
  SPEAKER_00: People are working longer hours.
• 1:42:34
  SPEAKER_00: We've got studies that show this now.
• 1:42:35
  SPEAKER_00: That's not necessarily AI's fault.
• 1:42:38
  SPEAKER_00: It's just How are we using it?
• 1:42:39
  SPEAKER_00: It's like, we need to readdress that a little bit.
• 1:42:41
  SPEAKER_01: Can we talk about complexity and AI maybe adding it?
• 1:42:45
  SPEAKER_01: I'm going to quote you from about it from a few months back.
• 1:42:48
  SPEAKER_01: You wrote,
• 1:42:49
  SPEAKER_01: I worry AI could just become an exercise in doubling down
• 1:42:52
  SPEAKER_01: on current complexity rather than stripping things back to simplify,
• 1:42:55
  SPEAKER_01: at least in the platform space.
• 1:42:56
  SPEAKER_00: Yeah, I can't remember the exact context in which we had that conversation.
• 1:43:00
  SPEAKER_00: But I mean, the platform space was, I mean, the platform space is kind of interesting, right?
• 1:43:03
  SPEAKER_00: Because that was an attempt to make things simpler for developers because...
• 1:43:07
  SPEAKER_00: Kubernetes became popular and it's not developer-friendly.
• 1:43:10
  SPEAKER_00: That's not what it's designed for.
• 1:43:11
  SPEAKER_01: But most developers never have to touch it.
• 1:43:13
  SPEAKER_00: No.
• 1:43:13
  SPEAKER_00: I mean, well, if you have a platform.
• 1:43:15
  SPEAKER_00: If you don't have a platform, yeah.
• 1:43:17
  SPEAKER_00: There's a platform team's problem, right?
• 1:43:19
  SPEAKER_00: And that's why they get paid.
• 1:43:20
  SPEAKER_00: And so I think there was this idea that I was sort of hoping that there would
• 1:43:25
  SPEAKER_00: a bit of a complexity collapse around that space.
• 1:43:27
  SPEAKER_00: I had this hope for things like WSGI,
• 1:43:29
  SPEAKER_00: for example, simplifying down those programming models a little bit.
• 1:43:33
  SPEAKER_00: And I think there was starting to be a bit of a A bit of a conversation is starting
• 1:43:37
  SPEAKER_00: to be had about some of that stuff going on.
• 1:43:39
  SPEAKER_00: What then happens,
• 1:43:40
  SPEAKER_00: of course, is AI has come out and everyone's like, I've got to AI all the things.
• 1:43:44
  SPEAKER_00: And so rather than thinking about what we need and trying to strip
• 1:43:47
  SPEAKER_00: that complexity back,
• 1:43:48
  SPEAKER_00: we've now just got another piece that requires us to still have all
• 1:43:50
  SPEAKER_00: the old complexity of adding new complexity on top.
• 1:43:55
  SPEAKER_00: And somebody's got to look after all that stuff.
• 1:43:57
  SPEAKER_01: Yeah, but I think it goes back to a little bit of like,
• 1:43:59
  SPEAKER_01: we seem to have more work with AI and probably it doesn't help
• 1:44:02
  SPEAKER_01: that it can add more complexity, especially when we're not...
• 1:44:06
  SPEAKER_01: Being clear of what we want,
• 1:44:07
  SPEAKER_01: we're not being critical of the output when we have cognitive depth
• 1:44:11
  SPEAKER_01: or cognitive surrender, especially not with cognitive surrender.
• 1:44:14
  SPEAKER_01: So these things might be amplifying the fact that it's just more complexity, right?
• 1:44:19
  SPEAKER_00: Yeah, and I think there's also that connection through to what the AI is doing.
• 1:44:23
  SPEAKER_00: It still comes full circle a little bit,
• 1:44:25
  SPEAKER_00: which is the idea of potentially not looking at the code isn't to say
• 1:44:29
  SPEAKER_00: that you don't care about what the system is that you're building.
• 1:44:32
  SPEAKER_00: Here's to say that you've got a better way of understanding what you're building.
• 1:44:36
  SPEAKER_00: So one of the concerns is if I'm using AI to create code,
• 1:44:39
  SPEAKER_00: it's going to create awful loads of debt and a complicated system.
• 1:44:41
  SPEAKER_00: No one's going to understand.
• 1:44:43
  SPEAKER_00: Well, we can't let that happen.
• 1:44:45
  SPEAKER_00: We still need to remain connected to the system that we're building,
• 1:44:48
  SPEAKER_00: which is what I'm a big fan of, like modularizing system.
• 1:44:51
  SPEAKER_00: Whether or not you put those modules in different computers, I don't actually care.
• 1:44:55
  SPEAKER_00: Getting AI to work within the modules,
• 1:44:56
  SPEAKER_00: having the humans think carefully about what those modules are and the connections
• 1:45:00
  SPEAKER_00: between them.
• 1:45:01
  SPEAKER_00: Let the AI go hog wild inside those boundaries that are easy to change.
• 1:45:06
  SPEAKER_00: The gaps between them,
• 1:45:07
  SPEAKER_00: that's where the human beings spend to think about the mental mold of how things fit
• 1:45:11
  SPEAKER_00: and slots together.
• 1:45:13
  SPEAKER_00: That feels a very sane way for us to balance our time and our energies appropriately.
• 1:45:19
  SPEAKER_01: So do I understand correctly that you're saying that it's probably
• 1:45:21
  SPEAKER_01: a sensible approach is have the architecture in your head,
• 1:45:25
  SPEAKER_01: know how your system works,
• 1:45:26
  SPEAKER_01: know how the boxes work,
• 1:45:27
  SPEAKER_01: know where there's AI inside it,
• 1:45:29
  SPEAKER_01: maybe it's generating code, maybe it's running your incidence response in ISO.
• 1:45:34
  SPEAKER_01: And then spend your time talking with other people in the business,
• 1:45:39
  SPEAKER_01: on your team,
• 1:45:40
  SPEAKER_01: with your customers,
• 1:45:41
  SPEAKER_01: because as you said,
• 1:45:42
  SPEAKER_01: systems like software is people's ideas connected and you have things in your head
• 1:45:48
  SPEAKER_01: or you write it down.
• 1:45:49
  SPEAKER_01: But now you're using AI as a tool to enable you to maybe take on larger chunks of
• 1:45:54
  SPEAKER_01: the business and maybe make better decisions because maybe you have more time to,
• 1:45:57
  SPEAKER_01: as an engineer,
• 1:45:58
  SPEAKER_01: understand more on the business,
• 1:46:00
  SPEAKER_01: on what is working, what is important, those kind of things.
• 1:46:02
  SPEAKER_00: Yeah, and also to make the structure on the model a bit more explicit as well.
• 1:46:08
  SPEAKER_00: I mean, that's one thing a good modular structure does.
• 1:46:10
  SPEAKER_00: I mean, this is why Shopify have been so successful keeping their thing as a monolith, right?
• 1:46:14
  SPEAKER_00: It's because they spend a lot of time building their monolithic application as
• 1:46:17
  SPEAKER_00: a modular domain-driven architecture, right?
• 1:46:19
  SPEAKER_00: Because it makes the structure easier to understand.
• 1:46:22
  SPEAKER_00: You can zoom out and see how those things fit together.
• 1:46:25
  SPEAKER_00: But also in the short term,
• 1:46:27
  SPEAKER_00: while we're having these debates about are we going full software factory or not,
• 1:46:33
  SPEAKER_00: we don't have to make a one-size-fits-all approach.
• 1:46:35
  SPEAKER_00: We can say, look,
• 1:46:36
  SPEAKER_00: this is our architecture,
• 1:46:37
  SPEAKER_00: being a modular monolith or a microservice architecture doesn't really matter.
• 1:46:40
  SPEAKER_00: And over here,
• 1:46:41
  SPEAKER_00: this area here is where we're trying a full software factory approach
• 1:46:44
  SPEAKER_00: where we don't look at the code anymore, but we specify the interface.
• 1:46:48
  SPEAKER_00: We have verification around that.
• 1:46:50
  SPEAKER_00: We know what good looks like for that module, right?
• 1:46:52
  SPEAKER_00: We've got functional tests,
• 1:46:54
  SPEAKER_00: tests that look at, say, some of the operational requirements as well.
• 1:46:58
  SPEAKER_00: These areas over here, we're still grappling.
• 1:46:59
  SPEAKER_00: These are really key and really important to us.
• 1:47:01
  SPEAKER_00: We're not ready to give up on those yet because we don't have the trust level yet.
• 1:47:05
  SPEAKER_00: We don't want to take that risk just yet.
• 1:47:07
  SPEAKER_00: And so over here,
• 1:47:08
  SPEAKER_00: we're going to work with the AI more of a collaborative way while we work together.
• 1:47:13
  SPEAKER_00: And we're still going to be intimately involved with the code
• 1:47:14
  SPEAKER_00: because we're not ready to make that step here.
• 1:47:16
  SPEAKER_00: But if we approach our whole system with one big black box,
• 1:47:18
  SPEAKER_00: then we're forced into making one or either decision.
• 1:47:21
  SPEAKER_00: I don't think that's healthy.
• 1:47:23
  SPEAKER_01: You're saying that you think that the software factory,
• 1:47:26
  SPEAKER_01: the dark factory,
• 1:47:27
  SPEAKER_01: you're giving it a spec and the AI either building it or operating it,
• 1:47:32
  SPEAKER_01: that you see that as viable.
• 1:47:33
  SPEAKER_00: There are some companies I know that are already all in on this.
• 1:47:36
  SPEAKER_00: Not everyone should be doing it.
• 1:47:37
  SPEAKER_00: However,
• 1:47:38
  SPEAKER_00: I think if that is our future,
• 1:47:40
  SPEAKER_00: if that is our future,
• 1:47:41
  SPEAKER_00: and I think there's a lot of ifs there,
• 1:47:43
  SPEAKER_00: the question is about how you go about adopting and trying.
• 1:47:46
  SPEAKER_00: So what I'm trying to do is find ways in which we can lower the barrier to entry
• 1:47:49
  SPEAKER_00: to try these ideas out in a safe way.
• 1:47:51
  SPEAKER_01: And like low-risk environments and maybe simpler domains, those kind of things, right?
• 1:47:55
  SPEAKER_00: But also like a lot of dark factories don't create finished products.
• 1:47:59
  SPEAKER_00: Like real world,
• 1:48:00
  SPEAKER_00: beyond the fact that real world dark factories,
• 1:48:02
  SPEAKER_00: there aren't many of them, most of them create components.
• 1:48:05
  SPEAKER_00: They don't create the finished product.
• 1:48:07
  SPEAKER_00: So in the same way,
• 1:48:09
  SPEAKER_00: if you think about a module being a piece of the overarching system,
• 1:48:14
  SPEAKER_00: well, let's have a factory around these bits here.
• 1:48:17
  SPEAKER_00: And actually looking at it,
• 1:48:19
  SPEAKER_00: if we had to rewrite that from scratch,
• 1:48:20
  SPEAKER_00: it might take us,
• 1:48:22
  SPEAKER_00: like with the collaborative way with AI,
• 1:48:24
  SPEAKER_00: it might take us a few weeks to have to rewrite that if we had to.
• 1:48:27
  SPEAKER_00: So actually, we're quite relaxed about leaving that to go as is.
• 1:48:29
  SPEAKER_00: And it's been working fine.
• 1:48:31
  SPEAKER_00: It's a module around which you've got great test coverage already,
• 1:48:33
  SPEAKER_00: for example, where it's not on the critical path from a latency point of view.
• 1:48:37
  SPEAKER_00: Or it's a service where we've got really good observability data and really well-defined SLOs.
• 1:48:42
  SPEAKER_00: And we've got good steady state.
• 1:48:43
  SPEAKER_00: Well, we can start running it as a factory now and seeing what's happening in production.
• 1:48:47
  SPEAKER_00: We can have a look at the SLOs.
• 1:48:49
  SPEAKER_00: We maybe add some special SLOs around it.
• 1:48:51
  SPEAKER_01: I'm hearing correctly that you're saying that as a professional,
• 1:48:54
  SPEAKER_01: as someone who Wants to be up to date with the new techniques that we can use.
• 1:49:00
  SPEAKER_01: It's probably if you have the opportunity to start to build a software factory,
• 1:49:05
  SPEAKER_01: a somewhat autonomous place in a component in something low risk,
• 1:49:08
  SPEAKER_01: it's probably worth trying it out to see how it works,
• 1:49:10
  SPEAKER_01: to see what you learn, to see if it even works or where it breaks down.
• 1:49:14
  SPEAKER_01: But do it in a way where you're minimizing the last rate.
• 1:49:16
  SPEAKER_01: It's a little bit like when,
• 1:49:18
  SPEAKER_01: you know,
• 1:49:18
  SPEAKER_01: when microservices came out,
• 1:49:19
  SPEAKER_01: the concept, you didn't start by moving your biggest thing.
• 1:49:22
  SPEAKER_01: You started to build a small service that didn't really matter all that much.
• 1:49:25
  SPEAKER_01: And you were okay to learn from it when it goes down and it wasn't on the half path
• 1:49:30
  SPEAKER_01: for the business.
• 1:49:31
  SPEAKER_00: Yeah, I think it's the thing to start off with is getting your head around specs.
• 1:49:34
  SPEAKER_00: So looking into spectrum development, whether that's Kiro or TESOL or SpecKit, whatever that is.
• 1:49:39
  SPEAKER_00: I mean, they're all slightly different.
• 1:49:40
  SPEAKER_00: Have a look at Brigitte's article on that as a good on-ramp to get ahead of it.
• 1:49:43
  SPEAKER_00: Start off by playing around with that because that's the stepping stone.
• 1:49:47
  SPEAKER_00: That's like you have to define what good looks like as an input to your factory.
• 1:49:51
  SPEAKER_00: Start with a spec.
• 1:49:52
  SPEAKER_00: Good input and also
• 1:49:53
  SPEAKER_01: for the output.
• 1:49:53
  SPEAKER_00: Yes.
• 1:49:54
  SPEAKER_00: And then it's like how you verify.
• 1:49:56
  SPEAKER_00: How are you going to go about verifying the output from the factory?
• 1:49:58
  SPEAKER_00: Have a think about that.
• 1:49:59
  SPEAKER_00: And, you know, like not just in terms of the kind of normal sort of behavioral stuff.
• 1:50:07
  SPEAKER_00: I agree with Kevin Hedy on this.
• 1:50:08
  SPEAKER_00: He said he hates non-functional requirements because if something is non-functional,
• 1:50:12
  SPEAKER_00: it means it doesn't work.
• 1:50:13
  SPEAKER_00: So he breaks the world down into development requirements,
• 1:50:17
  SPEAKER_00: operational requirements, and business requirements.
• 1:50:19
  SPEAKER_00: So I think it's a nice way of doing that.
• 1:50:20
  SPEAKER_00: So along those three axes,
• 1:50:23
  SPEAKER_00: How do I verify that the software being created
• 1:50:25
  SPEAKER_00: by this module has matched my developer requirements, so for, say,
• 1:50:28
  SPEAKER_00: maintainability,
• 1:50:30
  SPEAKER_00: my operational requirements,
• 1:50:31
  SPEAKER_00: latency, uptime, and my business requirement, is it working, features working?
• 1:50:36
  SPEAKER_00: So as long as you've got a way of validating your software around that,
• 1:50:41
  SPEAKER_00: and if you don't know what the good looks like for those three things, guess what?
• 1:50:44
  SPEAKER_00: This isn't for you.
• 1:50:45
  SPEAKER_00: But you can still do specs, by the way, but you can't do a factory.
• 1:50:48
  SPEAKER_00: But you can do specs, and that's your edging.
• 1:50:52
  SPEAKER_00: And I think it might not work.
• 1:50:54
  SPEAKER_00: There may well be some blind alleys.
• 1:50:56
  SPEAKER_00: This could be model-driven development all over again,
• 1:51:01
  SPEAKER_00: which we do not have time to get into because I get very angry about it.
• 1:51:03
  SPEAKER_00: But it could be that.
• 1:51:05
  SPEAKER_00: But I have a hope it might actually work this time, unlike MDM.
• 1:51:11
  SPEAKER_01: And then as closing,
• 1:51:12
  SPEAKER_01: I think a lot of what we talked about is it's important to hold the architecture
• 1:51:16
  SPEAKER_01: in your head,
• 1:51:17
  SPEAKER_01: to think about the architecture,
• 1:51:18
  SPEAKER_01: to think how to modularize now for engineers who want to get better at this kind
• 1:51:22
  SPEAKER_01: of high level thinking,
• 1:51:23
  SPEAKER_01: strategic thinking,
• 1:51:24
  SPEAKER_01: architectural thinking outside of Reading your book,
• 1:51:27
  SPEAKER_01: which I think brings a lot of really good ideas on architecture as well
• 1:51:30
  SPEAKER_01: and not just resilient systems.
• 1:51:31
  SPEAKER_01: But outside of that, what would your recommendation be for a tech professional, for an engineer?
• 1:51:35
  SPEAKER_01: I'd like to get better at architecting systems to know what good looks like.
• 1:51:41
  SPEAKER_00: I think there's a few books I can point people at.
• 1:51:44
  SPEAKER_00: I think the Software Architecture Fundamentals by Neil Ford and Mark Richards is a good read.
• 1:51:50
  SPEAKER_00: Just to be clear, they're friends of mine, so I'm not unbiased in recommending it.
• 1:51:54
  SPEAKER_00: I would say around the modularity,
• 1:51:55
  SPEAKER_00: Vlad Conlon offers Christoph's book on balanced coupling is fantastic.
• 1:52:00
  SPEAKER_00: His book on DDD is also very good.
• 1:52:02
  SPEAKER_00: But that balanced coupling book is really great.
• 1:52:05
  SPEAKER_00: He's also developed some skills in Claw to take some of those ideas.
• 1:52:07
  SPEAKER_00: So definitely, definitely take a look at those.
• 1:52:09
  SPEAKER_00: I think if you're looking at things architecture and you're thinking in terms
• 1:52:11
  SPEAKER_00: of modules, for me, I don't agree with everything in Mark and Neil's book.
• 1:52:14
  SPEAKER_00: There's lots of great stuff in there.
• 1:52:15
  SPEAKER_00: But for me,
• 1:52:16
  SPEAKER_00: it actually all starts,
• 1:52:17
  SPEAKER_00: I sort of mentioned earlier that writing the second mission
• 1:52:19
  SPEAKER_00: of building microservices was really all about smuggling in Getting people thinking
• 1:52:23
  SPEAKER_00: about this thing called information hiding.
• 1:52:24
  SPEAKER_00: It's a very hard idea for the 1970s.
• 1:52:27
  SPEAKER_00: It still comes back there to me.
• 1:52:28
  SPEAKER_00: Modules are still the whole thing for me.
• 1:52:29
  SPEAKER_00: So I would read Vlad's book.
• 1:52:31
  SPEAKER_00: And if you want to go read the original Parnas papers from 1971,
• 1:52:34
  SPEAKER_00: 1972, they're still worth reading.
• 1:52:37
  SPEAKER_00: But those would be good jumping off points, I would say.
• 1:52:39
  SPEAKER_00: You can still,
• 1:52:40
  SPEAKER_00: I think the nice thing about thinking in terms of modules is even within
• 1:52:42
  SPEAKER_00: an existing code base, you can start to modularize it.
• 1:52:47
  SPEAKER_00: In some small ways,
• 1:52:47
  SPEAKER_00: you can start to think about boundaries and things about clear separation,
• 1:52:51
  SPEAKER_00: about using encapsulation as a way of information.
• 1:52:54
  SPEAKER_01: And I guess it's fair to say, like, look,
• 1:52:56
  SPEAKER_01: if you have an idea on how you can make your existing code is a bit more modular,
• 1:53:00
  SPEAKER_01: you have zero excuse not to do it.
• 1:53:02
  SPEAKER_01: I mean, we don't trust agents fully,
• 1:53:03
  SPEAKER_01: but this is a textbook example of like,
• 1:53:05
  SPEAKER_01: if you can say,
• 1:53:06
  SPEAKER_01: here's my goal,
• 1:53:07
  SPEAKER_01: refactor this,
• 1:53:08
  SPEAKER_01: like if it would have taken by hand X amount of time, it will be faster.
• 1:53:12
  SPEAKER_01: So you should be empowered to at least experiment with these, right?
• 1:53:17
  SPEAKER_00: Especially if you've got a decent test suite.
• 1:53:19
  SPEAKER_00: And if not, start with that.
• 1:53:20
  SPEAKER_00: Start with that.
• 1:53:21
  SPEAKER_00: If you've got a strategy type language,
• 1:53:23
  SPEAKER_00: you don't even need to, just a good old fashioned IDE does an amazing job, right?
• 1:53:27
  SPEAKER_00: Dynamically tight language now don't have the excuse,
• 1:53:29
  SPEAKER_00: especially if it's a modern one that's well understood by the NLM.
• 1:53:31
  SPEAKER_00: So if you're using,
• 1:53:32
  SPEAKER_00: you know,
• 1:53:32
  SPEAKER_00: Python or Ruby,
• 1:53:33
  SPEAKER_00: it's so much better now than it used to be for refraction as code bases.
• 1:53:36
  SPEAKER_00: Absolutely.
• 1:53:37
  SPEAKER_00: You need to get a test suite in around that.
• 1:53:38
  SPEAKER_00: But yeah, you can start modularizing.
• 1:53:40
  SPEAKER_00: And that's about building.
• 1:53:42
  SPEAKER_00: And that's also a little bit coming back again to taking,
• 1:53:45
  SPEAKER_00: we don't want the architecture only in your head, right?
• 1:53:48
  SPEAKER_00: We want the code...
• 1:53:51
  SPEAKER_00: To sort of represent that mental model.
• 1:53:54
  SPEAKER_00: Because we communicate through the code.
• 1:53:56
  SPEAKER_00: Maybe in the future we'll be communicating through the specs.
• 1:53:59
  SPEAKER_00: And when you make those things more explicit,
• 1:54:02
  SPEAKER_00: then I think that helps everyone stay on the same page.
• 1:54:04
  SPEAKER_00: There's George Fairbanks wrote a book.
• 1:54:06
  SPEAKER_00: I've forgotten the name of the book now.
• 1:54:08
  SPEAKER_00: But in it he has a pattern called an architecturally evident coding style.
• 1:54:12
  SPEAKER_00: And that we should code in such a way that it clearly communicates the architecture
• 1:54:17
  SPEAKER_00: of the system.
• 1:54:18
  SPEAKER_00: And I think our programming languages are quite bad at having a good concept
• 1:54:22
  SPEAKER_00: of modules in general.
• 1:54:23
  SPEAKER_00: So I think we have to go a bit above and beyond ourselves.
• 1:54:27
  SPEAKER_00: But we could do more to make the architecture of our systems that little bit more explicit.
• 1:54:31
  SPEAKER_00: This is also why I spend time production,
• 1:54:33
  SPEAKER_00: because especially if your modular structure ends up being a process structure,
• 1:54:37
  SPEAKER_00: in other words, it's process boundaries.
• 1:54:39
  SPEAKER_00: I can go look at the networks to find out my architecture.
• 1:54:41
  SPEAKER_00: I shouldn't have to, but at least, you know, I know production is truth.
• 1:54:45
  SPEAKER_00: It's like football is life, production is truth.
• 1:54:48
  SPEAKER_00: So where can people get your book?
• 1:54:51
  SPEAKER_00: You can go and order it now.
• 1:54:52
  SPEAKER_00: If you can, go and buy it from a local bookshop.
• 1:54:54
  SPEAKER_00: That's always a great idea.
• 1:54:56
  SPEAKER_00: It's also going to be available for people on the O'Reilly platform in sort of digital form.
• 1:55:00
  SPEAKER_00: So you can go and read it as part.
• 1:55:01
  SPEAKER_00: If you're a subscriber at O'Reilly, you can get it there as well.
• 1:55:03
  SPEAKER_00: But lots of people love having the physical copy.
• 1:55:06
  SPEAKER_01: And we have these in the show notes below, so check them out to get the book directly.
• 1:55:10
  SPEAKER_01: And also, you've been doing some videos, right?
• 1:55:12
  SPEAKER_00: Yeah, well, I've been doing some stuff on Dave Farley's Modern Software Engineering channel.
• 1:55:16
  SPEAKER_00: It's kind of a good excuse to do some sort of semi-ranty,
• 1:55:21
  SPEAKER_00: you know, monologues to video and have some fun chats.
• 1:55:23
  SPEAKER_00: So yeah,
• 1:55:24
  SPEAKER_00: go to the Modern Software Engineering channel on YouTube and you can find videos
• 1:55:27
  SPEAKER_00: from me and Dave Farley and Kevin Henney's there.
• 1:55:30
  SPEAKER_00: We've got stuff from Kent, Trisha G., Emily Borch, Dan North is there as well.
• 1:55:36
  SPEAKER_00: I'm pretty sure I've forgotten somebody else.
• 1:55:37
  SPEAKER_00: There's almost a couple of people there as well,
• 1:55:39
  SPEAKER_00: but we try and put videos out once a week and it's a bit of a fun place.
• 1:55:42
  SPEAKER_01: And we're also doing that one.
• 1:55:44
  SPEAKER_01: Yeah.
• 1:55:44
  SPEAKER_01: And if people would want to work with you, where can they find you?
• 1:55:47
  SPEAKER_01: How can they find you?
• 1:55:49
  SPEAKER_01: What kind of stuff can they get help from you for?
• 1:55:52
  SPEAKER_00: I also do a lot of consulting for clients,
• 1:55:53
  SPEAKER_00: helping them around re-architecting their systems and stuff like that.
• 1:55:56
  SPEAKER_00: If you know more about that, actually go to my
• 1:56:07
  SPEAKER_00: www.samnewman.io and you can contact me there.
• 1:56:10
  SPEAKER_00: You can also find me on LinkedIn.
• 1:56:12
  SPEAKER_00: But if you come straight to the website, use email.
• 1:56:15
  SPEAKER_00: I'm still a bit old-fashioned like that and I work with clients all over the world.
• 1:56:18
  SPEAKER_00: So if I'm able to help, yeah, head over to samnewman.io.
• 1:56:21
  SPEAKER_00: My contact details are there.
• 1:56:22
  SPEAKER_01: Wonderful.
• 1:56:23
  SPEAKER_01: Well, Sam, thanks again.
• 1:56:24
  SPEAKER_01: This was great.
• 1:56:25
  SPEAKER_00: Thank you.
• 1:56:26
  SPEAKER_01: Thanks very much to Sam for this conversation.
• 1:56:28
  SPEAKER_01: I really like Sam's three rules of distributed systems.
• 1:56:31
  SPEAKER_01: One, it takes time.
• 1:56:33
  SPEAKER_01: Two, sometimes the thing you want to talk to isn't there.
• 1:56:36
  SPEAKER_01: Three, resource pools are not infinite.
• 1:56:39
  SPEAKER_01: Sam said how most of the outages happen because of this last one.
• 1:56:42
  SPEAKER_01: Resource pools not being infinite and services are getting saturated.
• 1:56:46
  SPEAKER_01: But I was chuckling when he was talking about rabbits stirring
• 1:56:49
  SPEAKER_01: through networking cables between office buildings.
• 1:56:51
  SPEAKER_01: Then again,
• 1:56:52
  SPEAKER_01: a few weeks ago,
• 1:56:53
  SPEAKER_01: with thousands of flights stranded on the East Coast of the US,
• 1:56:56
  SPEAKER_01: it was because workers accidentally cut a telecom cable in New Jersey used by
• 1:56:59
  SPEAKER_01: the FAA.
• 1:57:00
  SPEAKER_01: Another food for thought What I love
• 1:57:01
  SPEAKER_01: from Sam is how failure should be handled differently depending on the type
• 1:57:04
  SPEAKER_01: of business.
• 1:57:05
  SPEAKER_01: An e-commerce shop might decide to show you the item when the inventory service is down.
• 1:57:09
  SPEAKER_01: Worst case,
• 1:57:10
  SPEAKER_01: after you order and pay,
• 1:57:12
  SPEAKER_01: they just refund you because we're like, oh, we don't have the item.
• 1:57:15
  SPEAKER_01: Basically, failing open is fine in this case.
• 1:57:17
  SPEAKER_01: But on a concert ticketing website,
• 1:57:19
  SPEAKER_01: you don't want to sell a ticket that might or might not be there.
• 1:57:23
  SPEAKER_01: And so failing clothes is the right call to make.
• 1:57:25
  SPEAKER_01: Because again,
• 1:57:25
  SPEAKER_01: as we discussed in the episode,
• 1:57:27
  SPEAKER_01: if you sell the ticket,
• 1:57:28
  SPEAKER_01: people might book flights, they might get upset, you might lose them as customers.
• 1:57:31
  SPEAKER_01: Finally, I appreciated how level-headed, perhaps even cautious Sam is about AI.
• 1:57:36
  SPEAKER_01: He's got no doubts that AI is very useful at the micro level,
• 1:57:39
  SPEAKER_01: basically at tactical, at things that we can already improve.
• 1:57:42
  SPEAKER_01: But he is worried about the economics of the model providers.
• 1:57:45
  SPEAKER_01: His suggestion is pretty pragmatic.
• 1:57:47
  SPEAKER_01: Hedge your vendors and your models.
• 1:57:49
  SPEAKER_01: Be multi-vendor and multi-tool,
• 1:57:51
  SPEAKER_01: and wherever you can,
• 1:57:52
  SPEAKER_01: swap non-deterministic AI agent workflows to deterministic ones.
• 1:57:56
  SPEAKER_01: Basically, have LLMs generate deterministic code, which I find very, very...
• 1:58:01
  SPEAKER_01: Refreshing thinking.
• 1:58:02
  SPEAKER_01: Check the show notes below for the story about scaling Uber in the early days
• 1:58:05
  SPEAKER_01: with Uber's first CTO, Twan Palm, that we discussed and it's an awesome story.
• 1:58:10
  SPEAKER_01: And for more, The Pragmatic Engineer deep ties on resilient systems.
• 1:58:13
  SPEAKER_01: If you enjoyed this show, do subscribe to your favorite podcast player.
• 1:58:16
  SPEAKER_01: And thank you very much if you recommend the podcast to your friends or your colleagues.
• 1:58:20
  SPEAKER_01: Thanks and I'll see you in the next one.