Latest / The Pragmatic Engineer / Context engineering with Dex Horthy

Transcript

Context engineering with Dex Horthy · The Pragmatic Engineer

• 0:00
  SPEAKER_00: What happens when you let AI agents ship code for months, and no developer reads a single line?
• 0:05
  SPEAKER_00: Today's guest tried exactly that.
• 0:06
  SPEAKER_00: He built a light-off software factory,
• 0:08
  SPEAKER_00: and four months later,
• 0:09
  SPEAKER_00: he had no choice but to shut it down as things just stopped working.
• 0:13
  SPEAKER_00: Dex Horthy is the founder of HumanLayer and a person who coined
• 0:16
  SPEAKER_00: the term context engineering days
• 0:17
  SPEAKER_00: before Andrzej Karpathy & Tooby Luzka made it famous.
• 0:20
  SPEAKER_00: He spent the last two years talking to hundreds of AI engineers
• 0:22
  SPEAKER_00: about what actually works when you build with LLMs,
• 0:25
  SPEAKER_00: and is testing the most extreme ideas with his own team.
• 0:27
  SPEAKER_00: In today's conversation, We'll discuss.
• 0:29
  SPEAKER_00: Context Engineering.
• 0:30
  SPEAKER_00: What it is and the physics of context windows, including what the dump zone is.
• 0:34
  SPEAKER_00: Loop Engineering.
• 0:35
  SPEAKER_00: From the Ralph Wiggum technique to the slow loop that Dex's team runs every night
• 0:39
  SPEAKER_00: to wake up to code clean up PRs.
• 0:41
  SPEAKER_00: The rise of software factories.
• 0:42
  SPEAKER_00: From a NATO conference in 19...
• 0:44
  SPEAKER_00: 1968 through DevOps to today's agentic factories,
• 0:47
  SPEAKER_00: Spectrum and Development development and why specs always drift from
• 0:50
  SPEAKER_00: the code itself, and many more.
• 0:52
  SPEAKER_00: If you want to understand increasingly important concepts like concept engineering
• 0:55
  SPEAKER_00: and harness engineering,
• 0:57
  SPEAKER_00: or want to know how far you can push the let agents build everything idea
• 1:00
  SPEAKER_00: from someone who pushed it further than almost anyone, then this episode is for you.
• 1:04
  SPEAKER_00: This episode is presented by Antisysys.
• 1:06
  SPEAKER_00: If you work with agents,
• 1:07
  SPEAKER_00: your job is no longer just writing code, it's specifying and testing it.
• 1:11
  SPEAKER_00: And antithesis is the most effective method of verifying agendic code today.
• 1:16
  SPEAKER_00: Today's episode is brought to you by BuildKite,
• 1:17
  SPEAKER_00: the CI orchestration platform trusted by OpenAI,
• 1:20
  SPEAKER_00: Entropic, Cursor, NVIDIA, Uber, Canva, and more.
• 1:23
  SPEAKER_00: Today we're talking about pushing the right context into models so that they write better code.
• 1:27
  SPEAKER_00: Right after that starts working, your agents will write more code.
• 1:30
  SPEAKER_00: A lot more.
• 1:31
  SPEAKER_00: Trusting that code avalanche is where many teams face a challenge today.
• 1:34
  SPEAKER_00: Every change that an agent makes still has to be built, tested, and proven safe before it ships.
• 1:39
  SPEAKER_00: Worked on my machine is not enough, so you obviously need CI.
• 1:43
  SPEAKER_00: But when agents are pushing 5,
• 1:45
  SPEAKER_00: 10 or 50 times the commit volume to your pipelines faster CI runners won't save you.
• 1:50
  SPEAKER_00: Shaving 30 seconds off a single build is meaningless when a queue is 100 plus jobs deep.
• 1:54
  SPEAKER_00: What you really want is a CI system that gets faster as the volume grows and CI
• 1:59
  SPEAKER_00: that offers instant parallelization to give you unlimited concurrency and
• 2:02
  SPEAKER_00: to intelligently route changes at runtime.
• 2:05
  SPEAKER_00: This is what BuildKite does and why global software leaders continue to rely on it.
• 2:09
  SPEAKER_00: The same architecture that observed the scale of Shopify and Uber
• 2:12
  SPEAKER_00: a decade ago 1 .4 billion job minutes minutes a week across Cursor,
• 2:16
  SPEAKER_00: Meta, Reddit and Snowflake.
• 2:17
  SPEAKER_00: While the rest of CI world are cracking under the weight
• 2:33
  SPEAKER_00: As you're entering the context you'll give to your agents,
• 2:35
  SPEAKER_00: think about how you'll verify what they hand back.
• 2:37
  SPEAKER_00: If your system is buckling under the increased volume head
• 2:40
  SPEAKER_00: to buildkite .com slash pragmatic 30 -day all access trial no credit card and
• 2:44
  SPEAKER_00: an actual human engineer on standby his name's Ola and he's very helpful So Dex,
• 2:49
  SPEAKER_00: welcome to the podcast.
• 2:51
  SPEAKER_00: Super stoked to be here, dude.
• 2:52
  SPEAKER_01: Before we get into some of the context engineering and some of the more spicy stuff
• 2:57
  SPEAKER_01: as well, how did you get into tech?
• 2:59
  SPEAKER_01: How did you fall in love with computers?
• 3:01
  SPEAKER_02: Oh man, so I was doing undergrad as a physics major and I realized that I didn't like it.
• 3:09
  SPEAKER_02: Academia And there's like bass.
• 3:10
  SPEAKER_02: Basically like Two or three paths out of physics is basically you go get a PhD
• 3:17
  SPEAKER_02: or you go into finance or you go do programming.
• 3:21
  SPEAKER_02: At that time,
• 3:21
  SPEAKER_02: this was 2012, 2011, when I was in the middle of undergrad and deciding what to do.
• 3:25
  SPEAKER_02: And I had done...
• 3:26
  SPEAKER_02: An internship.
• 3:27
  SPEAKER_02: Internship when I was in high school, I was working with NASA researchers To, uh...
• 3:31
  SPEAKER_02: jet propulsion lab in California.
• 3:33
  SPEAKER_02: They had just gotten this really high fidelity,
• 3:36
  SPEAKER_02: like the most,
• 3:37
  SPEAKER_02: you know,
• 3:38
  SPEAKER_02: fine grained data set of altitudes,
• 3:40
  SPEAKER_02: like the heights of very,
• 3:42
  SPEAKER_02: very like top top topographical map of the South Pole of the moon.
• 3:46
  SPEAKER_02: And the south pole of moon is really interesting because some of
• 3:48
  SPEAKER_02: the craters there are so deep. Because of the angle it has, it got hit
• 3:51
  SPEAKER_02: by meteor storms like no other part of the moon.
• 3:54
  SPEAKER_02: So there's very deep craters that have never seen sunlight.
• 3:57
  SPEAKER_02: And so there's frozen liquid water in there from the formation of the moon.
• 4:02
  SPEAKER_02: And so scientists were really interested in getting down there and exploring.
• 4:04
  SPEAKER_02: And so we had this really fine grained map and it was like, okay, cool.
• 4:07
  SPEAKER_02: Let's build software so that I have point A to point B.
• 4:11
  SPEAKER_02: I know the limitations of my Rover can,
• 4:13
  SPEAKER_02: you know, max incline up is this, max incline down is that.
• 4:16
  SPEAKER_02: Find a path from point A to point B that doesn't like break those rules of the incline.
• 4:20
  SPEAKER_02: So I was, you know, 17.
• 4:22
  SPEAKER_02: I had never cracked a CS textbook.
• 4:24
  SPEAKER_02: So I wrote,
• 4:24
  SPEAKER_02: I basically like wrote a really naive,
• 4:26
  SPEAKER_02: bad version of Dijkstra's algorithm for pathfinding.
• 4:30
  SPEAKER_02: So I was in college, and I was like...
• 4:31
  SPEAKER_02: like, I don't know if I want to do the academics thing,
• 4:32
  SPEAKER_02: but I really enjoyed programming back in the day.
• 4:34
  SPEAKER_02: And so I decided to go.
• 4:36
  SPEAKER_02: I got like half of a CS minor and then And started working on it.
• 4:40
  SPEAKER_02: On an API platform team at a software company in Chicago.
• 4:43
  SPEAKER_02: Sprout Social, right?
• 4:45
  SPEAKER_02: Yes.
• 4:45
  SPEAKER_02: And, uh...
• 4:47
  SPEAKER_01: basically never went back.
• 4:48
  SPEAKER_01: Yeah.
• 4:49
  SPEAKER_01: And then And where did you...
• 4:50
  SPEAKER_01: you go from there?
• 4:52
  SPEAKER_01: Where did you pick up like the parts of the trade?
• 4:53
  SPEAKER_01: Because very early on, your first job, that's not really common.
• 4:56
  SPEAKER_01: You were doing platform engineering back in, you know, more than a decade ago.
• 5:00
  SPEAKER_02: From that point,
• 5:01
  SPEAKER_02: it took me about two or three months to notice that like,
• 5:04
  SPEAKER_02: The most valuable work that was being done in the company was being done by like,
• 5:08
  SPEAKER_02: of course,
• 5:08
  SPEAKER_02: it's obviously like the first couple engineers who know everything and understand
• 5:12
  SPEAKER_02: where everything was.
• 5:13
  SPEAKER_02: And like,
• 5:13
  SPEAKER_02: you spend a day on a support ticket from a customer and they solve it
• 5:16
  SPEAKER_02: in five minutes, but like, you have to solve it so you learn and whatever.
• 5:19
  SPEAKER_02: And I realized like the most Valuable people in the company were the people
• 5:23
  SPEAKER_02: that were building the developer platform.
• 5:26
  SPEAKER_02: CICD, sandbox environments, preview stuff.
• 5:29
  SPEAKER_02: And so I kind of like, that was my first step into the journey.
• 5:32
  SPEAKER_02: And I've basically been obsessed with software factories since that like three
• 5:37
  SPEAKER_02: or six months into my first job.
• 5:38
  SPEAKER_01: We talk about software factories now, but you're talking about software factories back then.
• 5:42
  SPEAKER_01: So like you were You are starting to already think think
• 5:44
  SPEAKER_01: that this is how we can produce better software.
• 5:47
  SPEAKER_01: This is pre-AI world, right?
• 5:49
  SPEAKER_02: Well, and I'm always surprised.
• 5:50
  SPEAKER_02: Like, there's a huge class of developers that say, I don't want to work on CICD.
• 5:53
  SPEAKER_02: I hate CICD.
• 5:54
  SPEAKER_02: I'm like, really?
• 5:55
  SPEAKER_02: because building the thing that builds the thing and building the thing
• 5:58
  SPEAKER_02: that builds a thing, the bills. The thing is
• 5:59
  SPEAKER_02: It's like, as.
• 6:00
  SPEAKER_02: As software engineers, we're lazy.
• 6:01
  SPEAKER_02: We want to do the most high leverage thing that makes our job easier.
• 6:04
  SPEAKER_02: So how do we,
• 6:05
  SPEAKER_02: if we can build a thing that helps us build a thing that helps us move faster,
• 6:08
  SPEAKER_02: then that's the best use of my time as a lazy engineer.
• 6:11
  SPEAKER_01: And then you went to another startup, Aspernation.
• 6:15
  SPEAKER_01: Aspiration.
• 6:16
  SPEAKER_01: Aspiration, yeah.
• 6:17
  SPEAKER_01: Aspiration, also platform engineering.
• 6:19
  SPEAKER_02: Yeah, I was brought in and then like three months into the job,
• 6:22
  SPEAKER_02: the VP of engineering who hired me quit or got fired.
• 6:25
  SPEAKER_02: I don't know, there was some drama about it.
• 6:26
  SPEAKER_02: I probably shouldn't talk about it.
• 6:27
  SPEAKER_02: And then I was there for about a year and was kind of like acting CTO for a while,
• 6:32
  SPEAKER_02: like hired a couple of people,
• 6:33
  SPEAKER_02: helped hire the new VP of engineering, but I was out of there.
• 6:36
  SPEAKER_01: I don't think I'll ever do consumer again.
• 6:37
  SPEAKER_01: I think I'm actually a B2B guy.
• 6:39
  SPEAKER_01: Good to know.
• 6:40
  SPEAKER_01: And then you went to Replicator,
• 6:41
  SPEAKER_01: where you spent a good solid four years and went from engineer,
• 6:45
  SPEAKER_01: forward-deployed engineer, to product manager.
• 6:47
  SPEAKER_02: Yeah, I did core engineering for two years.
• 6:49
  SPEAKER_02: We were building a container orchestrator Like, before Kubernetes.
• 6:52
  SPEAKER_02: Kubernetes,
• 6:52
  SPEAKER_02: before Docker Swarm was really We built our own orchestrator.
• 6:56
  SPEAKER_02: The founders had this vision that like,
• 6:57
  SPEAKER_02: oh, Docker is going to make it much easier to ship on -prem software.
• 7:01
  SPEAKER_02: And when I say on-prem, I don't mean literally like a rack in a colo.
• 7:05
  SPEAKER_02: It's more like, hey, look,
• 7:06
  SPEAKER_02: bring the app to where the data is rather than sending the data up
• 7:09
  SPEAKER_02: to some cloud vendor.
• 7:10
  SPEAKER_02: And Docker makes it much more, much easier for them to package apps and move them around.
• 7:14
  SPEAKER_02: And so they had this thesis that like, basically you could build a platform that.
• 7:18
  SPEAKER_02: The experience that you get when you use GitHub Enterprise,
• 7:20
  SPEAKER_02: which is like you install it and has this admin panel,
• 7:22
  SPEAKER_02: but then you just get GitHub running in your data center and your code never has
• 7:26
  SPEAKER_02: to leave your data center.
• 7:28
  SPEAKER_02: Suddenly you could build a generic SaaS where everybody could have that.
• 7:32
  SPEAKER_01: So I did two years in engineer there.
• 7:34
  SPEAKER_02: And then our head of sales, we parted ways with our head of sales.
• 7:37
  SPEAKER_02: And honestly, I was having a lot of arguments about the software factory with our CTO.
• 7:42
  SPEAKER_02: And it's kind of like almost like a too many cooks in the kitchen kind of thing.
• 7:45
  SPEAKER_02: I'm sure many listeners have had this experience.
• 7:47
  SPEAKER_02: Of like, well yeah, I know I have these
• 7:57
  SPEAKER_02: Dex, I need you to stop fixing the build pipeline and do the tickets I gave you.
• 8:02
  SPEAKER_02: I'm sure you've had this experience, perhaps.
• 8:04
  SPEAKER_02: Yeah.
• 8:04
  SPEAKER_02: And then And was this what?
• 8:06
  SPEAKER_02: What Love you too.
• 8:07
  SPEAKER_01: To either forward deploy engineering?
• 8:09
  SPEAKER_02: Yeah.
• 8:10
  SPEAKER_02: So I like I really loved our customers.
• 8:11
  SPEAKER_02: Our customers are HashiCorp,
• 8:13
  SPEAKER_02: DataStax, Puppet, all these really cool engineering brands, Travis CI, Circles.
• 8:17
  SPEAKER_02: I was like, yeah, I actually love working with our customers.
• 8:19
  SPEAKER_02: Our customers are awesome.
• 8:21
  SPEAKER_02: And And it was a great way to like.
• 8:22
  SPEAKER_02: To get in the trenches.
• 8:23
  SPEAKER_02: A lot of really good engineers who were solving the hardest problem at the company,
• 8:26
  SPEAKER_02: which is like,
• 8:27
  SPEAKER_02: how do we take this three to five year old SaaS platform and pack package it all up
• 8:32
  SPEAKER_02: so that someone who knows nothing about our architecture can run it reliably
• 8:35
  SPEAKER_02: in their own AWS VPC, in their own on-prem data center, whatever it was.
• 8:40
  SPEAKER_02: And so I spent, I was our first kind of customer facing Yeah.
• 8:43
  SPEAKER_02: Engineer.
• 8:44
  SPEAKER_02: And it was about three months.
• 8:47
  SPEAKER_02: I, we closed.
• 8:48
  SPEAKER_02: I met with every customer that was like kind of kind of in the pipeline
• 8:51
  SPEAKER_02: but wasn't moving sales-wise,
• 8:53
  SPEAKER_02: and we closed 12 deals in three months,
• 8:55
  SPEAKER_02: and the CEO was like, holy crap, Dex, the investors are taking my calls again.
• 9:00
  SPEAKER_02: I I don't, I know you want to get back to.
• 9:01
  SPEAKER_02: To coding,
• 9:02
  SPEAKER_02: but I need you to go hire three people and build this team out
• 9:05
  SPEAKER_02: because I think you might have been born for this.
• 9:07
  SPEAKER_02: Wow.
• 9:08
  SPEAKER_01: Yeah.
• 9:08
  SPEAKER_02: So I did that for about four years, built that org to like 25 people.
• 9:11
  SPEAKER_02: And then Zerp happened and it got a lot smaller.
• 9:14
  SPEAKER_02: And we kind of realized like, hey, we have a product that's like pretty good.
• 9:16
  SPEAKER_02: And we've been solved.
• 9:18
  SPEAKER_02: Solving Lots of early starters.
• 9:19
  SPEAKER_02: Startups do is like, OK, there's some usability issues.
• 9:22
  SPEAKER_02: We'll get a bunch of smart people, throw them in the trenches with our customers.
• 9:24
  SPEAKER_02: Great for sales, great for retention, all this stuff.
• 9:27
  SPEAKER_02: And it was like, oh, we actually like the margins on that aren't good enough.
• 9:31
  SPEAKER_02: And so we basically were like, cool,
• 9:32
  SPEAKER_02: we actually just need to make the product way more usable,
• 9:34
  SPEAKER_02: do a more PLG shaped thing.
• 9:36
  SPEAKER_01: PLG meaning product-led growth?
• 9:37
  SPEAKER_02: Product-led growth.
• 9:38
  SPEAKER_02: Make it a little more self-service so you don't need an expert to teach you how to use it.
• 9:41
  SPEAKER_02: And I was like, cool,
• 9:42
  SPEAKER_02: if that's the most important thing,
• 9:43
  SPEAKER_02: then I want to go be a product manager because I have tons of opinions.
• 9:46
  SPEAKER_02: I've now spent four years in the trenches with our customers.
• 9:48
  SPEAKER_02: I have a laundry list of roadmap things that I think would make
• 9:51
  SPEAKER_02: the product way easier to use and adopt and implement and deploy.
• 9:54
  SPEAKER_02: And now you went towards a dark side.
• 9:57
  SPEAKER_02: Exactly.
• 9:57
  SPEAKER_02: Yeah, I did.
• 9:58
  SPEAKER_02: I was like, this is going to kill my street cred, isn't it?
• 10:00
  SPEAKER_02: But I was really glad.
• 10:01
  SPEAKER_02: You know,
• 10:02
  SPEAKER_02: I think a lot of engineers are afraid that if they go do a customer facing thing,
• 10:05
  SPEAKER_02: they lose all their credibility.
• 10:07
  SPEAKER_02: And like, yes, I wasn't coding for 10 hours a day.
• 10:09
  SPEAKER_02: I was coding for like three or four hours on a Saturday for fun.
• 10:12
  SPEAKER_02: Not, but I mean we were.
• 10:14
  SPEAKER_02: Helping people build YAML.
• 10:15
  SPEAKER_02: We were building CLIs.
• 10:16
  SPEAKER_02: We owned a lot of the tooling that customers use,
• 10:18
  SPEAKER_02: but it was like the last mile delivery side of it, not the core platform.
• 10:21
  SPEAKER_02: And like,
• 10:22
  SPEAKER_02: I'm more - more personal note,
• 10:23
  SPEAKER_02: I had spent the last like most of my 20s feeling like,
• 10:26
  SPEAKER_02: OK, a little bit introverted, a little bit like socially awkward.
• 10:30
  SPEAKER_02: What a lot of engineers, I'm sure, experience.
• 10:33
  SPEAKER_02: And I had talked to Ooh, my uncle's a musical.
• 10:36
  SPEAKER_02: Music producer.
• 10:37
  SPEAKER_02: So he used to work with like Randy Newman and a bunch of like really famous musicians.
• 10:40
  SPEAKER_00: Oh, wow.
• 10:41
  SPEAKER_02: Yeah.
• 10:41
  SPEAKER_02: This guy Mitchell Froome.
• 10:42
  SPEAKER_02: And he I was sitting with dinner with him at some point.
• 10:45
  SPEAKER_02: And when I was I think it was when I was still in undergrad, but he gave me this lecture.
• 10:48
  SPEAKER_02: He's basically like, if you want to be really good at something.
• 10:50
  SPEAKER_02: You have to make it the only thing you do.
• 10:53
  SPEAKER_02: The guy playing guitar nights and weekends trying to get his band off
• 10:56
  SPEAKER_02: the ground will probably never achieve greatness.
• 10:59
  SPEAKER_02: The people who become great are the people who basically make it like,
• 11:03
  SPEAKER_02: if I don't play guitar, I don't eat.
• 11:05
  SPEAKER_02: And you go and you sit on the street all day and you play for 14 hours a day or whatever it is.
• 11:10
  SPEAKER_02: That's the only way to become great.
• 11:12
  SPEAKER_02: So I said, okay,
• 11:13
  SPEAKER_02: instead of trying to like read self-help books about how to be less introverted
• 11:16
  SPEAKER_02: and less socially awkward,
• 11:17
  SPEAKER_02: like what if I just made it my freaking job to just talk to people and make friends
• 11:21
  SPEAKER_02: and like help people and solve their problems?
• 11:23
  SPEAKER_02: And I think it worked out.
• 11:25
  SPEAKER_02: I recommend it.
• 11:26
  SPEAKER_02: I think everyone should spend a year or two
• 11:27
  SPEAKER_02: at least doing something really like customer facing.
• 11:30
  SPEAKER_01: Did you do this because...
• 11:32
  SPEAKER_01: You felt that it was holding you back being introverted or like what?
• 11:36
  SPEAKER_01: And I know you got the motivation from the whole musician motivation.
• 11:39
  SPEAKER_01: I get it on one part.
• 11:40
  SPEAKER_01: but what was it that you said like all right is the customer facing thing
• 11:43
  SPEAKER_01: that I'm gonna be doing because clearly
• 11:45
  SPEAKER_01: you were pretty great at like writing code.
• 11:47
  SPEAKER_01: You could argue you you were doing it night and day.
• 11:49
  SPEAKER_01: So where did you find that?
• 11:50
  SPEAKER_01: Like I actually,
• 11:51
  SPEAKER_01: I think like customer facing or like getting this introvert off of me.
• 11:55
  SPEAKER_01: Did you feel that about holding you back or you just wanted to be good
• 11:58
  SPEAKER_02: at it?
• 11:58
  SPEAKER_02: It was just kind of a thing that was like interfering
• 12:01
  SPEAKER_02: with my like general life satisfaction.
• 12:03
  SPEAKER_02: And it's also like, I'm not a very type A person.
• 12:05
  SPEAKER_02: I'm very disorganized.
• 12:06
  SPEAKER_02: I don't know if people call it like, okay, I'm like ADHD now.
• 12:09
  SPEAKER_02: That's why I can run 30 clods in parallel or whatever it is.
• 12:11
  SPEAKER_02: But it was like, I was really bad at email and calendars and spreadsheets.
• 12:14
  SPEAKER_02: It was like, didn't care about these, didn't understand them.
• 12:16
  SPEAKER_02: And so like another side effect of this was like,
• 12:18
  SPEAKER_02: it just forced me to be organized and keep a lot of things going.
• 12:21
  SPEAKER_02: And so like,
• 12:21
  SPEAKER_02: I don't know,
• 12:22
  SPEAKER_02: there's like weird benefits you get from like stepping outside your comfort zone
• 12:25
  SPEAKER_02: and learning like industrial disciplines that are separate
• 12:28
  SPEAKER_02: from what you've been doing.
• 12:29
  SPEAKER_01: And so the opportunity presented itself and I was like, oh, I like working.
• 12:32
  SPEAKER_01: I'll try this for a little bit.
• 12:33
  SPEAKER_01: Started going
• 12:33
  SPEAKER_02: really well.
• 12:34
  SPEAKER_02: I'm like, cool, let's see how far this thread goes.
• 12:37
  SPEAKER_01: And then And afterwards, afterwards, you're now in your second startup.
• 12:40
  SPEAKER_01: You became a founder and you also got involved in AI pretty early As I, as.
• 12:46
  SPEAKER_01: Was even before it was so obvious that it would change how we develop software, right?
• 12:51
  SPEAKER_02: Well, I would say I was,
• 12:53
  SPEAKER_02: I was later than I could have been because we started the company,
• 12:56
  SPEAKER_02: uh, Me and a buddy, I'm sure.
• 12:57
  SPEAKER_02: Chicago started a company in the data engineering space in about 2020,
• 13:01
  SPEAKER_02: November, 2020, we decided in like August of 2020.
• 13:03
  SPEAKER_02: This is Metalitics.
• 13:04
  SPEAKER_02: Metalitics.
• 13:05
  SPEAKER_02: Um, technically still the same company as human layer.
• 13:06
  SPEAKER_02: We just like pivoted the mission.
• 13:09
  SPEAKER_02: But yeah,
• 13:10
  SPEAKER_02: the advice I got from every angel investor that,
• 13:13
  SPEAKER_02: you know,
• 13:13
  SPEAKER_02: people who just knew CTOs I'd worked for before and stuff,
• 13:16
  SPEAKER_02: they were just like, look, hitting a lot of heads wins.
• 13:18
  SPEAKER_02: I don't know if you know,
• 13:18
  SPEAKER_02: like the whole DBT,
• 13:20
  SPEAKER_02: data engineering,
• 13:21
  SPEAKER_02: Fivetran,
• 13:21
  SPEAKER_02: that whole arc where it was like this huge party and tons of investor money going
• 13:25
  SPEAKER_02: into all these different companies.
• 13:27
  SPEAKER_02: And then within by like 2021, 2022,
• 13:30
  SPEAKER_02: There was kind of the Zerp thing and just this general realization that the TAM
• 13:33
  SPEAKER_02: for those sorts of tools is not as big as everyone thought it was.
• 13:36
  SPEAKER_02: Yes, the total addressable market for those sort of tools was not quite as big
• 13:41
  SPEAKER_02: as we all thought it was.
• 13:44
  SPEAKER_02: So it was a hard place to raise money.
• 13:45
  SPEAKER_02: It was a hard place to get customers.
• 13:47
  SPEAKER_01: Yeah, and then I met.
• 13:48
  SPEAKER_01: Met you while you were at Human Layer NSF at an event.
• 13:52
  SPEAKER_01: You actually talked and we chatted afterwards.
• 13:54
  SPEAKER_01: But by that, this was about a year ago, You were already.
• 13:57
  SPEAKER_01: Started You start to have some really.
• 13:59
  SPEAKER_01: Really strong opinions That's on.
• 14:00
  SPEAKER_01: On using AI.
• 14:02
  SPEAKER_01: And one of them was this now famous 12-factor agents manifesto.
• 14:07
  SPEAKER_02: Are we calling
• 14:07
  SPEAKER_01: it a Man of F***er?
• 14:08
  SPEAKER_01: Manifesto now?
• 14:09
  SPEAKER_01: I'm calling it a manifesto.
• 14:10
  SPEAKER_01: It's a manifesto.
• 14:11
  SPEAKER_01: I'm calling it.
• 14:12
  SPEAKER_01: Let's talk about this.
• 14:13
  SPEAKER_01: This was 12 engineering principles It's a builder reliable.
• 14:15
  SPEAKER_01: Reliable production-ready apps.
• 14:17
  SPEAKER_01: How did you come up with it?
• 14:18
  SPEAKER_01: With this?
• 14:18
  SPEAKER_01: And maybe we can also talk about some of them.
• 14:20
  SPEAKER_02: Yeah.
• 14:21
  SPEAKER_02: So I'll kind of like go to like around In August, the.
• 14:25
  SPEAKER_02: The co-founder I was working with, they kind of burned out and left.
• 14:27
  SPEAKER_02: And it was very, we're on good terms.
• 14:29
  SPEAKER_02: It was very mutual.
• 14:30
  SPEAKER_02: And I decided to just start messing with AI stuff.
• 14:32
  SPEAKER_02: And I was building AI agents.
• 14:34
  SPEAKER_02: And what was really in folk right then was like,
• 14:36
  SPEAKER_02: The Langchain, the Crew AI, these agent frameworks.
• 14:39
  SPEAKER_02: And it seemed like there was a ton of... You go in the Crew AI Discord, there's 10,000 people.
• 14:43
  SPEAKER_02: It's like, okay, this feels like the right shape.
• 14:45
  SPEAKER_02: And there's clearly this eco.
• 14:47
  SPEAKER_02: You go into every single one of those projects, they have a ChromaDB plugin.
• 14:50
  SPEAKER_02: They have a Composio plugin.
• 14:53
  SPEAKER_02: Clearly, this is the shared interface that everybody is building for.
• 14:57
  SPEAKER_02: I said, okay, what's missing from all of this?
• 14:58
  SPEAKER_02: The agents can call tools.
• 15:00
  SPEAKER_02: But it's really hard to like control which tools they call.
• 15:05
  SPEAKER_02: And if it's a chatbot, obviously you can show approved deny in the UI of your application.
• 15:09
  SPEAKER_02: But I kind of was obsessed with what I would call like outer loop agents
• 15:13
  SPEAKER_02: or proactive agents that would run in the background, get triggered by events.
• 15:16
  SPEAKER_02: I mean, OpenClaw is basically like the biggest manifestation of this,
• 15:19
  SPEAKER_02: of like you have a heartbeat,
• 15:20
  SPEAKER_02: it wakes up, it sees if there's any work to do, it tries to do stuff.
• 15:23
  SPEAKER_02: And my thought was like, I'm not going to trust that agent to do anything meaningful.
• 15:29
  SPEAKER_02: If I can't get like a Slack message or an iMessage or something when it wants
• 15:33
  SPEAKER_02: to do something and kind of guarantee deterministically that I can approve or deny
• 15:37
  SPEAKER_02: that or deny it with feedback and say, actually, no, do it like this.
• 15:40
  SPEAKER_02: So we played in that space for a while and...
• 15:43
  SPEAKER_02: Talk to a lot of founders and founding engineers and builders.
• 15:46
  SPEAKER_02: We came and did YC in the fall of 2024 with this idea.
• 15:49
  SPEAKER_02: We're building out this API platform and it was sort of like pager duty,
• 15:53
  SPEAKER_02: but like it wasn't who's on call to fix the servers.
• 15:55
  SPEAKER_02: It was like who's on call to this like routing mechanism for like who needs
• 15:59
  SPEAKER_02: to approve this agent and can they like escalate it or delegate it or defer it,
• 16:03
  SPEAKER_02: all this stuff.
• 16:04
  SPEAKER_02: And we built it for this ecosystem.
• 16:06
  SPEAKER_02: Great.
• 16:06
  SPEAKER_02: I like chain Bye.
• 16:07
  SPEAKER_02: Five.
• 16:08
  SPEAKER_02: There's so many grip tape.
• 16:09
  SPEAKER_02: There was so many in that in that time.
• 16:11
  SPEAKER_02: And then I talked to tons of AI engineers
• 16:12
  SPEAKER_02: who were actually building really interesting things
• 16:14
  SPEAKER_02: and like actually making money doing six figure contracts,
• 16:18
  SPEAKER_02: shipping AI to the enterprise.
• 16:20
  SPEAKER_02: And all of them had tried that stuff for like a month or two.
• 16:23
  SPEAKER_02: And then they had thrown it out and they were just writing all the API calls by hand.
• 16:27
  SPEAKER_02: And they were building more things that look more like pipelines and workflows
• 16:30
  SPEAKER_02: than these sort of like hands off call tools in a loop kind of thing.
• 16:34
  SPEAKER_02: And so I talked to a hundred people and I spent a lot of time,
• 16:37
  SPEAKER_02: a lot, a lot of time hanging out with one of my best friends, This is Vybov from...
• 16:40
  SPEAKER_02: from Boundary.
• 16:41
  SPEAKER_02: So they built a programming, they built like this like protobufs for AI thing.
• 16:45
  SPEAKER_02: And I think they're about to launch their like full fat,
• 16:47
  SPEAKER_02: like programming language Turing complete thing.
• 16:49
  SPEAKER_02: But he had this way of thinking about agents and building with models and building
• 16:53
  SPEAKER_02: with inference where it was a lot more about Understanding what's going on.
• 16:56
  SPEAKER_02: Structured output really is under the hood.
• 16:59
  SPEAKER_02: And every single step in your AI workflow is just tokens in, tokens out.
• 17:03
  SPEAKER_02: And your job as an engineer is figure out okay what tokens do I need to put
• 17:07
  SPEAKER_02: in maximize the chance that the tokens out are going to be good And kind
• 17:10
  SPEAKER_02: of distilled all these ideas into about 12 principles and wrote about it on GitHub,
• 17:14
  SPEAKER_02: posted just like this like 12 page GitHub repo,
• 17:17
  SPEAKER_02: threw it on Hacker News, got like 500, it was on the front page for like two days.
• 17:21
  SPEAKER_02: And I think it really resonated with a lot of people.
• 17:23
  SPEAKER_01: Yeah, so I'll just quickly read the 12 principles and then let's talk about like one
• 17:28
  SPEAKER_01: or two that resonate.
• 17:29
  SPEAKER_01: So the 12 are...
• 17:30
  SPEAKER_01: Natural language of tool calls.
• 17:32
  SPEAKER_01: Own your prompts,
• 17:33
  SPEAKER_01: own your context window.
• 17:35
  SPEAKER_01: Tools are just structured outputs.
• 17:37
  SPEAKER_01: Unify execution state and business state. Launch, pause,
• 17:40
  SPEAKER_01: resume with simple APIs.
• 17:42
  SPEAKER_01: Contact humans with tool calls,
• 17:43
  SPEAKER_01: own you control flow,
• 17:45
  SPEAKER_01: compact errors into context window,
• 17:47
  SPEAKER_01: small focused agents,
• 17:48
  SPEAKER_01: trigger from anywhere,
• 17:49
  SPEAKER_01: meet user where they are, make your agent a stateless reducer.
• 17:52
  SPEAKER_02: Haha the stateless yeah The stateless reducer one was a little,
• 17:56
  SPEAKER_02: actually someone hit me up on Twitter and Correct me.
• 17:59
  SPEAKER_02: Me.
• 17:59
  SPEAKER_02: It's actually a transducer because there's technically multiple steps in
• 18:02
  SPEAKER_02: the workflow, but
• 18:04
  SPEAKER_01: there we go.
• 18:06
  SPEAKER_01: But of this one, this was a year ago, So like, which is like forever.
• 18:09
  SPEAKER_01: Forever in And how the tooling is.
• 18:12
  SPEAKER_01: Is evolving.
• 18:13
  SPEAKER_01: Which ones Sticks still stick with you.
• 18:15
  SPEAKER_01: You wherever you're like, all right, these were good that still seem to hold off?
• 18:18
  SPEAKER_02: Yeah, I think I spent most of March writing it.
• 18:21
  SPEAKER_02: Published this in April, And then Swick's hit.
• 18:24
  SPEAKER_02: Hit me up from AI.Engineer, and he said, hey, can you come, you want to come talk about this?
• 18:28
  SPEAKER_02: So I gave this talk to 12-factor agents in like June 6th, I think, And, uh...
• 18:33
  SPEAKER_02: small room, maybe like, it was packed, but it was like maybe 100 people.
• 18:36
  SPEAKER_02: That was the year at AI Engineer where like the lower physically,
• 18:39
  SPEAKER_02: like on the second basement floor was all the super corporate stuff and you go up
• 18:43
  SPEAKER_02: a level, it's a little bit more.
• 18:44
  SPEAKER_02: And then like on the top floor is all the like weird cutting edge,
• 18:47
  SPEAKER_02: like startup stuff that like you probably shouldn't care about yet kind of thing.
• 18:51
  SPEAKER_02: So we were up there on the top of this like weird way of thinking about agents.
• 18:55
  SPEAKER_02: And then about a week later,
• 18:56
  SPEAKER_02: two weeks later,
• 18:57
  SPEAKER_02: Toby Lutke from Shopify says, I really like this idea of like context engineering.
• 19:01
  SPEAKER_02: And I'm like, I wrote about this two months ago.
• 19:04
  SPEAKER_02: This is great.
• 19:05
  SPEAKER_02: Toby gets it.
• 19:06
  SPEAKER_02: And then a week later,
• 19:07
  SPEAKER_02: Andre Karpathy is like, well,
• 19:08
  SPEAKER_02: I really like,
• 19:09
  SPEAKER_02: I think what we should think about is not prompt engineering,
• 19:11
  SPEAKER_02: but context engineering.
• 19:12
  SPEAKER_02: I was like, yes, that's my thing.
• 19:13
  SPEAKER_02: Anyways, I don't know.
• 19:14
  SPEAKER_02: If you ask Gemini, depends what day it is.
• 19:16
  SPEAKER_02: We'll tell you either me or Toby or Andre came up with context engineering.
• 19:21
  SPEAKER_02: You can't really own a word.
• 19:22
  SPEAKER_02: No one remembers who invented the word prompt engineering.
• 19:25
  SPEAKER_02: But of all the factors,
• 19:26
  SPEAKER_02: factor three of own your context window and basically the only way you can,
• 19:29
  SPEAKER_02: whether it's agentic or a single step at a pipeline,
• 19:32
  SPEAKER_02: the only way you can impact the quality of your output from AI is by carrying a lot
• 19:36
  SPEAKER_02: of lot about what the inputs and crafting them.
• 19:38
  SPEAKER_01: So let's talk about context engineering, which I am going to create.
• 19:42
  SPEAKER_01: Credit you that you coined it.
• 19:43
  SPEAKER_01: I did some research on, like, I think you were earlier, but a few days.
• 19:46
  SPEAKER_01: So there we go.
• 19:47
  SPEAKER_01: You coined it.
• 19:49
  SPEAKER_01: We're adding to SEO juice.
• 19:50
  SPEAKER_01: We'll have it in a transcript.
• 19:52
  SPEAKER_01: Dex coined context engineering.
• 19:54
  SPEAKER_02: And like asterisk Asterix on that.
• 19:55
  SPEAKER_02: That is basically like I learned about context engineering from talking
• 19:58
  SPEAKER_02: to these hundred engineers and founders.
• 20:00
  SPEAKER_02: I just kind of like what was the same about what they were all doing?
• 20:04
  SPEAKER_02: And I put a name on it.
• 20:05
  SPEAKER_02: So like I didn't invent doing it.
• 20:06
  SPEAKER_02: I was just like,
• 20:07
  SPEAKER_02: I think I think there's this thing and like vocabulary
• 20:09
  SPEAKER_02: and names are really important and having like clean ways to talk about the problem,
• 20:13
  SPEAKER_02: especially when like a lot of the content about AI right now is so much hype
• 20:17
  SPEAKER_02: and jargon that is like meaningless.
• 20:19
  SPEAKER_02: I was like, okay,
• 20:19
  SPEAKER_02: I think there's a word here that is useful to builders
• 20:22
  SPEAKER_02: that explains how they should be thinking about building their software.
• 20:25
  SPEAKER_01: So what is context engineering?
• 20:27
  SPEAKER_02: It's kind of like de-abstracting a lot of the abstractions that have been layered on top.
• 20:31
  SPEAKER_02: So you have a rag, you have memory, you have agentic history, you have structured output.
• 20:36
  SPEAKER_02: You have all these things that are like different ideas in the frame of agentic programming.
• 20:41
  SPEAKER_02: And at the end of the day,
• 20:42
  SPEAKER_02: they're all like different ways to pass tokens into a model and ask it
• 20:46
  SPEAKER_02: to produce usually some structured output.
• 20:48
  SPEAKER_02: And on.
• 20:49
  SPEAKER_02: And understanding this.
• 20:50
  SPEAKER_02: That is a lot more powerful than trying to learn memory and trying
• 20:55
  SPEAKER_02: to pick some agent framework off the shelf and some memory framework off the shelf.
• 20:59
  SPEAKER_02: I mean, these things are all really good if you want to get to like 80%,
• 21:02
  SPEAKER_02: you want to get a really good demo.
• 21:04
  SPEAKER_02: But when you have to go from 80% 90 % to 95%.
• 21:07
  SPEAKER_02: 95% or 99%,
• 21:09
  SPEAKER_02: you need to go down a level and think about what's everything we're putting into
• 21:12
  SPEAKER_02: the context window, what order is it going in, depending on which model we're doing.
• 21:16
  SPEAKER_02: And all of this stuff matters.
• 21:17
  SPEAKER_02: You have all of these levers that you can pull.
• 21:20
  SPEAKER_02: And it just felt like the right abstraction for thinking about how do I get AI to do
• 21:25
  SPEAKER_02: the thing I want as accurately as possible.
• 21:27
  SPEAKER_01: Why has context engineering started to become more talked about?
• 21:31
  SPEAKER_01: It was about a year ago.
• 21:32
  SPEAKER_01: Did it have to do with the context window that we could pass onto LLMs pretty much?
• 21:38
  SPEAKER_01: Did it start to expand or did we just start to realize that we can We do a lot more
• 21:43
  SPEAKER_01: by passing on from, you know, the easiest one is of course system prompts.
• 21:46
  SPEAKER_01: But of course whenever you build an LLM behind
• 21:50
  SPEAKER_01: the scenes you will pass additional context as well.
• 21:52
  SPEAKER_01: Not just to prompt the user,
• 21:54
  SPEAKER_01: you will add a bunch of stuff that's I guess a dirty secret about any LLMs
• 21:57
  SPEAKER_01: but why do you think the focus is moving on to like all right context is important?
• 22:01
  SPEAKER_02: I think it always was important.
• 22:20
  SPEAKER_02: There's It's just like the.
• 22:21
  SPEAKER_02: the, the ease.
• 22:23
  SPEAKER_02: Easiest way to get to really high quality AI applications is by thinking at
• 22:27
  SPEAKER_02: that token level,
• 22:28
  SPEAKER_02: thinking about a string of different LLM calls,
• 22:31
  SPEAKER_02: like rather than just tools in a loop and it's kind of open ended and very flexible,
• 22:35
  SPEAKER_02: but not that reliable.
• 22:37
  SPEAKER_02: Thinking of agents as workflows, as pipelines, as some mix between agents.
• 22:42
  SPEAKER_02: Maybe a couple tools in a loop versus just, hey,
• 22:44
  SPEAKER_02: I have my tools and I have my model and I have my system prompt and these are
• 22:48
  SPEAKER_02: the only levers I have.
• 22:49
  SPEAKER_02: And it's actually, no, you have way more levers.
• 22:51
  SPEAKER_02: It's going to take more work and you're going to have to like understand the LLM
• 22:54
  SPEAKER_02: with a deeper intuition.
• 22:56
  SPEAKER_02: But it was a thing that we always needed and it just took time for people to build
• 22:59
  SPEAKER_02: with this technology to figure out that like this is the layer of abstraction
• 23:03
  SPEAKER_02: that allows you to break through the quality ceiling.
• 23:06
  SPEAKER_01: And How are calls?
• 23:07
  SPEAKER_01: Cost and context engineering connected?
• 23:11
  SPEAKER_02: Yeah, I don't know.
• 23:12
  SPEAKER_02: I was talking about this with someone this morning Um, about like when.
• 23:16
  SPEAKER_02: you're working with LLM.
• 23:17
  SPEAKER_02: LLMs, one of the things I like to say is kind of like make it run, make it right, make it fast.
• 23:22
  SPEAKER_02: See if the world's best LLM at the time,
• 23:24
  SPEAKER_02: I think we did a podcast episode that at the time was like 03.
• 23:27
  SPEAKER_02: See if 03 can solve your problem.
• 23:29
  SPEAKER_02: And then give it to people and see if they want that.
• 23:32
  SPEAKER_02: And then if people want it and use it a lot,
• 23:34
  SPEAKER_02: then go do a bunch of context engineering because your engineering time is always
• 23:38
  SPEAKER_02: the bottleneck.
• 23:39
  SPEAKER_02: Like humans trying to figure out and solve problems and build evals and improve
• 23:43
  SPEAKER_02: and try different dimensions or set up JEPA or whatever it is,
• 23:46
  SPEAKER_02: is always going to be more expensive than just using a smarter model.
• 23:49
  SPEAKER_02: Until you have millions of requests a day.
• 23:51
  SPEAKER_02: And then it's like, okay,
• 23:52
  SPEAKER_02: we're going to do a bunch of context engineering,
• 23:54
  SPEAKER_02: break this up into three calls and get it to work on GPT-40.
• 23:56
  SPEAKER_02: And then we're going to take two of those and make those two work on GPT-40.
• 24:00
  SPEAKER_02: I'm using old model names.
• 24:01
  SPEAKER_02: But the point is like for a certain task in your workflow,
• 24:04
  SPEAKER_02: can you get GPT-OSS-120B, which is like one one thousandth of the cost of Opus?
• 24:10
  SPEAKER_02: Can you get it to solve parts of the problem so that the tokens and
• 24:13
  SPEAKER_02: the things you're using the smartest frontier models for are just the thing.
• 24:16
  SPEAKER_02: Things that you really need that level of intelligence?
• 24:18
  SPEAKER_02: But you shouldn't go build all of that and overengineer it until you've proved
• 24:21
  SPEAKER_02: that you need it,
• 24:22
  SPEAKER_02: that it's valuable,
• 24:23
  SPEAKER_02: that it's like, OK, this is now I'm going to get to Eli Goldratt.
• 24:26
  SPEAKER_02: And like, what is the he had this book, The Goal, right?
• 24:29
  SPEAKER_02: It was about how to model your factory.
• 24:30
  SPEAKER_02: And I'm sure we'll get to that when we talk about software factories.
• 24:32
  SPEAKER_02: It was like, what is the bottleneck in your system?
• 24:35
  SPEAKER_02: And one day it will be latency and cost.
• 24:36
  SPEAKER_02: But it's probably not that when you first start out.
• 24:38
  SPEAKER_02: And context engineering is how you move from the you add human effort to
• 24:43
  SPEAKER_02: the equation to improve the efficiency,
• 24:45
  SPEAKER_02: the speed, the price, the cost efficiency of your system.
• 24:49
  SPEAKER_02: Interesting.
• 24:50
  SPEAKER_01: And then one thing that came up more recently and a lot later recently is harness engineering.
• 24:56
  SPEAKER_01: What is harness engineering?
• 24:58
  SPEAKER_02: So I made a post in like October,
• 25:00
  SPEAKER_02: I think about,
• 25:01
  SPEAKER_02: or maybe November of like,
• 25:02
  SPEAKER_02: hey, there's this new thing that I see is like, I'm calling it harness engineering.
• 25:06
  SPEAKER_02: My definition that I had at the time is not what actually this guy Viv,
• 25:10
  SPEAKER_02: who's at Langchain now,
• 25:11
  SPEAKER_02: does a lot of really good writing on agents and how to think about harness.
• 25:14
  SPEAKER_02: And he had written something called harness engineering,
• 25:16
  SPEAKER_02: like A couple weeks before me, but I hadn't read it at that point.
• 25:19
  SPEAKER_02: And my take was basically like, okay, when you build an agent, you or use Context Engineering.
• 25:24
  SPEAKER_02: Engineering.
• 25:25
  SPEAKER_02: When you use an agent, because we gave this talk in August of 2025 about like how.
• 25:29
  SPEAKER_02: How to apply context engineering to how you use coding agents.
• 25:32
  SPEAKER_02: And that kind of evolved into this idea of like, How do you take a Farnace.
• 25:37
  SPEAKER_02: Like CloudCode, like Codex?
• 25:40
  SPEAKER_02: How do you engineer against the integration points of that harness?
• 25:43
  SPEAKER_02: So commands, MCU.
• 25:44
  SPEAKER_02: MCPs, skills, how you organize your code base?
• 25:47
  SPEAKER_02: How do you kind of optimize the environment that the coding agent runs in One, two.
• 25:51
  SPEAKER_02: To get the best results?
• 25:53
  SPEAKER_02: The same way with context engineer, how do you optimize
• 25:55
  SPEAKER_02: the inputs to every single prompt?
• 25:56
  SPEAKER_02: Well, harness engineering just is like,
• 25:58
  SPEAKER_02: how do I raise the floor so that every single turn of this thing,
• 26:02
  SPEAKER_02: the results are as good as possible?
• 26:04
  SPEAKER_01: And the term got super blurry.
• 26:06
  SPEAKER_01: And some people think harness engineering means building a harness.
• 26:08
  SPEAKER_02: And some people think harness engineering means building around a harness.
• 26:11
  SPEAKER_02: I actually like what Martin Fowler came up with.
• 26:14
  SPEAKER_02: As usual, he's very good at naming things.
• 26:15
  SPEAKER_02: And he kind of defined the you have the LLM And then you have the inner heart.
• 26:19
  SPEAKER_02: Harness,
• 26:20
  SPEAKER_02: which is like the thing that the tool definitions and the integration points
• 26:23
  SPEAKER_02: that like, say, like a cloud code or a codex or AMP actually exposes.
• 26:27
  SPEAKER_02: That's your inner harness.
• 26:29
  SPEAKER_02: And then you have the outer harness,
• 26:30
  SPEAKER_02: which is the stuff that you,
• 26:31
  SPEAKER_02: the human,
• 26:32
  SPEAKER_02: do to customize that for your specific needs, your code base, your languages, etc.
• 26:37
  SPEAKER_02: That's the best definition I think we have for harness engineering.
• 26:40
  SPEAKER_01: It's interesting how naming I think it's still.
• 26:42
  SPEAKER_01: so important.
• 26:43
  SPEAKER_01: Important, isn't it?
• 26:44
  SPEAKER_02: Well, it's like as soon as you name anything,
• 26:45
  SPEAKER_02: people or most people,
• 26:46
  SPEAKER_02: I'm actually surprised that context engineering still means the same thing
• 26:50
  SPEAKER_02: to most people that it did a year ago, and that it's even still relevant.
• 26:53
  SPEAKER_02: Like, that's honestly the The craziest thing
• 26:55
  SPEAKER_02: to me is like you wrote how many things are written
• 26:57
  SPEAKER_02: about AI 15 months ago still matter or still interesting or Or are still like...
• 27:03
  SPEAKER_02: have good advice baked into them.
• 27:06
  SPEAKER_02: Stuff changes a lot.
• 27:07
  SPEAKER_02: I think context engineering has been So long live.
• 27:09
  SPEAKER_02: Lived because it's it's grounded in the fundamentals of how transformer attention works.
• 27:16
  SPEAKER_02: And until we have post transformer models or linear attention or whatever it is,
• 27:20
  SPEAKER_02: which who knows when that's going to happen.
• 27:22
  SPEAKER_02: Context engineering will be interesting and important to anyone building on AI.
• 27:25
  SPEAKER_01: Can we talk about the physics of context?
• 27:30
  SPEAKER_01: You had a tweet, And this...
• 27:33
  SPEAKER_01: this one, the context reality check.
• 27:35
  SPEAKER_01: This is a graph of...
• 27:37
  SPEAKER_01: As you get to one million.
• 27:38
  SPEAKER_01: Million context, just the quality just drops.
• 27:42
  SPEAKER_01: It goes down.
• 27:43
  SPEAKER_01: What do we need to know about the consequences?
• 27:44
  SPEAKER_01: Because again, we now have models that do have a 1 million context window.
• 27:48
  SPEAKER_01: Maybe we'll have even longer ones.
• 27:50
  SPEAKER_01: But when you start to just put in more stuff into the context,
• 27:53
  SPEAKER_01: it starts to become less efficient.
• 27:54
  SPEAKER_01: Like,
• 27:55
  SPEAKER_01: what do we know so far in terms of from the practical perspective of like someone
• 27:58
  SPEAKER_01: who is using the context window to add on a bunch of stuff made at the end?
• 28:01
  SPEAKER_01: May that be MCP?
• 28:03
  SPEAKER_01: May that be tools?
• 28:03
  SPEAKER_01: May that be skills?
• 28:04
  SPEAKER_01: May that be all of these things?
• 28:06
  SPEAKER_02: Yeah, I mean, so the longer context windows Customers are good, you can.
• 28:09
  SPEAKER_02: Can talk to it for longer.
• 28:10
  SPEAKER_02: Like, they're doing a good job.
• 28:11
  SPEAKER_02: But at the end of the day, like, especially when you had, like, Opus...
• 28:15
  SPEAKER_02: It was, like, Opus 4 .5 and
• 28:17
  SPEAKER_02: then Opus 4 .5, Opus 4.51 mil or 4.6 and 4.61 mil.
• 28:21
  SPEAKER_02: You're not actually getting a, like...
• 28:24
  SPEAKER_02: Smarter model.
• 28:25
  SPEAKER_02: Like the intelligence of the model is what drives its ability to attend To all of
• 28:30
  SPEAKER_02: the tokens in the context window To figure out on the next turn Which parts
• 28:34
  SPEAKER_02: of this 100k or 200k context window Are the most relevant to making the decision
• 28:38
  SPEAKER_02: I don't know.
• 28:55
  SPEAKER_02: Instructions before it starts to drop off.
• 28:57
  SPEAKER_02: Their ability to follow all the instructions just like drops off pretty quickly.
• 29:00
  SPEAKER_02: And I think Lori Voss at Arise,
• 29:02
  SPEAKER_02: I haven't actually looked at the data,
• 29:04
  SPEAKER_02: but they did a study with like the next generation models a year later.
• 29:07
  SPEAKER_02: And it looks like it's like much better the number of instructions you can get in.
• 29:10
  SPEAKER_02: But in any case you have like, I split context engineering into like two categories.
• 29:15
  SPEAKER_02: you have like the most people think about like the information budget
• 29:19
  SPEAKER_02: of okay I can do rag and I can pull out chunks of this document Rather than putting
• 29:24
  SPEAKER_02: the entire book into my context window, I can just go grab the pages that matter.
• 29:29
  SPEAKER_02: But it's also your instruction budget is like if you give
• 29:31
  SPEAKER_02: the model too many instructions and especially too many conflicting instructions,
• 29:34
  SPEAKER_02: and that's in your initial prompt and also like.
• 29:37
  SPEAKER_02: If you have a conversation and you start going down a path,
• 29:39
  SPEAKER_02: and then you change your mind and you're like,
• 29:42
  SPEAKER_02: actually I don't want to do any of that.
• 29:42
  SPEAKER_02: I want to this.
• 29:43
  SPEAKER_02: It's a lot of computation the model has to do to notice that it has to ignore
• 29:48
  SPEAKER_02: that whole thing And when both of those things are kind of far back enough in
• 29:52
  SPEAKER_02: the context window that they're only half getting attended to Your likelihood
• 29:56
  SPEAKER_02: that it's actually going to remember the exact instructions you gave
• 30:00
  SPEAKER_02: at 100 ,000 tokens ago goes down quite significantly
• 30:03
  SPEAKER_01: This is all very interesting
• 30:04
  SPEAKER_01: because as engineers,
• 30:06
  SPEAKER_01: we are expected when You know,
• 30:08
  SPEAKER_01: when we're AI and engineers,
• 30:09
  SPEAKER_01: which now a lot of software engineers are,
• 30:10
  SPEAKER_01: meaning you just use LLMs to to build software like software.
• 30:14
  SPEAKER_01: Underneath, there's an LLM layer somewhere.
• 30:16
  SPEAKER_01: You're an AI engineer, congratulations.
• 30:18
  SPEAKER_01: But it sounds like the expectation is To be a good, to be.
• 30:21
  SPEAKER_01: Good software engineer pre-AI, you need to understand how to write good code.
• 30:26
  SPEAKER_01: And it helps when you understand a little bit of the underlying.
• 30:29
  SPEAKER_01: We didn't need to do that that much over time, but it never hurts.
• 30:33
  SPEAKER_01: It sounds like right now we're in this phase that to be an engineer who can write
• 30:38
  SPEAKER_01: an efficient AI system that use LLMs,
• 30:42
  SPEAKER_01: you need to understand the dynamics of the content.
• 30:45
  SPEAKER_01: Context.
• 30:45
  SPEAKER_01: You need to understand why Stuffing your concept.
• 30:48
  SPEAKER_01: Context one way or the other can Maybe compute can introduce latents.
• 30:51
  SPEAKER_01: Latency.
• 30:52
  SPEAKER_01: And all of these, it sounds like it's kind of More of an intuition.
• 30:56
  SPEAKER_01: And of course,
• 30:57
  SPEAKER_01: there's some understanding,
• 30:58
  SPEAKER_01: but from talking to you, you're like, well, it does this competition.
• 31:00
  SPEAKER_01: Computation.
• 31:01
  SPEAKER_01: You know because you've tried it out, right?
• 31:03
  SPEAKER_02: I'm not a PhD in machine learning.
• 31:05
  SPEAKER_02: I couldn't actually go draw a mathematical proof of how this works,
• 31:09
  SPEAKER_02: but we know attention is quadratic.
• 31:10
  SPEAKER_02: And the more stuff you put in, the more it has to spread this attention out over everything.
• 31:16
  SPEAKER_01: This just feels like an Absolutely new area.
• 31:18
  SPEAKER_01: Area and a little bit very different to what we're used to software engineering,
• 31:21
  SPEAKER_01: which is pretty kind of black and white, right?
• 31:24
  SPEAKER_01: It compiles or doesn't compile.
• 31:25
  SPEAKER_02: That's true.
• 31:26
  SPEAKER_02: I mean, there's a different kind of intuition.
• 31:27
  SPEAKER_02: I was talking about this earlier as well.
• 31:28
  SPEAKER_02: I was like,
• 31:29
  SPEAKER_02: there's a different kind of intuition that you develop over years as
• 31:32
  SPEAKER_02: a software engineer.
• 31:33
  SPEAKER_02: And, uh...
• 31:34
  SPEAKER_02: there's many categories of it.
• 31:36
  SPEAKER_02: But the one I'll call attention to that is like a thing that you cannot teach,
• 31:40
  SPEAKER_02: you cannot learn in a textbook.
• 31:42
  SPEAKER_02: The only way to learn it is like,
• 31:43
  SPEAKER_02: I know bad pa - bad patterns in software because I have debugged them at three in
• 31:47
  SPEAKER_02: the morning.
• 31:47
  SPEAKER_02: This is my buddy Jake from Netflix said this in his talk at AI Engineer Code.
• 31:51
  SPEAKER_02: It's just like,
• 31:51
  SPEAKER_02: There's no better way to learn what is good and what is bad and what works
• 31:55
  SPEAKER_02: and what doesn't than And suffering through the...
• 31:57
  SPEAKER_02: the thing that doesn't work.
• 31:58
  SPEAKER_01: Well, speaking of suffering through the things that doesn't work,
• 32:02
  SPEAKER_01: a new paradigm that is spreading Spartacus is a loop.
• 32:05
  SPEAKER_01: Loops.
• 32:06
  SPEAKER_01: Loop engineering, the idea that instead of writing prompts, just write loops, set up your loops.
• 32:10
  SPEAKER_01: And this all started with the Ralph Wiggum technique where it will just,
• 32:14
  SPEAKER_01: well, I guess that's an early version of loops that were just loops around.
• 32:18
  SPEAKER_01: And now we're hearing what some of the biggest labs are.
• 32:21
  SPEAKER_01: Talking about that, they're actually just doing loop venturing.
• 32:24
  SPEAKER_01: What is your take on, have you done some loop venturing yourself?
• 32:28
  SPEAKER_01: Have you set up some loops?
• 32:30
  SPEAKER_01: And what do you think is good about it?
• 32:33
  SPEAKER_01: And what do you think is bad about it?
• 32:35
  SPEAKER_02: Yeah.
• 32:35
  SPEAKER_02: So I think of loops as, I mean, this could, I could ramble on this for 10 minutes.
• 32:39
  SPEAKER_02: This is an entire talk, but I'll try to, I'll try to Lay out some highlights.
• 32:43
  SPEAKER_02: Level stuff and then we can dig in wherever you think is most interesting.
• 32:45
  SPEAKER_02: We had Ralph Wiggum.
• 32:46
  SPEAKER_02: It was actually A year and four days ago was the first time I I saw the Ralph Wiggins.
• 32:50
  SPEAKER_02: Wiggum demo and like Jeff Hundley was just like visiting SF and he just like came
• 32:54
  SPEAKER_02: through and like dropped everybody's jaws
• 32:57
  SPEAKER_02: with his like yeah I just ran Sonnet around the clock and spent six grand
• 33:00
  SPEAKER_02: in six weeks and like I built an entire Gen Z programming language and look
• 33:04
  SPEAKER_02: at it compiles and it has a stage two compiler where the compiler for
• 33:07
  SPEAKER_02: the language is written in the language itself and all that.
• 33:10
  SPEAKER_02: Insane.
• 33:11
  SPEAKER_02: And the core lesson from all of that,
• 33:14
  SPEAKER_02: I think,
• 33:14
  SPEAKER_02: was the idea of back pressure,
• 33:17
  SPEAKER_02: which is basically,
• 33:18
  SPEAKER_02: and I think a lot of people were doing this for a very,
• 33:21
  SPEAKER_02: have been doing this for a long time,
• 33:22
  SPEAKER_02: which is how do I let the model check its own work?
• 33:26
  SPEAKER_02: How do I automate the process of getting feedback into the model?
• 33:29
  SPEAKER_02: And there's lots and lots of different flavors of this.
• 33:32
  SPEAKER_02: You can have deterministic linters.
• 33:34
  SPEAKER_02: You can have unit tests.
• 33:36
  SPEAKER_02: Like part of what made the programming language easy to build with Ralph is
• 33:38
  SPEAKER_02: a programming language can be infinitely verified.
• 33:41
  SPEAKER_02: You write the code in the language.
• 33:43
  SPEAKER_02: You compile it.
• 33:43
  SPEAKER_02: If the compiler fails, you go fix the compiler.
• 33:46
  SPEAKER_02: You run the program.
• 33:47
  SPEAKER_02: If the program fails, you go fix the compiler.
• 33:49
  SPEAKER_02: Like it's like It's very, very verifiable.
• 33:53
  SPEAKER_02: And I think the lesson in loops engineering is like,
• 33:55
  SPEAKER_02: if you can make a problem very verifiable,
• 33:59
  SPEAKER_02: you can kind of like treat it like a black box.
• 34:03
  SPEAKER_01: And then have it loop because it will keep improving itself because of
• 34:06
  SPEAKER_01: the verification loop is already there.
• 34:08
  SPEAKER_02: Exactly.
• 34:09
  SPEAKER_02: And so like, you can do this with CICD is like, I do this every time I do a release.
• 34:12
  SPEAKER_02: I'm like, I'm tired.
• 34:13
  SPEAKER_02: The CICD is slow.
• 34:14
  SPEAKER_02: Cool.
• 34:14
  SPEAKER_02: Go research the code base, make a change, make a pull request, run the test, see if it's faster.
• 34:19
  SPEAKER_02: Try again, run the test, push to the branch, check again.
• 34:24
  SPEAKER_02: And so it's like if it can verify its own work in a loop,
• 34:26
  SPEAKER_02: instead of saying,
• 34:28
  SPEAKER_02: let's try this approach or let's try that approach and being really back and forth,
• 34:33
  SPEAKER_02: you just say like, my goal is to make CI faster.
• 34:36
  SPEAKER_02: And you tell the model, here's the steps.
• 34:37
  SPEAKER_02: Here's the five steps.
• 34:38
  SPEAKER_02: You're going to write some code.
• 34:40
  SPEAKER_02: You're going to commit it.
• 34:40
  SPEAKER_02: You're going to push it.
• 34:41
  SPEAKER_02: You're going to launch a subagent to watch the job until it's finished.
• 34:44
  SPEAKER_02: It's going to tell you what happened.
• 34:46
  SPEAKER_02: Then you're going to decide what to do next.
• 34:47
  SPEAKER_02: And so that's like the very simplest example I have of like, Designing loops.
• 34:52
  SPEAKER_01: And And you just said the...
• 34:52
  SPEAKER_01: the goal, which is It's Claude Coulton.
• 34:55
  SPEAKER_01: Code, and I think codecs have both chip slash goal,
• 34:57
  SPEAKER_01: which is you just set the goal and it iterates until it reaches it or as long
• 35:01
  SPEAKER_01: as it makes progress towards it.
• 35:03
  SPEAKER_02: Exactly.
• 35:04
  SPEAKER_02: And so it's like, if it's verifiable, if you can measure, this is auto research too.
• 35:08
  SPEAKER_02: Auto research is like, hey, go make this model twice as fast.
• 35:10
  SPEAKER_02: And like,
• 35:11
  SPEAKER_02: it's just a prompt that tells the model, like,
• 35:13
  SPEAKER_02: go to it over and over again and try things until it actually has good results.
• 35:16
  SPEAKER_01: So that's what I think of loops engineering.
• 35:18
  SPEAKER_01: I don't know.
• 35:18
  SPEAKER_01: We do a very interesting kind of loops
• 35:20
  SPEAKER_02: engineering where like,
• 35:21
  SPEAKER_02: The challenge is like,
• 35:23
  SPEAKER_02: I think it's very easy to get very excited about building the thing that builds
• 35:27
  SPEAKER_02: the thing or building a thing that builds the thing we talked about.
• 35:31
  SPEAKER_02: And so people say, oh, we need to like redo everything as this big, Like, a gentic...
• 35:36
  SPEAKER_02: agentic first.
• 35:37
  SPEAKER_02: Factory, maybe even a dark.
• 35:38
  SPEAKER_02: Dark factory.
• 35:39
  SPEAKER_02: And they're like redesigning their entire thing to be their infrastructure for
• 35:42
  SPEAKER_02: the next five years.
• 35:43
  SPEAKER_02: And I'm sure one thing we know of in engineering And especially practice.
• 35:46
  SPEAKER_02: Especially pragmatic engineering is uh how how can you make this more incremental?
• 35:50
  SPEAKER_02: How can you make it more continuous?
• 35:52
  SPEAKER_02: And a lot of people don't have to do that.
• 35:53
  SPEAKER_02: Have the option to just, hey,
• 35:54
  SPEAKER_02: I ran a Ralph loop for three days and it fixed every linter error in our code base.
• 35:59
  SPEAKER_02: Here's a 60,000...
• 36:00
  SPEAKER_02: 60,000 line PR.
• 36:01
  SPEAKER_02: Who wants to review it and who wants to sign off on merging and deploying it and
• 36:05
  SPEAKER_02: that there's not going to be any bugs?
• 36:07
  SPEAKER_02: Nobody.
• 36:07
  SPEAKER_02: So I think
• 36:08
  SPEAKER_02: the thing I'm most excited is actually like what we call like iterated loops
• 36:11
  SPEAKER_02: or like slow loops where we basically have a cron job.
• 36:14
  SPEAKER_02: We have the loop.
• 36:15
  SPEAKER_02: The structure of the loop is really easy.
• 36:17
  SPEAKER_02: It's like run this linter, fix one thing, commit and push.
• 36:21
  SPEAKER_02: And then we run that every night in our GitHub actions.
• 36:24
  SPEAKER_02: And we wake up every morning to one PR that makes the code base a little bit better.
• 36:27
  SPEAKER_01: Mmm, I like the stuff.
• 36:29
  SPEAKER_01: Slow loops.
• 36:29
  SPEAKER_01: Yeah.
• 36:30
  SPEAKER_02: And it has two dimensions.
• 36:31
  SPEAKER_02: So you can add, now we have a blueprint for it.
• 36:33
  SPEAKER_02: And actually Kyle just shipped a skill so that you can build these yourself.
• 36:36
  SPEAKER_02: You can add more Like, feedback mechanism.
• 36:38
  SPEAKER_02: Mechanisms.
• 36:39
  SPEAKER_02: So we have React Doctor for the front end.
• 36:41
  SPEAKER_02: We have another anti-pattern.
• 36:42
  SPEAKER_02: And that has no deterministic tooling.
• 36:44
  SPEAKER_02: But Kyle's just like, here's what good looks like.
• 36:45
  SPEAKER_02: Here's what bad looks like.
• 36:46
  SPEAKER_02: Go fix one thing and bring it back.
• 36:47
  SPEAKER_02: It was like prop narrowing, basically.
• 36:49
  SPEAKER_02: We have a bunch of optional props and most of them don't need to be optional.
• 36:51
  SPEAKER_02: It's like,
• 36:52
  SPEAKER_02: here's how to make the prop not optional so that you know that
• 36:54
  SPEAKER_02: the code just is like cleaner and easier to reason about.
• 36:57
  SPEAKER_02: And so you can add more conditions, more things of like fix one thing.
• 37:01
  SPEAKER_02: I want to wake up to a PR.
• 37:02
  SPEAKER_02: So now we wake up to like four PRs because there's four separate things.
• 37:05
  SPEAKER_02: And then the other dimension you can do here is as you gain confidence,
• 37:08
  SPEAKER_02: you can increase the scope.
• 37:09
  SPEAKER_02: Instead of fixing one thing, fix four things.
• 37:12
  SPEAKER_02: And so these are like other ways to think about loops
• 37:14
  SPEAKER_02: where it's like something that's not a human triggers it to start,
• 37:18
  SPEAKER_02: whether it's You know, an alert from Sentry.
• 37:20
  SPEAKER_02: Whether it's a user feedback support ticket.
• 37:21
  SPEAKER_02: User Feedback.
• 37:24
  SPEAKER_02: Whether it is PM writes a ticket.
• 37:26
  SPEAKER_02: Whether it has test is failing or it runs on a schedule.
• 37:31
  SPEAKER_02: But the trigger should be something that you don't have to press a button on
• 37:34
  SPEAKER_02: and there's defined workflow and it makes everything a little bit better.
• 37:38
  SPEAKER_00: That just describes letting agents fix things without a human pressing a button.
• 37:42
  SPEAKER_00: But what if a bug is too difficult,
• 37:44
  SPEAKER_00: not just for an agent, but also for a human to reproduce, let alone fix?
• 37:48
  SPEAKER_00: This is where presenting sponsor Antithesis comes in.
• 37:50
  SPEAKER_00: I was recently pairing with the Antithesis team,
• 37:52
  SPEAKER_00: where we did a walkthrough of how they helped fix a nasty bug in etcd,
• 37:56
  SPEAKER_00: the open source key value store used by Kubernetes.
• 37:59
  SPEAKER_00: This is a bug that actually happened in etcd.
• 38:01
  SPEAKER_00: The team noticed that the linearization validation assertion failed during
• 38:04
  SPEAKER_00: the regular Antithesis runs.
• 38:06
  SPEAKER_00: This is not good because the linearization guarantees strong consistency,
• 38:09
  SPEAKER_00: so this needs to be fixed.
• 38:11
  SPEAKER_00: So what the etcd team did was run a casualty analysis inside Antithesis.
• 38:16
  SPEAKER_00: This generates this graph, which is a bug probability graph.
• 38:20
  SPEAKER_00: Here the x-axis is virtual time And the y -axis is
• 38:24
  SPEAKER_00: Y-axis is probability.
• 38:25
  SPEAKER_00: Now we see that something happened just before virtual time 24 that caused
• 38:29
  SPEAKER_00: a huge jump in the probability that the bug would occur.
• 38:32
  SPEAKER_00: Going deeper, we can look at the entire set of timelines.
• 38:35
  SPEAKER_00: Vertical lines going down represent events branching off from the same state.
• 38:39
  SPEAKER_00: And the purple also dots are where the bug happens.
• 38:42
  SPEAKER_00: If we look closely enough, we see that all of the failures come from one parent branch.
• 38:46
  SPEAKER_00: Gotcha.
• 38:47
  SPEAKER_00: This is such a useful debugging tool.
• 38:49
  SPEAKER_00: In the end,
• 38:49
  SPEAKER_00: a team was able to figure out that process pauses were causing
• 38:52
  SPEAKER_00: the bug using all these Antisys debugging tools.
• 38:54
  SPEAKER_00: This non-deterministic bug was diagnosed in a deterministic way!
• 38:59
  SPEAKER_00: How cool is that?
• 39:00
  SPEAKER_00: Oh and this an actual bug then got fixed in etcd.
• 39:03
  SPEAKER_00: You can see the bug and fix on etcd's github repo.
• 39:06
  SPEAKER_00: Honestly,
• 39:07
  SPEAKER_00: the tools that Antisysis built for debugging feel pretty darn futuristic
• 39:10
  SPEAKER_00: but they are also really powerful.
• 39:11
  SPEAKER_00: Head over to antisis .com slash pragmatic to learn more I'd like to talk
• 39:15
  SPEAKER_00: about our season Sponsor Sentry.
• 39:18
  SPEAKER_00: Sentry is a tool I use for application monitoring on all of my projects,
• 39:21
  SPEAKER_00: including their pragmatic engine backend.
• 39:23
  SPEAKER_00: I've used it for 10 years now, starting with when I worked at Uber.
• 39:26
  SPEAKER_00: A neat Sentry feature I'm liking is their Sear AI agent,
• 39:29
  SPEAKER_00: which helps investigate production errors.
• 39:31
  SPEAKER_00: For example, here's an actual error I had in my application.
• 39:34
  SPEAKER_00: I can just ask Sear what might be the root cause,
• 39:36
  SPEAKER_00: and it brings context,
• 39:37
  SPEAKER_00: and it can also make a plan to fix it right from the web interface.
• 39:40
  SPEAKER_00: And the nice thing is how Sear also works great from Slack as well, not just from the web.
• 39:44
  SPEAKER_00: One place I find even more handy to use Sentry is from codex or clock code using Sentry MCP.
• 39:49
  SPEAKER_00: Also, you can set up neat automations like when a resolve Sentry issue resurfaces,
• 39:53
  SPEAKER_00: you can kick off a cursor agent or GitHub Copilot agent to investigate
• 39:56
  SPEAKER_00: the regression, Read their relevant code.
• 39:58
  SPEAKER_00: Code, and open a PR with a suggested fix.
• 40:00
  SPEAKER_00: I'm not a fan for using AI tools just for the sake of it,
• 40:03
  SPEAKER_00: but I really like the practical integrations where I can fix errors faster and
• 40:07
  SPEAKER_00: with more context.
• 40:08
  SPEAKER_00: Check out Sentry at sentry.io slash pragmatic and start monitoring and fixing regressions today.
• 40:14
  SPEAKER_00: And with this, let's get back to Dex and to agentic loops that trigger themselves.
• 40:18
  SPEAKER_01: Now, you said we can get more ambitious and we can add more things to it,
• 40:22
  SPEAKER_01: but I'm going to quote you with one of your tweets,
• 40:26
  SPEAKER_01: which says,
• 40:27
  SPEAKER_01: this may surprise you that this is coming from me,
• 40:29
  SPEAKER_01: but I think we're in for a one to three year period where stuff might break at 3am
• 40:33
  SPEAKER_01: and you're relying on loops to fix it and nobody understands what's under the hood
• 40:37
  SPEAKER_01: and you're looking at it Ex -extension.
• 40:38
  SPEAKER_01: Existential threat to your company.
• 40:40
  SPEAKER_02: Yes.
• 40:41
  SPEAKER_02: Yeah, that one was great.
• 40:42
  SPEAKER_02: That one did a lot of numbers.
• 40:43
  SPEAKER_02: Uh.
• 40:45
  SPEAKER_01: It resonated.
• 40:46
  SPEAKER_01: Here's the other side of
• 40:46
  SPEAKER_02: it is like,
• 40:47
  SPEAKER_02: I think that the today with today's models,
• 40:50
  SPEAKER_02: today's programming languages, today's infrastructure, You m -
• 40:53
  SPEAKER_02: You might get a away with not reading the code.
• 40:56
  SPEAKER_02: Problem with loops is like at a certain point,
• 40:58
  SPEAKER_02: you're going to generate so much code that you can't read it anymore.
• 41:01
  SPEAKER_02: This is The Strong BM.
• 41:02
  SPEAKER_02: StrongDM Dark Factory.
• 41:03
  SPEAKER_02: This is like Ryan LaPopolo's like Harness Engineering.
• 41:05
  SPEAKER_02: Just spend as many tokens as possible.
• 41:08
  SPEAKER_02: We tried this.
• 41:08
  SPEAKER_02: We built a Lightsoft software factory in July of 2020.
• 41:12
  SPEAKER_02: 2025.
• 41:12
  SPEAKER_02: And by November, we had shut it down.
• 41:14
  SPEAKER_02: I think it takes about three to six months of you shipping all the time
• 41:17
  SPEAKER_02: with nobody reading the code before you realize like, wow,
• 41:20
  SPEAKER_02: this is getting way worse and it's easier to start over than it is to fix it.
• 41:23
  SPEAKER_02: Like the models have made the code base so bad that it is actually going
• 41:26
  SPEAKER_02: to be easier to just rethink this from scratch.
• 41:28
  SPEAKER_02: Maybe that's OK because we have AI and it'S easier to rebuild things from nothing.
• 41:34
  SPEAKER_02: And usually
• 41:45
  SPEAKER_02: when engineers
• 41:50
  SPEAKER_02: say like,
• 41:54
  SPEAKER_02: oh, we
• 42:00
  SPEAKER_02: can't fix this, we have
• 42:29
  SPEAKER_02: To get models to write code that is more maintainable
• 42:33
  SPEAKER_02: over time versus They're all trained on sweepers.
• 42:35
  SPEAKER_02: Sweebench and Sweebench looking things, right?
• 42:37
  SPEAKER_02: All of the benchmarks are basically like, here's a commit in Django.
• 42:41
  SPEAKER_02: Here's an issue that was filed around that time.
• 42:44
  SPEAKER_02: See if you can create the fix that the human created.
• 42:46
  SPEAKER_02: And it's Django and it's Apache and there's 100 repos in Go and C++ and TypeScript
• 42:51
  SPEAKER_02: and Java and all these different languages.
• 42:53
  SPEAKER_02: But they're It's all, it's like the.
• 42:54
  SPEAKER_02: The problem with training models on maintainability is like,
• 42:58
  SPEAKER_02: The cost function of bad architecture and bad program design can't be evaluated
• 43:02
  SPEAKER_02: by running the unit test because it hits you three to six months later
• 43:05
  SPEAKER_02: when you're like, holy crap, the The software has become smart.
• 43:09
  SPEAKER_02: So hard to change.
• 43:10
  SPEAKER_01: Is this not similar to how...
• 43:12
  SPEAKER_01: Senior software engineers,
• 43:14
  SPEAKER_01: Here's why it took took years for someone to become a senior,
• 43:17
  SPEAKER_01: because typically,
• 43:18
  SPEAKER_01: and in some environments,
• 43:19
  SPEAKER_01: you can become a senior faster,
• 43:21
  SPEAKER_01: typically fast moving,
• 43:22
  SPEAKER_01: where there's a bunch of issues and you have to keep fixing it.
• 43:25
  SPEAKER_01: Sometimes,
• 43:25
  SPEAKER_01: you know,
• 43:25
  SPEAKER_01: some people are working in the same place for 10 years and they're still not
• 43:28
  SPEAKER_01: that level.
• 43:29
  SPEAKER_01: The point was, it just takes time for you to understand the The small mistake that you made.
• 43:33
  SPEAKER_01: You make right now that snowballs into like something disastrous later.
• 43:37
  SPEAKER_01: And You get hit by it and you realize like, okay,
• 43:40
  SPEAKER_01: things like,
• 43:41
  SPEAKER_01: you know,
• 43:41
  SPEAKER_01: like testing matters, architecture matters, tech debt can actually be a killer.
• 43:45
  SPEAKER_01: You know,
• 43:45
  SPEAKER_01: we don't talk about it anymore,
• 43:46
  SPEAKER_01: but we used to talk about how tech debt kills or slows down companies
• 43:49
  SPEAKER_01: so badly Pre -AI pre-AI that their competitors can overtake them
• 43:53
  SPEAKER_01: or they're just like stuck with a two-year refactor not shipping any new features
• 43:57
  SPEAKER_01: and the competition, you know, ships a bunch of other stuff and now they're ahead.
• 44:01
  SPEAKER_02: And I will say like, It is possible that GPT-7 will fix this.
• 44:06
  SPEAKER_02: But if you are turning the lights off in your software factory
• 44:09
  SPEAKER_02: and you're saying like, hey, you know what?
• 44:11
  SPEAKER_02: Like, we're not going to read the code.
• 44:12
  SPEAKER_02: It's fine.
• 44:13
  SPEAKER_02: The models are smart enough.
• 44:14
  SPEAKER_02: If we give it the right feedback and just throw enough tokens at the problem,
• 44:17
  SPEAKER_02: it will keep getting better.
• 44:18
  SPEAKER_02: This is what led to this tweet.
• 44:19
  SPEAKER_02: Like, That might work,
• 44:21
  SPEAKER_02: but if nobody read the code in three months and you replace all of your I love your,
• 44:25
  SPEAKER_02: like, code review with Luke.
• 44:26
  SPEAKER_02: Loops of like, hey, if a user complains, we give it to an agent.
• 44:29
  SPEAKER_02: If something crashes, we give it to an agent.
• 44:31
  SPEAKER_02: If a PM writes a ticket, we give it to an agent.
• 44:33
  SPEAKER_02: If a CEO writes an obnoxious essay about what we should be building in Slack,
• 44:36
  SPEAKER_02: we give it to an agent.
• 44:38
  SPEAKER_02: And then you stop reading the code because that's going to produce way too much.
• 44:41
  SPEAKER_02: Like no one can read it.
• 44:42
  SPEAKER_02: And like the PR reviews become the bottleneck.
• 44:44
  SPEAKER_02: So you replace that with agentic testing and agentic Agenda code review.
• 44:48
  SPEAKER_02: Review.
• 44:49
  SPEAKER_02: But none of these things have intuition for software architecture
• 44:52
  SPEAKER_02: because we haven't trained it in yet.
• 44:53
  SPEAKER_02: And so you're going to wake up one day and you're going to have an issue
• 44:55
  SPEAKER_02: with this happened to us.
• 44:56
  SPEAKER_02: And like, we got through it.
• 44:57
  SPEAKER_02: And at the time, it was still worth it.
• 44:59
  SPEAKER_02: It was like,
• 45:00
  SPEAKER_02: spent three weeks onboarding back into the code base
• 45:02
  SPEAKER_02: that we had stopped reading three months ago,
• 45:05
  SPEAKER_02: because no matter how much sophisticated expert prompting,
• 45:08
  SPEAKER_02: we could not get Opus.
• 45:09
  SPEAKER_02: I think it was Opus 4 .1 At the time,
• 45:11
  SPEAKER_02: we could not get Opus 4 .1 to actually find the root cause.
• 45:14
  SPEAKER_02: We had to go spend several days digging through the code and figuring out like, oh,
• 45:18
  SPEAKER_02: there's just actually a primary key that's being routed through this whole thing
• 45:21
  SPEAKER_02: that needs to be changed to a different type of object than it needs its own table.
• 45:24
  SPEAKER_02: Oh, so this actually happened to you?
• 45:24
  SPEAKER_02: This happened to us, yeah.
• 45:26
  SPEAKER_02: And when it happened, I was like, you know what?
• 45:27
  SPEAKER_02: That sucked.
• 45:28
  SPEAKER_02: That was terrible.
• 45:29
  SPEAKER_02: But we did it.
• 45:29
  SPEAKER_02: We solved it.
• 45:30
  SPEAKER_02: And And it's still worth it.
• 45:32
  SPEAKER_02: Still worth not reading the code for most of the time at the cost of every once in a
• 45:36
  SPEAKER_02: while, I'm going to have to spend two weeks fixing an issue by hand.
• 45:39
  SPEAKER_02: And I don't believe that anymore because I think the amount of code we're able
• 45:42
  SPEAKER_02: to write now is actually like 10x or 100x.
• 45:44
  SPEAKER_02: And I think the problem is just getting worse.
• 45:47
  SPEAKER_01: So let's talk about software factories.
• 45:49
  SPEAKER_01: Yeah.
• 45:49
  SPEAKER_01: In your mind, because I feel it's an overloaded word, but What do you think of it?
• 45:53
  SPEAKER_01: Of a software factory before AI and now post AI?
• 45:57
  SPEAKER_01: Do you know the first definition of software factory the first time it
• 45:59
  SPEAKER_02: was used?
• 46:00
  SPEAKER_02: No.
• 46:01
  SPEAKER_02: It was a NATO conference in 1968.
• 46:03
  SPEAKER_02: Oh, great.
• 46:04
  SPEAKER_02: Grady, which would know about this.
• 46:05
  SPEAKER_02: Yeah, exactly.
• 46:05
  SPEAKER_02: Yeah, great.
• 46:06
  SPEAKER_02: You should ask Grady about it.
• 46:07
  SPEAKER_02: They talked about the idea of like,
• 46:08
  SPEAKER_02: OK, you actually need to build a system of steps and like just like a factory floor.
• 46:13
  SPEAKER_02: You have like the coding part and the testing part and the validation part and
• 46:17
  SPEAKER_02: the integration part.
• 46:18
  SPEAKER_02: We had no system.
• 46:18
  SPEAKER_02: CICD.
• 46:19
  SPEAKER_02: We barely had version control, Like, but you needed a factory.
• 46:22
  SPEAKER_02: Factory.
• 46:22
  SPEAKER_02: And then it was adopted by They like Toshiba.
• 46:25
  SPEAKER_02: Toshiba and a bunch of companies.
• 46:27
  SPEAKER_02: And then And the next moment was like, was DevOps.
• 46:29
  SPEAKER_02: And you have this idea of, OK, we're going to do CICD.
• 46:32
  SPEAKER_02: We're going to automate.
• 46:33
  SPEAKER_02: We're going to use Chef.
• 46:33
  SPEAKER_02: Ansible, Puppet, whatever.
• 46:34
  SPEAKER_02: All these technologies is like,
• 46:36
  SPEAKER_02: instead of having dudes running around data centers,
• 46:38
  SPEAKER_02: like resizing disks and stuff, or clicking around the AWS console.
• 46:42
  SPEAKER_02: Yeah, exactly.
• 46:43
  SPEAKER_02: It was like, cool, we build loops.
• 46:45
  SPEAKER_02: The server hits 90% disk space.
• 46:47
  SPEAKER_02: That sends an alert to Nagios.
• 46:48
  SPEAKER_02: Nagios triggers a Chef run.
• 46:49
  SPEAKER_02: Chef makes The Disc Bigger.
• 46:51
  SPEAKER_02: Bigger.
• 46:51
  SPEAKER_02: Feedback loops, right?
• 46:53
  SPEAKER_02: This has been around for a while.
• 46:54
  SPEAKER_02: And in 2018,
• 46:56
  SPEAKER_02: I want to say this guy,
• 46:57
  SPEAKER_02: Nick Chalain,
• 46:57
  SPEAKER_02: who was he was like the CTO or chief software officer of the Air Force.
• 47:02
  SPEAKER_02: He wrote this 100 page essay of, hey, the DOD needs a support.
• 47:07
  SPEAKER_02: The Department of Defense Yeah, the Department of Defense.
• 47:10
  SPEAKER_02: And the Air Force,
• 47:11
  SPEAKER_02: and he called it DevSecOps Factory.
• 47:13
  SPEAKER_02: And he said we need all the things that all of the good enterprises are using.
• 47:17
  SPEAKER_02: We need Jenkins, we need code quality scanning, we We need security scanning.
• 47:21
  SPEAKER_02: We need CICD.
• 47:22
  SPEAKER_02: We're shipping once every three months or once a year.
• 47:26
  SPEAKER_02: We need to be able to ship every day like all these other companies.
• 47:29
  SPEAKER_02: And the way we do that is we actually embrace all these automations and technologies
• 47:34
  SPEAKER_02: so that 90% of the issues are caught by automations instead
• 47:38
  SPEAKER_02: of people actually manually checking it or manually reading the code
• 47:41
  SPEAKER_02: or manually integrating modules together.
• 47:43
  SPEAKER_01: Wow.
• 47:44
  SPEAKER_01: Talk about forwards.
• 47:45
  SPEAKER_01: Forward thinking in the government.
• 47:47
  SPEAKER_02: I know.
• 47:48
  SPEAKER_02: No, as I was surprised, like, oh, nice.
• 47:49
  SPEAKER_02: Like, this is, I mean,
• 47:50
  SPEAKER_02: and that was part of it is like, hey, look,
• 47:52
  SPEAKER_02: we're falling behind in like,
• 47:53
  SPEAKER_02: you know, I don't know exactly all the reason, but I I imagine also about like a.
• 47:56
  SPEAKER_02: attracting really good talent is like, hey, look,
• 47:58
  SPEAKER_02: if we have like the modern software stack and we're building things fast and we care
• 48:02
  SPEAKER_02: about efficiency and we care about people using people's time well,
• 48:05
  SPEAKER_02: we care about them spending time on the hard parts of the job,
• 48:07
  SPEAKER_02: not Manually looking for SQL injections, like you could automate that.
• 48:11
  SPEAKER_01: So this was Software Factory's pre-AI.
• 48:13
  SPEAKER_01: Pre-AI.
• 48:14
  SPEAKER_01: Now, So I've heard the term.
• 48:15
  SPEAKER_01: Term a lot more because of AI.
• 48:17
  SPEAKER_01: Yeah.
• 48:17
  SPEAKER_01: Is it the same?
• 48:18
  SPEAKER_01: Is it different?
• 48:19
  SPEAKER_02: So this is really hard to say without a drawing, but I'll try to draw it out.
• 48:23
  SPEAKER_02: At the core of a software factory, you have like a source of work.
• 48:27
  SPEAKER_02: You can imagine a linear, a JIRA, The state's source of truth.
• 48:30
  SPEAKER_02: Truth, object, whether it's a spreadsheet or.
• 48:32
  SPEAKER_02: Or whatever.
• 48:32
  SPEAKER_02: You have like, What stages is
• 48:34
  SPEAKER_02: The work in?
• 48:34
  SPEAKER_02: Yep.
• 48:35
  SPEAKER_02: And pre -AI you you would You know, you would maybe do some architecture.
• 48:38
  SPEAKER_02: Architecture review planning,
• 48:39
  SPEAKER_02: you would maybe do some sprint planning,
• 48:40
  SPEAKER_02: and then people would take tickets off the queue, and they would go build them.
• 48:43
  SPEAKER_02: And then you would make a pull request,
• 48:45
  SPEAKER_02: and people would review it,
• 48:46
  SPEAKER_02: and you would run CI checks, And then you would send it to practice.
• 48:48
  SPEAKER_02: Prod.
• 48:49
  SPEAKER_02: And then it would make contact with your users and your users would complain
• 48:51
  SPEAKER_02: about stuff and that would go to your support team and back into your work tracker.
• 48:54
  SPEAKER_02: And it would crash and you would have issues and that would go
• 48:57
  SPEAKER_02: into your monitoring stack and that would go into your tracker.
• 48:59
  SPEAKER_02: And that was your loop.
• 49:01
  SPEAKER_02: And then people would take stuff off the tracker based on priorities,
• 49:04
  SPEAKER_02: product managers, engineering managers, engineers, prioritizing work.
• 49:07
  SPEAKER_02: And then we go and do that.
• 49:08
  SPEAKER_02: And the first change.
• 49:09
  SPEAKER_01: It's like this long,
• 49:11
  SPEAKER_01: lots of phase.
• 49:12
  SPEAKER_01: And this is also why when a developer shifts a bug,
• 49:15
  SPEAKER_01: but by the time it comes back to you,
• 49:16
  SPEAKER_01: it might be two or three months or even longer.
• 49:19
  SPEAKER_01: And by the day I get fixed, it might have been a year or two.
• 49:22
  SPEAKER_01: This is why when you're using a piece of software,
• 49:24
  SPEAKER_01: it's that annoying bug and you talk with customer support,
• 49:26
  SPEAKER_01: but it's just very long latencies at each part of the factory, if you will.
• 49:33
  SPEAKER_02: Yeah. And the step where someone pulls a work item off the queue and starts working
• 49:37
  SPEAKER_02: on it is You know, a couple of years.
• 49:59
  SPEAKER_02: And now with an agentic factory,
• 50:01
  SPEAKER_02: what you do is you take out that person building the thing and you replace it with
• 50:04
  SPEAKER_02: an agent building the thing.
• 50:05
  SPEAKER_02: And so you have orchestration to trigger things.
• 50:07
  SPEAKER_02: You have a sandbox,
• 50:08
  SPEAKER_02: you have an LLM,
• 50:09
  SPEAKER_02: you have an inner harness,
• 50:10
  SPEAKER_02: you have an outer harness,
• 50:11
  SPEAKER_02: which is like the dev environment you build for the agent.
• 50:14
  SPEAKER_02: And maybe you give it a browser, you give it a video recorder.
• 50:16
  SPEAKER_02: If you use like things like cursor background agents,
• 50:18
  SPEAKER_02: they've kind of built this outer harness around the inner harness that is
• 50:21
  SPEAKER_02: the coding agent.
• 50:22
  SPEAKER_02: And then you make PRs with that.
• 50:24
  SPEAKER_02: Problem there is that like,
• 50:25
  SPEAKER_02: okay, now it takes 10 minutes to do a build instead of two hours or two days.
• 50:29
  SPEAKER_02: And so now the bottleneck is code review.
• 50:31
  SPEAKER_02: So, okay,
• 50:32
  SPEAKER_02: let's throw a bunch of AI agents at code review and let's do agentic testing so
• 50:35
  SPEAKER_02: that like we can basically catch a lot of the easy stuff and humans are only focused
• 50:39
  SPEAKER_02: on the most like important critical core parts of the code base.
• 50:44
  SPEAKER_02: And then the next level up of your agentic factory as you do the top is like,
• 50:47
  SPEAKER_02: okay, and then it gets deployed.
• 50:48
  SPEAKER_02: It goes to prod.
• 50:49
  SPEAKER_02: And a user complains, you just hook your support queue right up to the agent.
• 50:53
  SPEAKER_02: Someone complains about something, agent tries to fix it.
• 50:55
  SPEAKER_02: And instead of looking at a ticket and then saying, okay, you just close that loop.
• 50:59
  SPEAKER_02: And instead, every time something goes wrong, you just get a PR.
• 51:02
  SPEAKER_02: And then every time something crashes in Sentry or Datadog or whatever...
• 51:05
  SPEAKER_02: It goes into the tracker, it gets picked up by an agent, and you get a PR.
• 51:09
  SPEAKER_02: This is the ramp inspect thing.
• 51:11
  SPEAKER_02: The only difference is then you have so much code to review and people say,
• 51:14
  SPEAKER_02: well, let's try turning the lights off.
• 51:16
  SPEAKER_02: Let's just take all the human testing and review steps out and we'll say, okay, cool.
• 51:20
  SPEAKER_02: If users complain, then it's broken.
• 51:22
  SPEAKER_02: And if users don't complain, then it's working.
• 51:24
  SPEAKER_02: And we're not going to read the code.
• 51:25
  SPEAKER_02: We're going to treat the whole system as a black box.
• 51:28
  SPEAKER_01: So you said you tried this out When it was like Opus 4.
• 51:30
  SPEAKER_01: 4.1.
• 51:31
  SPEAKER_01: You built the software factory.
• 51:32
  SPEAKER_01: It was running beautifully until it just blew up on your faces.
• 51:35
  SPEAKER_01: How do you think of this model?
• 51:36
  SPEAKER_01: Because I can see how I do.
• 51:38
  SPEAKER_01: Ideal world where it works, but clearly we're not in an ideal world.
• 51:41
  SPEAKER_01: Where do you think we are?
• 51:44
  SPEAKER_01: And could some of this actually work at some point?
• 51:48
  SPEAKER_01: What progress are you seeing right now?
• 51:50
  SPEAKER_01: And And what is the today?
• 51:51
  SPEAKER_01: Today the situation?
• 51:53
  SPEAKER_01: How much of this do you believe we can automate or should we automate?
• 51:56
  SPEAKER_02: Yep.
• 51:57
  SPEAKER_02: So if you know me and you follow me on Twitter, I'll see you next time.
• 51:58
  SPEAKER_02: Follow my stuff, you know I stand for three things.
• 52:00
  SPEAKER_02: Number one is like cutting through the hype and the jargon and going trying things
• 52:04
  SPEAKER_02: and talking to people who are using things and figuring out which parts
• 52:06
  SPEAKER_02: of this actually work and are valuable.
• 52:08
  SPEAKER_02: Number two, we talked about words.
• 52:10
  SPEAKER_02: I try to find and protect useful bits of language because I think it helps us all move forward.
• 52:15
  SPEAKER_02: And when you take a useful word like like agents or you take
• 52:18
  SPEAKER_02: a useful world like software factory factory and then you semantically diffuse it,
• 52:22
  SPEAKER_02: this is another Martin Fowler word.
• 52:23
  SPEAKER_02: You make it mean everybody likes the word and it all becomes hype
• 52:26
  SPEAKER_02: and everyone starts... Agents means nothing anymore.
• 52:29
  SPEAKER_02: Agents could be a chatbot,
• 52:31
  SPEAKER_02: it could be A Slack bot,
• 52:32
  SPEAKER_02: it Could Be A Coding Agent, It Could Be Tools In A Loop, whatever it is.
• 52:35
  SPEAKER_02: So I like to protect important useful words and help us all elevate
• 52:39
  SPEAKER_02: the conversation out of that hype and jargon.
• 52:42
  SPEAKER_02: And then I care a lot about going one level down beneath where I'm generally working.
• 52:48
  SPEAKER_02: I think there's always, airing it's like I was
• 52:51
  SPEAKER_02: rarely actually going and like building LLMs or understanding
• 52:54
  SPEAKER_02: or training LLMs,
• 52:55
  SPEAKER_02: but knowing how they're trained and how Transformers works informs how you build
• 52:59
  SPEAKER_02: at one layer up.
• 53:01
  SPEAKER_02: And for the software factory,
• 53:02
  SPEAKER_02: my version of that is I spent the last couple weeks going really deep on,
• 53:06
  SPEAKER_02: uh, reinforcement.
• 53:06
  SPEAKER_02: Reinforcement learning with verifiable rewards, RLVR,
• 53:10
  SPEAKER_02: which is like this very productionized, like,
• 53:13
  SPEAKER_02: it's not like RLH,
• 53:15
  SPEAKER_02: RLHF is still like fairly academic and pure,
• 53:18
  SPEAKER_02: RLVR is this like It's a machine in these labs of how we train these models.
• 53:22
  SPEAKER_02: And I'm studying like the benchmarks for coding agents and the techniques
• 53:25
  SPEAKER_02: for training them and how we like give it a small problem,
• 53:28
  SPEAKER_02: have it solve it,
• 53:29
  SPEAKER_02: delete the test changes it made, revert them, apply a test patch, see if it passed.
• 53:33
  SPEAKER_02: And then even the Frontier this year,
• 53:35
  SPEAKER_02: we have like,
• 53:35
  SPEAKER_02: we can get into this later,
• 53:36
  SPEAKER_02: but like Frontier Code and SWE Marathon,
• 53:38
  SPEAKER_02: these new benchmarks that are supposed to be like better
• 53:41
  SPEAKER_02: at evaluating models ability to maintain a code base over time
• 53:45
  SPEAKER_02: and write maintainable code.
• 53:47
  SPEAKER_02: And they are better, but I don't think they're sufficient.
• 53:49
  SPEAKER_02: But it's basically this idea that like the only thing
• 53:52
  SPEAKER_02: that made cloud code good was reinforcement learning.
• 53:55
  SPEAKER_02: And the dimension along which it got good was like we made a model,
• 53:59
  SPEAKER_02: we trained the model and the harness together.
• 54:01
  SPEAKER_02: And so the model got really good at calling the specific tools in that harness,
• 54:06
  SPEAKER_02: really good at reading files,
• 54:07
  SPEAKER_02: writing files, searching for files, all this stuff through doing these problems.
• 54:11
  SPEAKER_02: And that was what made it feel so much better than all the other CLI coding agents
• 54:14
  SPEAKER_02: that came before it.
• 54:16
  SPEAKER_02: And so people are like, OK, that was so much better.
• 54:17
  SPEAKER_02: And they're just going to keep getting better.
• 54:18
  SPEAKER_02: But it's like it got really good in one dimension and the dimension
• 54:22
  SPEAKER_02: that they're not getting better in because it's hard, expensive.
• 54:26
  SPEAKER_02: Maybe we need to get a lot more creative with how we design these verifiers and benches.
• 54:30
  SPEAKER_02: Benchmarks is in how do I make code that in three months is going to improve
• 54:37
  SPEAKER_02: the productivity of humans and agents,
• 54:39
  SPEAKER_02: mostly agents,
• 54:39
  SPEAKER_02: but humans and agents in the code base instead of making it worse over time.
• 54:43
  SPEAKER_01: And so you think that part is just missing?
• 54:44
  SPEAKER_01: We haven't seen too much improvement?
• 54:47
  SPEAKER_02: I haven't seen...
• 54:48
  SPEAKER_02: Obviously,
• 54:48
  SPEAKER_02: no one knows what the labs are doing internally because it's all very secret.
• 54:51
  SPEAKER_02: But I think if we...
• 54:52
  SPEAKER_02: Looking at where the benchmarks tend to reflect where the labs are, right?
• 54:58
  SPEAKER_02: There is no benchmark that can And convey to me.
• 55:01
  SPEAKER_02: Me...
• 55:02
  SPEAKER_02: Did this model write code that is going to make my code base better or worse?
• 55:05
  SPEAKER_02: The best we have is I think frontier code from the cognition team is really interesting.
• 55:09
  SPEAKER_02: They have like, did the test pass and then they'd have like two layers of model review.
• 55:14
  SPEAKER_02: So they have a judge model that checks, okay,
• 55:16
  SPEAKER_02: is the patch the model made similar to the patch that is like the golden answer set?
• 55:21
  SPEAKER_02: So even if the model didn't write the exact code that the benchmark was expecting...
• 55:25
  SPEAKER_02: Did, was it functionally?
• 55:26
  SPEAKER_02: Functionally equivalent?
• 55:27
  SPEAKER_02: And the next one is like a code quality review from another judge model.
• 55:31
  SPEAKER_02: And that's better, but it's not sufficient.
• 55:36
  SPEAKER_02: And this is why I also think agentic code review is like,
• 55:38
  SPEAKER_02: yes, it will catch things and it will raise your floor.
• 55:41
  SPEAKER_02: But I don't believe like the model writing the code is the same model reading the code.
• 55:44
  SPEAKER_02: And if you ask a model, hey, is this code good?
• 55:46
  SPEAKER_02: It's going to be like, oh, yeah, it's great.
• 55:47
  SPEAKER_02: Comprehensive.
• 55:48
  SPEAKER_02: It's got unit tests.
• 55:49
  SPEAKER_02: You've tried this.
• 55:50
  SPEAKER_02: I'm sure.
• 55:51
  SPEAKER_02: And you say, OK,
• 55:52
  SPEAKER_02: review this PR that my coworker wrote and tell me everything that's wrong with it.
• 55:55
  SPEAKER_02: And I was like, oh, it has this problem and this problem and this is sycophantic.
• 55:58
  SPEAKER_02: And they want to tell you what you want to hear.
• 56:00
  SPEAKER_02: And so like it's really hard for me to trust a model to evaluate.
• 56:03
  SPEAKER_02: The quality of code that's written.
• 56:06
  SPEAKER_02: And so I have some ideas on like, okay,
• 56:08
  SPEAKER_02: can you build a benchmark where the model builds 20 features in a row and maintains
• 56:12
  SPEAKER_02: the code base the whole time?
• 56:13
  SPEAKER_02: And it doesn't know what features are coming.
• 56:15
  SPEAKER_02: You treat it like a real product team where you don't know what you're going
• 56:17
  SPEAKER_02: to build next week until you get there and you find out what's most important.
• 56:21
  SPEAKER_02: And then can we try to evaluate like,
• 56:24
  SPEAKER_02: can we build a problem like that that's hard enough that most frontier models fail
• 56:28
  SPEAKER_02: by issue six or seven?
• 56:29
  SPEAKER_01: Is it fair to say that,
• 56:31
  SPEAKER_01: you know,
• 56:31
  SPEAKER_01: like we've had the software factory, like before AI, it was just like lots of loop.
• 56:35
  SPEAKER_01: It was like the PM I'm giving a ticket to that.
• 56:38
  SPEAKER_01: The dev, the dev building and deploying to production, Customers use it.
• 56:43
  SPEAKER_01: Using it,
• 56:43
  SPEAKER_01: customer support getting tickets and then,
• 56:45
  SPEAKER_01: you know, Creating PM triaging and it kind of goes around in this loop.
• 56:50
  SPEAKER_01: Is it fair to say that the software factory of how a company,
• 56:54
  SPEAKER_01: a team builds and maintains software that is changing
• 56:57
  SPEAKER_01: because now everyone's replacing some parts of it,
• 57:00
  SPEAKER_01: you know,
• 57:00
  SPEAKER_01: maybe the least and Advanced teams will just be,
• 57:03
  SPEAKER_01: devs are starting to use cloud code or codex to write The faster they're not.
• 57:07
  SPEAKER_01: Not spending as much time on there.
• 57:08
  SPEAKER_01: Some others are also having the deployment, the feedback.
• 57:12
  SPEAKER_01: Some actually have the agents already One -shotting.
• 57:15
  SPEAKER_01: One-shotting bugs.
• 57:15
  SPEAKER_01: So like, is it fair to say that?
• 57:17
  SPEAKER_01: That the software is just changing everywhere,
• 57:19
  SPEAKER_01: maybe at different speeds,
• 57:20
  SPEAKER_01: but everyone,
• 57:21
  SPEAKER_01: I think every team who is building production software,
• 57:25
  SPEAKER_01: they're frantically experimenting, Trying on everything.
• 57:28
  SPEAKER_01: Everyone's at a different pace.
• 57:29
  SPEAKER_01: You'll have the AI native startups where most of this will have agents in them,
• 57:33
  SPEAKER_01: and you'll have the laggards who are Or more cautious.
• 57:36
  SPEAKER_01: Cautious ones.
• 57:37
  SPEAKER_01: They have agents in a few places, but not in the others.
• 57:39
  SPEAKER_02: Well, and I think that's the key is like,
• 57:41
  SPEAKER_02: if you want to do loops engineering,
• 57:42
  SPEAKER_02: you should build one loop at a time and you should keep them small and contained.
• 57:46
  SPEAKER_02: Basically, I think everything except stop reading the code is really good advice.
• 57:51
  SPEAKER_02: Take support tickets and turn them into tickets in your system and And then maybe turn those.
• 57:54
  SPEAKER_02: Those into PRs.
• 57:56
  SPEAKER_02: Great.
• 57:56
  SPEAKER_02: The advice that I have and like what we kind of like are chasing
• 57:59
  SPEAKER_02: at human layer is like, how can I add another checkpoint in that factory?
• 58:04
  SPEAKER_02: So instead of having one human viewpoint where you're reviewing PRs
• 58:07
  SPEAKER_02: and sometimes there are a hundred lines and sometimes there are a thousand lines,
• 58:11
  SPEAKER_02: but it's quite a lot of effort for,
• 58:13
  SPEAKER_02: especially if it's bad,
• 58:14
  SPEAKER_02: especially if it needs a rework,
• 58:15
  SPEAKER_02: it's quite a lot of effort for a human to be like, Okay, this is wrong.
• 58:18
  SPEAKER_02: Go change it in this way.
• 58:19
  SPEAKER_02: And then you loop back to the agent and then you come with another one
• 58:21
  SPEAKER_02: and like doing a lot of loops on there.
• 58:23
  SPEAKER_02: Once the direction has been committed to, it's really hard to steer off.
• 58:27
  SPEAKER_02: Like you're better off just kind of restarting from scratch.
• 58:30
  SPEAKER_02: How do you build like controls and mechanisms around that?
• 58:32
  SPEAKER_02: And then my take is like, If you do a little bit of human agent planning And like this.
• 58:37
  SPEAKER_02: And like discussion.
• 58:38
  SPEAKER_02: Discussion before you hand it to the implementer, Whether it's, I mean...
• 58:42
  SPEAKER_02: planning and specs.
• 58:42
  SPEAKER_02: Specs, whatever you want to call it.
• 58:43
  SPEAKER_02: Again,
• 58:43
  SPEAKER_02: this is spec-driven development is another word that has become very muddled as far
• 58:48
  SPEAKER_02: as what it means.
• 58:49
  SPEAKER_02: But basically, how can we spend an hour...
• 58:52
  SPEAKER_02: Before we start building so that the PR when we read it only takes 20 minutes
• 58:57
  SPEAKER_02: because the code is perfect instead of not touching
• 59:00
  SPEAKER_02: and just literally saying every user reported issue becomes a PR through the loop.
• 59:04
  SPEAKER_02: And then we read that PR and it takes six hours because it's back and forth.
• 59:07
  SPEAKER_02: We have to make changes and things.
• 59:08
  SPEAKER_02: It's all I'm all about like, let's find leverage.
• 59:11
  SPEAKER_02: And so you basically you have three options.
• 59:13
  SPEAKER_02: In the software factory world,
• 59:14
  SPEAKER_02: if you're going to go all in on agentic software factories,
• 59:17
  SPEAKER_02: you can turn the lights off and just let everything flow and pray
• 59:21
  SPEAKER_02: that you don't create too much slop and pray that the next generation
• 59:24
  SPEAKER_02: of models comes fast enough before you create a giant pile of ash.
• 59:29
  SPEAKER_02: You can slow way down and read every PR and read every line of code.
• 59:33
  SPEAKER_02: And then you're only going to really get modest benefits from AI because
• 59:36
  SPEAKER_02: that becomes, I think you should expect me.
• 59:38
  SPEAKER_02: Maybe 30 to 50% lift in productivity is kind of what I see when we go into teams.
• 59:44
  SPEAKER_02: Or you can find the right leverage points where humans can actually,
• 59:48
  SPEAKER_02: an hour spent over here in planning Can save you f***ing.
• 59:52
  SPEAKER_02: Four hours in implementation in terms of fixing and going back and getting the design right.
• 59:57
  SPEAKER_02: And that's what I call like seeking leverage.
• 59:59
  SPEAKER_02: You can find the right leverage points for the agents to guide the work.
• 1:00:02
  SPEAKER_02: Then you can actually move like two to three times faster while maintaining
• 1:00:07
  SPEAKER_02: a like 99% 11 % like accuracy.
• 1:00:09
  SPEAKER_02: Accuracy to like if the humans were carefully writing this code by hand, how would it come out?
• 1:00:14
  SPEAKER_01: Now jumping a little bit back to ideas, I will come back to this.
• 1:00:18
  SPEAKER_01: This was earlier, maybe it was last year, but you had the research plan implement.
• 1:00:22
  SPEAKER_01: Can we talk about the original research plan implement framework and
• 1:00:25
  SPEAKER_01: then also what you've learned about this approach, what you got wrong about it?
• 1:00:29
  SPEAKER_02: Yeah, sure.
• 1:00:29
  SPEAKER_02: Yeah.
• 1:00:30
  SPEAKER_02: So, I mean, the first time we talked about RPI was in August of 2025.
• 1:00:35
  SPEAKER_02: And it was basically like the research was this thing of like, hey,
• 1:00:37
  SPEAKER_02: before you go build anything,
• 1:00:39
  SPEAKER_02: go read lots and lots of code, use a bunch of Sub -agents in parallel.
• 1:00:43
  SPEAKER_02: Parallel,
• 1:00:43
  SPEAKER_02: understand all It was this technique that worked really well for hard problems
• 1:00:47
  SPEAKER_02: and complex code bases.
• 1:00:48
  SPEAKER_02: You just ask Claude to do a thing that would read three files and make a change.
• 1:00:53
  SPEAKER_02: It would have no context.
• 1:00:54
  SPEAKER_02: So you start the research.
• 1:00:55
  SPEAKER_02: You don't even tell it what you're working on.
• 1:00:57
  SPEAKER_02: You just tell it, hey,
• 1:00:59
  SPEAKER_02: can you tell me how this system works and this system and how they connect together?
• 1:01:02
  SPEAKER_02: And then you get a markdown doc out.
• 1:01:04
  SPEAKER_02: And this is the context engineering part is like that would take 100,000 tokens
• 1:01:07
  SPEAKER_02: of context, but you would get a 10K token doc out of it that summarized it.
• 1:01:11
  SPEAKER_02: Then you would start a new context window and you would do planning.
• 1:01:14
  SPEAKER_02: And the planning would be,
• 1:01:15
  SPEAKER_02: and I actually realized like the plans
• 1:01:16
  SPEAKER_02: that we were building last summer were actually terrible,
• 1:01:19
  SPEAKER_02: but it would basically be this long,
• 1:01:20
  SPEAKER_02: you would say, okay, now here's what we're building.
• 1:01:22
  SPEAKER_02: Here's the research doc, build a plan to implement it.
• 1:01:25
  SPEAKER_02: And in retrospect,
• 1:01:27
  SPEAKER_02: now that we see like everyone is obsessed with how do I get agents to work
• 1:01:30
  SPEAKER_02: for longer?
• 1:01:31
  SPEAKER_02: I think the reason why in like May, June, July,
• 1:01:34
  SPEAKER_02: August of 2021,
• 1:01:34
  SPEAKER_02: That a lot of people became really interested in planning was it was
• 1:01:38
  SPEAKER_02: a very powerful lever to get agents to work for longer.
• 1:01:44
  SPEAKER_02: If you said build me a B2B SaaS for burrito delivery, you'd get like a home page.
• 1:01:49
  SPEAKER_02: And that's it.
• 1:01:50
  SPEAKER_02: But if you said, build me a plan, it would build out this big plan.
• 1:01:53
  SPEAKER_02: And then in the next context window, you'd say, hey, here's the plan.
• 1:01:55
  SPEAKER_02: Here's all the changes we're going to make.
• 1:01:57
  SPEAKER_02: Go implement.
• 1:01:57
  SPEAKER_02: It would actually keep going until the plan was done.
• 1:01:59
  SPEAKER_02: So the plan was a really good way to anchor an agent and remind it that like,
• 1:02:03
  SPEAKER_02: hey, you're not done until this is all finished.
• 1:02:05
  SPEAKER_02: So that was the original RPI.
• 1:02:06
  SPEAKER_02: And the plan doc, what was bad about it is it didn't give you leverage.
• 1:02:09
  SPEAKER_02: The plan was every single line of code that was going to change,
• 1:02:12
  SPEAKER_02: like in diff blocks and like all the new stuff to write.
• 1:02:16
  SPEAKER_02: And so like people would review these plans.
• 1:02:18
  SPEAKER_02: We recommended this.
• 1:02:18
  SPEAKER_02: We told people to read the plans.
• 1:02:20
  SPEAKER_02: We read all our plans.
• 1:02:21
  SPEAKER_02: And then eventually I found myself like, I just kind of skimmed the plans.
• 1:02:24
  SPEAKER_02: And so you're not really using it as a way to re-steer the agent.
• 1:02:27
  SPEAKER_02: It's just kind of there.
• 1:02:29
  SPEAKER_02: And then you go write the code and there's a crap.
• 1:02:31
  SPEAKER_02: Some people would review the plans and the code.
• 1:02:33
  SPEAKER_02: And it's like, okay, well, the plan took you 20 minutes to read.
• 1:02:36
  SPEAKER_02: And then the pull request takes you 20 minutes to read.
• 1:02:38
  SPEAKER_02: And they're different.
• 1:02:39
  SPEAKER_02: And so you actually doubled the amount of time you're spending reading code instead
• 1:02:42
  SPEAKER_02: of like doing less of it.
• 1:02:43
  SPEAKER_02: You've been anti-leverage.
• 1:02:44
  SPEAKER_01: And hang on, was spec different development not related to this?
• 1:02:49
  SPEAKER_01: The one that Amazon Kiro,
• 1:02:50
  SPEAKER_01: for example,
• 1:02:51
  SPEAKER_01: and GitHub Workflows again a year ago did, which was Because it also, at first...
• 1:02:55
  SPEAKER_01: first generated a plan and it had the human review it and then it started to
• 1:02:59
  SPEAKER_01: and you could edit it as well.
• 1:03:00
  SPEAKER_01: And then it went off and implement this part and it looked beautifully on the surface.
• 1:03:04
  SPEAKER_01: It should have worked great.
• 1:03:05
  SPEAKER_01: But it's tossed into the garbage outside of some maintenance projects I think. Yeah.
• 1:03:10
  SPEAKER_01: It just didn't work like all the feedback I got people just stopped using it
• 1:03:14
  SPEAKER_01: because it just didnít really work that well.
• 1:03:15
  SPEAKER_01: It just rhymes to the RPI framework a little bit,
• 1:03:18
  SPEAKER_01: the original one right?
• 1:03:19
  SPEAKER_02: Well so our thing too,
• 1:03:20
  SPEAKER_02: like the biggest difference between RPI and spec-driven development,
• 1:03:23
  SPEAKER_02: and some people refer to RPI as spec-driven dev because for some people, SDD,
• 1:03:28
  SPEAKER_02: all it means is I use a bunch of markdown files while I'm coding and forget what's
• 1:03:33
  SPEAKER_02: in them.
• 1:03:33
  SPEAKER_02: I just spec-driven dev, those are my specs and I'm using them to drive development.
• 1:03:36
  SPEAKER_02: There was this open AI researcher who talked about spec-driven dev and like, hey,
• 1:03:40
  SPEAKER_02: stop reading the code,
• 1:03:41
  SPEAKER_02: just write the specs and treat like the coding part is compiling specs into code.
• 1:03:46
  SPEAKER_02: And that part.
• 1:03:48
  SPEAKER_02: Part never really materialized, maybe with GPT-7, you know, Um, but the challenge I'm on a.
• 1:03:53
  SPEAKER_02: a GitHub issue in SpecKit That has been over.
• 1:03:55
  SPEAKER_02: Been open for a year,
• 1:03:56
  SPEAKER_02: and every couple weeks I get it,
• 1:03:58
  SPEAKER_02: there's a new email on the thread of people complaining about this problem,
• 1:04:00
  SPEAKER_02: like, okay, I edit my specs to And then I edit the code And the code drift.
• 1:04:04
  SPEAKER_02: Drifts and the specs.
• 1:04:05
  SPEAKER_02: How do I keep the specs up to date as the code is changing?
• 1:04:07
  SPEAKER_02: And it's basically like, you now have two sources of truth and it's, it stops being useful.
• 1:04:12
  SPEAKER_02: And so like,
• 1:04:12
  SPEAKER_02: that's why when RPI,
• 1:04:14
  SPEAKER_02: the idea of the docs is they're all for a while,
• 1:04:15
  SPEAKER_02: we kept them around, but after two months.
• 1:04:17
  SPEAKER_02: Oh, these are actually like tactical execution docs. I do the research and do the plan.
• 1:04:22
  SPEAKER_02: I do The Implementation.
• 1:04:23
  SPEAKER_02: I throw the docs out.
• 1:04:24
  SPEAKER_02: And the next time I need research,
• 1:04:26
  SPEAKER_02: I just do it from scratch because tokens are cheap and my time is expensive.
• 1:04:30
  SPEAKER_02: And the amount of time I might waste if I reuse a research that is no longer in sync
• 1:04:35
  SPEAKER_02: with this real state of the code base. So we just create it live every time.
• 1:04:38
  SPEAKER_02: This is why context engineering still matters.
• 1:04:41
  SPEAKER_02: Creating artifacts that compress the state of the code base and compress the intent
• 1:04:45
  SPEAKER_02: of the builder into small things that can be reused in the future for the scope of
• 1:04:49
  SPEAKER_02: a task is a very powerful tactical approach.
• 1:04:54
  SPEAKER_02: But it's not a thing.
• 1:04:55
  SPEAKER_02: I have very few opinions on what sorts of docs
• 1:04:59
  SPEAKER_02: that you should leave lying around your code base that are evergreen.
• 1:05:02
  SPEAKER_02: I've seen people try to maintain parity between documentation or specs and the code itself.
• 1:05:08
  SPEAKER_02: And I don't think anyone actually found it very useful.
• 1:05:11
  SPEAKER_02: You can do it and it works, but it's the ratio of the effort it takes to keep them up to date.
• 1:05:15
  SPEAKER_02: And trivially,
• 1:05:17
  SPEAKER_02: you could do this with AI probably,
• 1:05:18
  SPEAKER_02: but I've never known anyone who was like,
• 1:05:20
  SPEAKER_02: yeah, this is great and we're glad we have it.
• 1:05:21
  SPEAKER_02: You could do it and it might help,
• 1:05:23
  SPEAKER_02: but I don't think anyone found it useful enough to maintain a system to keep
• 1:05:26
  SPEAKER_02: the specs and the code in sync
• 1:05:28
  SPEAKER_01: versus just using the code as the source
• 1:05:29
  SPEAKER_01: of truth always.
• 1:05:31
  SPEAKER_01: You mentioned something interesting,
• 1:05:32
  SPEAKER_01: which is with context engineering, you need to sometimes compact.
• 1:05:35
  SPEAKER_01: And You've previously He talks about intent.
• 1:05:38
  SPEAKER_01: Intentional compaction,
• 1:05:39
  SPEAKER_01: that when context is noisy,
• 1:05:41
  SPEAKER_01: deliberately compress the useful part into a clear markdown artifact,
• 1:05:44
  SPEAKER_01: verify it, and then start a fresh conversation.
• 1:05:47
  SPEAKER_01: Can we talk about this kind of compaction and why it's important?
• 1:05:51
  SPEAKER_01: And it sounds like it's going to be a building block where it already is
• 1:05:54
  SPEAKER_01: for context engineering, right?
• 1:05:56
  SPEAKER_01: Yeah.
• 1:05:56
  SPEAKER_02: Yeah, frequent intentional compaction is It's the building block.
• 1:05:59
  SPEAKER_02: Block.
• 1:06:00
  SPEAKER_02: It completely comes from context engineering.
• 1:06:03
  SPEAKER_02: Context engineering is like, how do we get the most out of today's models?
• 1:06:06
  SPEAKER_02: How do we change what we're putting into the model,
• 1:06:08
  SPEAKER_02: into the context window, into the agentic chat?
• 1:06:10
  SPEAKER_02: How do we control that in such a way that we get the best results possible?
• 1:06:14
  SPEAKER_02: Which means doing as much work as possible in the smart zone,
• 1:06:17
  SPEAKER_02: the first 100,000 tokens of the context window.
• 1:06:20
  SPEAKER_02: And And this intention.
• 1:06:21
  SPEAKER_02: This frequent intentional compaction is basically like, Okay, the research step.
• 1:06:25
  SPEAKER_02: We're going to go read a bunch of code and turn it into a doc.
• 1:06:27
  SPEAKER_02: That's our compaction.
• 1:06:28
  SPEAKER_02: We take that forward in the next session.
• 1:06:30
  SPEAKER_02: We're going to read the ticket and the intent and turn that into a design document
• 1:06:35
  SPEAKER_02: that we call it.
• 1:06:35
  SPEAKER_02: It's like, okay, here's the high level spec of what we want to do.
• 1:06:38
  SPEAKER_02: here's the high level like current state desired end state.
• 1:06:40
  SPEAKER_02: And then a bunch of design questions that model has kind of like
• 1:06:43
  SPEAKER_02: a very thorough, maybe even over-engineered like plan mode.
• 1:06:47
  SPEAKER_02: And then you take the research and the design and you do a new session,
• 1:06:51
  SPEAKER_02: new context when you're like, cool.
• 1:06:52
  SPEAKER_02: You've compressed the intent and you've compressed the state of the code base so
• 1:06:55
  SPEAKER_02: that you can then do your planning of like,
• 1:06:57
  SPEAKER_02: okay, we know what the end state looks like.
• 1:06:59
  SPEAKER_02: We know where we're going.
• 1:07:01
  SPEAKER_02: Now let's go.
• 1:07:02
  SPEAKER_02: Let's break down how we're going to get there.
• 1:07:03
  SPEAKER_02: All of these different steps of the process exist because...
• 1:07:07
  SPEAKER_02: models have short shortcomings in each of these phases.
• 1:07:09
  SPEAKER_02: So the research is pretty hands off.
• 1:07:11
  SPEAKER_02: I don't read the research docs.
• 1:07:12
  SPEAKER_02: It's just like go read a bunch of code and then like make a doc out of it.
• 1:07:15
  SPEAKER_02: Models are pretty damn good at that.
• 1:07:17
  SPEAKER_02: If you ask it to find a bug and have opinions about the code base, that's different.
• 1:07:20
  SPEAKER_02: But if you just ask it, what is the intent and how this stuff fit together?
• 1:07:23
  SPEAKER_02: That's usually pretty straightforward.
• 1:07:24
  SPEAKER_02: But designing the end state of the software, the architecture and the program design.
• 1:07:30
  SPEAKER_02: Models are not great at.
• 1:07:31
  SPEAKER_01: They make a lot of like, they make decisions and sometimes they're right and sometimes
• 1:07:34
  SPEAKER_02: they're wrong.
• 1:07:34
  SPEAKER_02: so we want to have a human in the loop there.
• 1:07:36
  SPEAKER_02: And then the steps to get there,
• 1:07:37
  SPEAKER_02: we talked about this before,
• 1:07:39
  SPEAKER_02: but models love making what I call like horizontal plans.
• 1:07:42
  SPEAKER_02: If you ask a model,
• 1:07:43
  SPEAKER_02: like build a plan of steps to go build this app,
• 1:07:45
  SPEAKER_02: It's like cool,
• 1:07:45
  SPEAKER_02: we're going to do the database and then we're gonna do the services layer.
• 1:07:48
  SPEAKER_02: Then we're going to do the API and then we're going to do the front end.
• 1:07:50
  SPEAKER_02: I was like, well,
• 1:07:50
  SPEAKER_02: that actually kind of sucks because we're going to be on the other side
• 1:07:52
  SPEAKER_02: of 2000 lines of code.
• 1:07:54
  SPEAKER_02: And let's imagine this is an existing code base, right?
• 1:07:56
  SPEAKER_02: We're going to make changes to all these different parts of the system.
• 1:07:58
  SPEAKER_02: I can't test it till the end.
• 1:07:59
  SPEAKER_02: And so what I would do is like, OK, how would I have built this if I were building my hand?
• 1:08:03
  SPEAKER_02: Well, OK,
• 1:08:03
  SPEAKER_02: I would probably create a mock API endpoint with fake data and then I would go kind
• 1:08:07
  SPEAKER_02: of get the front end kind of how I want it to look.
• 1:08:10
  SPEAKER_02: And then I would actually go like build a services layer and actually wire the data through.
• 1:08:14
  SPEAKER_02: And then I would make a database migration and make my new table.
• 1:08:17
  SPEAKER_02: And then I would actually add a lot of business logic.
• 1:08:20
  SPEAKER_02: And then I would add a bunch of air.
• 1:08:21
  SPEAKER_02: Error handling.
• 1:08:22
  SPEAKER_02: And it's completely orthogonal to how models will write the database layer and all
• 1:08:25
  SPEAKER_02: the error handling without Like anyone's ever touched.
• 1:08:28
  SPEAKER_02: Touched or seen the code or whatever it is.
• 1:08:30
  SPEAKER_02: And so this is another place where we like to have humans involved
• 1:08:32
  SPEAKER_02: because humans have really good taste and judgment
• 1:08:34
  SPEAKER_02: and like I would rather read five separate little mini-diffs of like things that.
• 1:08:40
  SPEAKER_02: That I can manually verify and explore than read 2,000 lines of code and be like,
• 1:08:44
  SPEAKER_02: well, it's not working.
• 1:08:45
  SPEAKER_02: I don't know where.
• 1:08:46
  SPEAKER_02: You don't know where.
• 1:08:47
  SPEAKER_02: Because you wrote the code.
• 1:08:47
  SPEAKER_02: You were supposed to get it right.
• 1:08:49
  SPEAKER_02: We talk about compaction context engineers.
• 1:08:51
  SPEAKER_02: How can you stay in the smart zone of the context window?
• 1:08:53
  SPEAKER_02: Which is, again, the dumb zone.
• 1:08:55
  SPEAKER_02: I will say, disclaimer, it's really good training wheels if you don't have intuition
• 1:08:59
  SPEAKER_01: about this.
• 1:08:59
  SPEAKER_01: So let's just Let's define these things.
• 1:09:02
  SPEAKER_01: Things.
• 1:09:02
  SPEAKER_01: What is a smart zone and what is a dumb zone?
• 1:09:04
  SPEAKER_02: So it's a little bit blurrier than like I would like, I would.
• 1:09:08
  SPEAKER_02: Like it to be.
• 1:09:09
  SPEAKER_02: I think in November we talked and said, oh, that's about the first 40% of the context window.
• 1:09:13
  SPEAKER_02: But then we had million token context windows.
• 1:09:16
  SPEAKER_02: So then I changed it to like the first hundred thousand tokens.
• 1:09:19
  SPEAKER_02: If it's a really like 4.8, I usually will go up to like 200K.
• 1:09:22
  SPEAKER_02: But basically The thing that Jeff Huntley had.
• 1:09:24
  SPEAKER_02: Had and Ralph Wiggum was like the less context window you use,
• 1:09:27
  SPEAKER_01: the better outcomes you'll get.
• 1:09:28
  SPEAKER_01: And basically the smart zone meaning if you have context in that first part,
• 1:09:33
  SPEAKER_01: it should work a lot better.
• 1:09:35
  SPEAKER_01: And then like the dumb zone is like once you have stuff there, it's kind of forget about it.
• 1:09:39
  SPEAKER_01: Like it'll be confused.
• 1:09:40
  SPEAKER_01: It's not going to do much.
• 1:09:41
  SPEAKER_01: Like it'll degrade.
• 1:09:43
  SPEAKER_02: Yeah, and there are times,
• 1:09:44
  SPEAKER_02: and this is an intuition thing,
• 1:09:45
  SPEAKER_02: like I will often go up to three, four hundred K tokens.
• 1:09:48
  SPEAKER_02: Four is rare,
• 1:09:49
  SPEAKER_02: but I will go up to two fifty,
• 1:09:50
  SPEAKER_02: three hundred K tokens for certain types of work where my intuition tells me
• 1:09:54
  SPEAKER_02: that I can keep working without degrading the performance.
• 1:09:57
  SPEAKER_02: But if you don't have good LLM intuition,
• 1:09:59
  SPEAKER_02: like a hundred K for smaller models,
• 1:10:01
  SPEAKER_02: two hundred K for these like really beefy like Codex and Opus 4.8 models is usually
• 1:10:06
  SPEAKER_02: a good like training wheel guideline of like if you pass their Your quality
• 1:10:12
  SPEAKER_02: of
• 1:10:12
  SPEAKER_02: Results may be degrading.
• 1:10:13
  SPEAKER_02: Grading the biggest tell I see for this is often the uh models trying to get
• 1:10:18
  SPEAKER_02: the test to pass And your 200K token, well, let me try this.
• 1:10:21
  SPEAKER_02: Okay, let me try that.
• 1:10:22
  SPEAKER_02: And it's like trying a bunch of stuff and it's getting more and more extreme.
• 1:10:25
  SPEAKER_02: And it's like, oh, let me delete your .end file and try again.
• 1:10:28
  SPEAKER_02: Like this is where things get really, really weird.
• 1:10:30
  SPEAKER_02: And so it's like,
• 1:10:31
  SPEAKER_02: if you start to see certain types,
• 1:10:33
  SPEAKER_02: if I'm like, oh, we're at 300K tokens and I need to like fix the unit test.
• 1:10:36
  SPEAKER_02: I'm like, cool,
• 1:10:38
  SPEAKER_02: write everything we did to a file or even I'll just do like
• 1:10:40
  SPEAKER_02: a built-in compaction depending on the model.
• 1:10:42
  SPEAKER_02: And then I'm starting a new session at 30K or 50K tokens.
• 1:10:45
  SPEAKER_02: And I'm like, cool,
• 1:10:46
  SPEAKER_02: we're going to do a hard thing,
• 1:10:47
  SPEAKER_02: which is you're going to get this fricking test to pass.
• 1:10:49
  SPEAKER_02: And you're not going to be stupid about it.
• 1:10:50
  SPEAKER_01: by the way one thing that you said like about the
• 1:10:53
  SPEAKER_01: With the model being dumb.
• 1:10:54
  SPEAKER_01: Dumb is you said that if the model ever tells you you are absolutely right,
• 1:10:59
  SPEAKER_01: you should start over.
• 1:11:01
  SPEAKER_01: And we've all had that when it tells me like, oh, you know, you're absolutely right.
• 1:11:05
  SPEAKER_01: And I'm like, we just get annoyed.
• 1:11:06
  SPEAKER_01: But why should we start over?
• 1:11:08
  SPEAKER_01: What's happening there in your observations?
• 1:11:11
  SPEAKER_02: Yeah, that's great.
• 1:11:12
  SPEAKER_02: Yeah.
• 1:11:12
  SPEAKER_02: And the new you're absolutely right, I think, is you're right to push back on that, right?
• 1:11:16
  SPEAKER_02: Yes.
• 1:11:17
  SPEAKER_02: That's Opus, right?
• 1:11:18
  SPEAKER_02: Yeah, Opus is like, you didn't run the test, did you?
• 1:11:20
  SPEAKER_02: You're right to push back on that.
• 1:11:22
  SPEAKER_02: I totally did it.
• 1:11:22
  SPEAKER_02: But no, for me,
• 1:11:23
  SPEAKER_02: you're absolutely right was always what the model would respond if you were like,
• 1:11:26
  SPEAKER_02: That's totally wrong.
• 1:11:28
  SPEAKER_02: If you said something where you were angry or frustrated or just wanted to point out
• 1:11:32
  SPEAKER_02: that it's done something wrong, It would respond with your.
• 1:11:34
  SPEAKER_02: You're absolutely right.
• 1:11:36
  SPEAKER_02: And most of us have had the experience of,
• 1:11:38
  SPEAKER_02: it says that, and then it continues to do the wrong thing.
• 1:11:41
  SPEAKER_02: So it's like,
• 1:11:41
  SPEAKER_02: once it starts doing dumb things,
• 1:11:43
  SPEAKER_02: because there's There's four things in your content.
• 1:11:45
  SPEAKER_02: Context window that matter.
• 1:11:46
  SPEAKER_02: There's the size of it, how many tokens.
• 1:11:49
  SPEAKER_02: There's the quality of the information.
• 1:11:51
  SPEAKER_02: Is there any incorrect information?
• 1:11:52
  SPEAKER_02: If the model had some thinking trace where it decided the wrong thing was true,
• 1:11:56
  SPEAKER_02: Is there missing information?
• 1:11:57
  SPEAKER_02: Does this like have context missing that it should have?
• 1:12:00
  SPEAKER_02: And then there's the trajectory.
• 1:12:01
  SPEAKER_02: And the trajectory is very subtle, but you may have had sessions.
• 1:12:05
  SPEAKER_01: The trajectory meaning you're prompting?
• 1:12:07
  SPEAKER_02: The actual history of everything.
• 1:12:09
  SPEAKER_02: I call it trajectory is like the actual history.
• 1:12:10
  SPEAKER_02: Of what the agent has done in the past.
• 1:12:13
  SPEAKER_02: And so if I say, hey,
• 1:12:14
  SPEAKER_02: make this change and the agent makes the change and then it runs the test
• 1:12:16
  SPEAKER_02: and they're broken.
• 1:12:17
  SPEAKER_02: And then it fixes the test.
• 1:12:18
  SPEAKER_02: I have very high confidence that next change I asked to make,
• 1:12:21
  SPEAKER_02: it's going to follow that path again because it's like, OK, here's a conversation.
• 1:12:25
  SPEAKER_02: And the last time the user asked me to do a thing, I made the change.
• 1:12:27
  SPEAKER_02: I ran the test, tests were broken, fixed the test in there.
• 1:12:29
  SPEAKER_02: Then I told the user.
• 1:12:30
  SPEAKER_02: But if I say make a change and it makes a change that doesn't run the tests,
• 1:12:34
  SPEAKER_02: Then I'm on a different track.
• 1:12:35
  SPEAKER_02: Trajectory.
• 1:12:35
  SPEAKER_02: And if I say, okay, make another change, it's like basically they're autoregressive.
• 1:12:39
  SPEAKER_02: So they're predicting what's the next message in this conversation.
• 1:12:42
  SPEAKER_02: And so the example we give...
• 1:12:43
  SPEAKER_02: we talked about in No Vibes Allowed was,
• 1:12:45
  SPEAKER_02: of course, they're like, hey, the model makes a mistake and then you yelled at it.
• 1:12:48
  SPEAKER_02: And then it made another mistake and then you yelled at it.
• 1:12:50
  SPEAKER_02: And then it's like, cool, what's the next message in this conversation?
• 1:12:53
  SPEAKER_02: Well, look, if I read the history, I should probably make an - another mistake.
• 1:12:57
  SPEAKER_02: So the human can yell at me.
• 1:12:58
  SPEAKER_02: The human can yell at me.
• 1:12:59
  SPEAKER_02: So I was like, okay, that's a great example of time to start over.
• 1:13:03
  SPEAKER_01: Let's talk about some observations on how software engineering is changing.
• 1:13:06
  SPEAKER_01: One thing you talked about recently on the evolution of the coding meta is going
• 1:13:10
  SPEAKER_01: from token harder to token smarter.
• 1:13:14
  SPEAKER_01: Can we talk about what you mean by token harder and token smarter?
• 1:13:18
  SPEAKER_02: Yeah, so Token Harder is, I mean,
• 1:13:20
  SPEAKER_02: I'm in a - I'm in a group chat called Hyper Engineering
• 1:13:22
  SPEAKER_02: and it's all like people trying to max out their cloud subs.
• 1:13:25
  SPEAKER_01: Oh wow, It's just like, how do I keep my...
• 1:13:27
  SPEAKER_01: That sounds like a f***ing
• 1:13:28
  SPEAKER_01: A Is this a fun place?
• 1:13:29
  SPEAKER_01: Place.
• 1:13:29
  SPEAKER_02: It's a f - fun place, but it's like All token heart.
• 1:13:31
  SPEAKER_02: Harder.
• 1:13:32
  SPEAKER_02: It's like, look at all the side projects.
• 1:13:33
  SPEAKER_02: I built. Look at everything I've got six cloud code accounts.
• 1:13:40
  SPEAKER_02: I've gotten all of them maxed out every five hour period.
• 1:13:42
  SPEAKER_02: I've timed it out.
• 1:13:43
  SPEAKER_02: So I always use all the tokens And it starts up a...
• 1:13:46
  SPEAKER_02: immediately when the limit resets.
• 1:13:48
  SPEAKER_02: And so I was I mean,
• 1:13:49
  SPEAKER_02: getting into Eli Goldrad and the goal is like optimizing for utilization
• 1:13:53
  SPEAKER_02: and efficiency of one node in your factory rather than the end to end goal of like,
• 1:13:57
  SPEAKER_02: how do we ship value and things that people like that are stable and like will last
• 1:14:00
  SPEAKER_02: a long time.
• 1:14:01
  SPEAKER_02: But that's my idea of token harder.
• 1:14:02
  SPEAKER_02: And it's the same thing with the dark factory thing is like, hey,
• 1:14:05
  SPEAKER_02: if you if you if you remove humans from code review,
• 1:14:07
  SPEAKER_02: you can push more tokens through the system.
• 1:14:09
  SPEAKER_01: So we talk about software factories, but what is the dark factory?
• 1:14:13
  SPEAKER_02: Ah, so the dark factory is this comes from this idea of like there are factories
• 1:14:17
  SPEAKER_02: where everything is automated by robotics.
• 1:14:20
  SPEAKER_02: So you can imagine like a car factory where it's all robots building the cars
• 1:14:23
  SPEAKER_02: and they don't have lights because there's no humans.
• 1:14:26
  SPEAKER_01: Oh, so that's where it comes from.
• 1:14:28
  SPEAKER_02: Dark factory.
• 1:14:29
  SPEAKER_02: Yeah, you walk in there, there's no lights.
• 1:14:30
  SPEAKER_02: There's not even light switches.
• 1:14:31
  SPEAKER_01: So it will be the fully automated software factory where It would be like, no human.
• 1:14:36
  SPEAKER_01: Human input, basically.
• 1:14:37
  SPEAKER_02: No human input.
• 1:14:37
  SPEAKER_02: Raw materials go in, cars come out.
• 1:14:39
  SPEAKER_02: Yep.
• 1:14:40
  SPEAKER_02: And I think in a micro,
• 1:14:42
  SPEAKER_02: you can have many loops that are dark in your thing of like, hey,
• 1:14:45
  SPEAKER_02: if the code review agent comes back with a problem,
• 1:14:48
  SPEAKER_02: you loop that back to the builder agent, it fixes it and comes back.
• 1:14:50
  SPEAKER_02: And that's dark.
• 1:14:51
  SPEAKER_02: You don't need a human loop for that.
• 1:14:53
  SPEAKER_02: But the full dark factor where you don't read any code,
• 1:14:55
  SPEAKER_02: yeah, it's a good way to maximize your token utilization.
• 1:14:58
  SPEAKER_02: It's like if your belief is like,
• 1:15:00
  SPEAKER_02: my job is to extract as much intelligence out of the machine god as I can,
• 1:15:05
  SPEAKER_02: Because that's how I get them.
• 1:15:06
  SPEAKER_02: The most value and the most leverage on my time, then token harder.
• 1:15:10
  SPEAKER_02: Um, it might take take is basically what we talked about before.
• 1:15:12
  SPEAKER_02: Token smarter is like, OK, how do I move faster?
• 1:15:16
  SPEAKER_02: How do I get as much value out of as AI as I can without having turned on.
• 1:15:20
  SPEAKER_02: The lights off while still maintaining control and taste and judgment
• 1:15:23
  SPEAKER_02: and understanding the system architecture and having a lot
• 1:15:26
  SPEAKER_02: of like applying my hard one opinions through 10 years of software engineering to
• 1:15:31
  SPEAKER_02: the design of the program so that I can feel confident that the code is going
• 1:15:35
  SPEAKER_02: to get better and more maintainable over time.
• 1:15:37
  SPEAKER_02: It's the same thing of like you look at like the SRE team inside Google.
• 1:15:40
  SPEAKER_02: They brought out this book, SRE, Site Reliability Engineering.
• 1:15:44
  SPEAKER_02: And the whole take was like, hey, we're going to go from one data center to five data centers.
• 1:15:48
  SPEAKER_02: And we need the same six person team to be able to manage five data centers.
• 1:15:52
  SPEAKER_02: And we need the same same 6 -person team to manage 50 data centers next year.
• 1:15:55
  SPEAKER_02: Year. And it's basically,
• 1:15:56
  SPEAKER_02: how do we apply software to this problem so that instead of scaling linearly
• 1:16:00
  SPEAKER_02: of like, okay, every data center needs five DevOps people.
• 1:16:03
  SPEAKER_02: So we need to scale the people with these things.
• 1:16:04
  SPEAKER_02: How do we continually automate the parts?
• 1:16:07
  SPEAKER_02: So a little bit orthogonal and maybe even like contradictory to what I just said.
• 1:16:11
  SPEAKER_02: But this idea of like, how do you find leverage and the
• 1:16:14
  SPEAKER_01: way the way.
• 1:16:15
  SPEAKER_01: Well, I think what you were saying there is like And Google didn't.
• 1:16:18
  SPEAKER_01: Never seek to remove those SREs from the processes.
• 1:16:22
  SPEAKER_01: They just said like, look,
• 1:16:24
  SPEAKER_01: can we think ahead and scale yourselves?
• 1:16:26
  SPEAKER_01: And they actually grew the team. It wasn't actually six people.
• 1:16:29
  SPEAKER_01: It was more like I think Google specifically said,
• 1:16:31
  SPEAKER_01: OK, we have five data centers next year will have 50 or six of you.
• 1:16:35
  SPEAKER_01: We do not want to have 60 people.
• 1:16:37
  SPEAKER_01: We don't want that.
• 1:16:37
  SPEAKER_01: And then management leader and all that is like,
• 1:16:39
  SPEAKER_01: how can we do it with like 12 or like, or like 10?
• 1:16:42
  SPEAKER_01: And then when we'll have 500 and now actually their SRE has grown,
• 1:16:46
  SPEAKER_01: but, but
• 1:16:46
  SPEAKER_01: of course, yeah, but, but they never, you You know, I think his engine.
• 1:16:49
  SPEAKER_01: Engineers,
• 1:16:50
  SPEAKER_01: like we feel pretty threatened when someone says like,
• 1:16:51
  SPEAKER_01: all right, we just want to have zero engineers.
• 1:16:53
  SPEAKER_01: Like, I mean, that's not a fun place to work at, but It sounds like talk.
• 1:16:58
  SPEAKER_02: Like it's not a possible place to work at.
• 1:16:59
  SPEAKER_02: If they have zero engineers, neither of us could work there.
• 1:17:02
  SPEAKER_01: Right.
• 1:17:02
  SPEAKER_01: but don't understand the token smarter is like let's keep humans in the loop let's keep
• 1:17:06
  SPEAKER_01: adding value and figure out what are the parts which are not as It's relevant.
• 1:17:11
  SPEAKER_01: Boring, where we don't need it.
• 1:17:12
  SPEAKER_01: And so like one developer can probably do more than before,
• 1:17:16
  SPEAKER_01: but you are built to like, be part of this whole thing.
• 1:17:19
  SPEAKER_01: and the lights are on in a factory.
• 1:17:20
  SPEAKER_02: Yep.
• 1:17:21
  SPEAKER_02: Yeah.
• 1:17:21
  SPEAKER_02: And it's like,
• 1:17:21
  SPEAKER_02: basically,
• 1:17:22
  SPEAKER_02: I think,
• 1:17:22
  SPEAKER_02: I think what I'm trying to get to is like the connection here is like SRE built
• 1:17:25
  SPEAKER_02: a thing where like headcount scales at like a square root function or
• 1:17:29
  SPEAKER_02: a logarithmic function, whereas their output scales like linearly.
• 1:17:33
  SPEAKER_02: And you want to say that the way you do that is with good architecture and good program design.
• 1:17:37
  SPEAKER_02: And so in order to like avoid this problem where you have to throw more people
• 1:17:41
  SPEAKER_02: or more tokens at, at the problem.
• 1:17:43
  SPEAKER_02: If you design good software in such a way that it gets more maintainable
• 1:17:46
  SPEAKER_02: and more scalable over time,
• 1:17:48
  SPEAKER_02: and like just today it doesn't feel like – basically you need humans in the loop
• 1:17:52
  SPEAKER_02: to be able to do that.
• 1:17:53
  SPEAKER_02: Let's talk about…
• 1:17:56
  SPEAKER_01: AI slop.
• 1:17:57
  SPEAKER_01: At one point you wrote, yeah, AI can write your code, but it can also write your specs and PRDs.
• 1:18:01
  SPEAKER_01: But the same rule is always slop in, slop out.
• 1:18:04
  SPEAKER_01: If you outsource your thinking, you're going to get garbage.
• 1:18:07
  SPEAKER_01: Yep.
• 1:18:09
  SPEAKER_02: So yeah,
• 1:18:09
  SPEAKER_02: that's basically the idea is like the way we think
• 1:18:12
  SPEAKER_02: about like getting high quality outputs is like,
• 1:18:15
  SPEAKER_02: yeah, you could write the code by hand.
• 1:18:17
  SPEAKER_02: Or you could sit with a model and work back and forth and go maybe
• 1:18:21
  SPEAKER_02: a little bit faster and you have control.
• 1:18:23
  SPEAKER_02: And every time it makes a change, you go read the change.
• 1:18:25
  SPEAKER_02: And if it's bad, you tell it, nope, we want it like this.
• 1:18:27
  SPEAKER_02: And you kind of incrementally slowly.
• 1:18:28
  SPEAKER_02: This is like kind of the stage two or stage three version of working with.
• 1:18:32
  SPEAKER_02: Agents were like agents writing all your code, but you're kind of very much in the loop.
• 1:18:36
  SPEAKER_02: And this will make you go faster, but it won't make you go that much faster.
• 1:18:39
  SPEAKER_02: It won't make you go anywhere near there's like there's like that level.
• 1:18:41
  SPEAKER_02: And then there's like the maximum speed you can go while still caring about the code.
• 1:18:45
  SPEAKER_02: And then there's like the maximum speed you can go if you turn the lights off.
• 1:18:48
  SPEAKER_02: And so we always think about it as like in terms of The leverage is like, okay.
• 1:18:51
  SPEAKER_02: OK, let me take everything starts with like a sentence or a voice note ramble.
• 1:18:54
  SPEAKER_02: Like I want to build this thing.
• 1:18:55
  SPEAKER_02: It's going to work like this, whatever it is.
• 1:18:57
  SPEAKER_02: So let's say like on average, like two sentences, I got to fix this thing.
• 1:19:00
  SPEAKER_02: Or there's a support ticket, I got to fix this thing.
• 1:19:02
  SPEAKER_02: If you can turn that with AI into a one pager and then turn that one page
• 1:19:06
  SPEAKER_02: and make sure that's correct.
• 1:19:07
  SPEAKER_02: And then turn that one pager into a three pager and make sure that's correct.
• 1:19:10
  SPEAKER_02: And then turn that three pager into a 10 page, like detailed outline.
• 1:19:14
  SPEAKER_02: Then you can write a hundred pages worth of code.
• 1:19:16
  SPEAKER_02: And it's me.
• 1:19:17
  SPEAKER_02: Maybe not perfect.
• 1:19:18
  SPEAKER_02: You shouldn't like sweat over these documents and make sure they're perfect,
• 1:19:21
  SPEAKER_02: but you're increasing the chance that like You're decreasing the un - uncertainty of
• 1:19:26
  SPEAKER_02: the outputs.
• 1:19:28
  SPEAKER_02: You have a line of where it's going and then you have the probabilities of
• 1:19:31
  SPEAKER_02: where like it might might go in that range.
• 1:19:34
  SPEAKER_02: If you are reviewing along the way as you get more and more detailed
• 1:19:38
  SPEAKER_02: into How what you're building and how.
• 1:19:39
  SPEAKER_02: How you want it to be built, you Kind of collapse the unscathed.
• 1:19:42
  SPEAKER_02: Uncertainty and the set of end states that you could land in.
• 1:19:46
  SPEAKER_02: That's It's me doing the physics.
• 1:19:47
  SPEAKER_02: Physics thing of like, you got to superimpose all these probabilities.
• 1:19:50
  SPEAKER_02: And like,
• 1:19:50
  SPEAKER_02: I don't know,
• 1:19:51
  SPEAKER_02: I have this thing that like,
• 1:19:52
  SPEAKER_02: I think people
• 1:19:52
  SPEAKER_02: who really like playing real time strategy games Those are probably going
• 1:19:55
  SPEAKER_02: to be really good.
• 1:19:56
  SPEAKER_02: Really good with AI because you kind of have to like,
• 1:19:59
  SPEAKER_02: I don't know, Matt Pocock was just talking about fog of Yeah.
• 1:20:21
  SPEAKER_02: And decide what's the most likely path that's going to lead us to success.
• 1:20:24
  SPEAKER_01: Speaking of the most likely path that leads you to success,
• 1:20:27
  SPEAKER_01: let's talk about your company that You've, you've just.
• 1:20:30
  SPEAKER_01: Come out of stealth, Human Layer.
• 1:20:33
  SPEAKER_01: What is Human Layer and what is the probability that you're setting up for success?
• 1:20:37
  SPEAKER_02: That's a good question.
• 1:20:38
  SPEAKER_02: A hundred percent.
• 1:20:39
  SPEAKER_02: A hundred percent probability.
• 1:20:41
  SPEAKER_02: Maybe a hundred and ten.
• 1:20:42
  SPEAKER_02: But no, so Human Layer is an AI IDE.
• 1:20:46
  SPEAKER_02: It's a collaboration platform and it is building blocks for your software factory.
• 1:20:51
  SPEAKER_02: And the basic pitch is like engineers solving hard problems and complex code bases.
• 1:20:54
  SPEAKER_02: Basically, there's two categories.
• 1:20:56
  SPEAKER_02: There's like vibe coders building side projects and
• 1:20:59
  SPEAKER_02: then there's people building production software where the stakes are high.
• 1:20:59
  SPEAKER_02: And then there's people building.
• 1:21:01
  SPEAKER_02: High. And if something breaks,
• 1:21:02
  SPEAKER_02: we're going to get fined millions of dollars or,
• 1:21:04
  SPEAKER_02: you know, we're gonna lose millions millions of money for the company.
• 1:21:07
  SPEAKER_02: And there's a whole spectrum in between there.
• 1:21:09
  SPEAKER_02: But it's like if you were kind of in the left half of that spectrum,
• 1:21:12
  SPEAKER_02: you're building software that matters and it has to last and be around for a while,
• 1:21:17
  SPEAKER_02: then we are helping people like that solve problems two to three times faster
• 1:21:20
  SPEAKER_02: without descending into slop.
• 1:21:22
  SPEAKER_02: How do you maintain that near human level of quality and move two to three times faster?
• 1:21:27
  SPEAKER_01: And what And were the idea ideas that you built and that you came with?
• 1:21:31
  SPEAKER_02: One idea that we're really excited about right now, I mean,
• 1:21:34
  SPEAKER_02: it all comes from this RPI and this like using specs to like,
• 1:21:37
  SPEAKER_02: I mean, I've kind of been I'm hinting at it this.
• 1:21:39
  SPEAKER_02: This whole time, right?
• 1:21:40
  SPEAKER_02: Of like, okay, cool.
• 1:21:41
  SPEAKER_02: Like start really high level and zoom in layer by layer and re-steer and like find
• 1:21:46
  SPEAKER_02: that leverage that helps you move faster and increase the chance
• 1:21:48
  SPEAKER_02: that your agent's going to build exactly what you want
• 1:21:50
  SPEAKER_02: or something that's really high quality.
• 1:21:53
  SPEAKER_02: The other thing I think that's really interesting that
• 1:21:55
  SPEAKER_02: where I just posted yesterday, I said, hey, chat, should we kill the pull request?
• 1:21:58
  SPEAKER_02: And that's It's something I can't talk to.
• 1:22:00
  SPEAKER_02: Too much about.
• 1:22:00
  SPEAKER_02: But basically,
• 1:22:00
  SPEAKER_02: the idea is like the IDE of the future needs to be rethought from the ground up
• 1:22:06
  SPEAKER_02: for agents.
• 1:22:07
  SPEAKER_02: And it might not even be a like.
• 1:22:09
  SPEAKER_02: I don't know.
• 1:22:10
  SPEAKER_02: A lot of editors kind of started with the text field and bolted on an agents tab.
• 1:22:14
  SPEAKER_02: And then eventually you've seen like cursor three.
• 1:22:16
  SPEAKER_02: I can't even find the text editor.
• 1:22:17
  SPEAKER_02: I know it exists.
• 1:22:18
  SPEAKER_02: People have told me you can get to a text view of files, but it's also very agent first.
• 1:22:23
  SPEAKER_02: And so we started from the ground up of like,
• 1:22:24
  SPEAKER_02: what is an IDE designed for helping a developer interact with and manage the work
• 1:22:28
  SPEAKER_02: of agents?
• 1:22:30
  SPEAKER_02: And then we zoomed out and said,
• 1:22:31
  SPEAKER_02: how do we make this collaborative and build in a sync engine and durable streams
• 1:22:34
  SPEAKER_02: and all of these like pieces of tech that enable...
• 1:22:38
  SPEAKER_02: Me to get human input and feedback on what I'm doing with agents in real time rather
• 1:22:44
  SPEAKER_02: than waiting for the pull request time.
• 1:22:46
  SPEAKER_02: And great engineering teams have been doing this for decades of like, hey,
• 1:22:48
  SPEAKER_02: we're going to have a design review where we're going to talk about how we're going
• 1:22:50
  SPEAKER_02: to build the thing as like a two page Google Doc.
• 1:22:53
  SPEAKER_02: Or whatever, 10 page, whatever.
• 1:22:55
  SPEAKER_01: PRD, ERD.
• 1:22:56
  SPEAKER_01: Yeah.
• 1:22:57
  SPEAKER_02: AR architecture requirements document.
• 1:22:59
  SPEAKER_02: And then you go to sprint planning and you break it down into little tickets
• 1:23:02
  SPEAKER_02: and you decide who's going to do what.
• 1:23:03
  SPEAKER_02: It's like AI can help with all of this.
• 1:23:05
  SPEAKER_02: If you're just using AI to write the code,
• 1:23:07
  SPEAKER_02: you're missing out on a lot of the benefits that AI can bring to your SDLC.
• 1:23:11
  SPEAKER_02: And a lot of people say like, well,
• 1:23:12
  SPEAKER_02: we don't need any of those meetings anymore because we have the loop,
• 1:23:14
  SPEAKER_02: we have the dark factory, things just fly around the loop.
• 1:23:16
  SPEAKER_02: But it's like, okay,
• 1:23:17
  SPEAKER_02: but if you want to actually move faster and maintain quality,
• 1:23:19
  SPEAKER_02: then like you should have these checkpoints before you go to actually write
• 1:23:22
  SPEAKER_02: the code.
• 1:23:22
  SPEAKER_02: And you should use AI to help with that.
• 1:23:24
  SPEAKER_02: So we built this like cloud platform that's kind of has like
• 1:23:26
  SPEAKER_02: a Google Docs style component where you can comment and
• 1:23:29
  SPEAKER_02: the agent can surface like mockups and mermaid diagrams and HTML
• 1:23:33
  SPEAKER_02: and all these things.
• 1:23:33
  SPEAKER_02: So basically, how do we make agents like Figma style?
• 1:23:37
  SPEAKER_02: Everything's in the cloud.
• 1:23:38
  SPEAKER_02: Everything's collaborative.
• 1:23:39
  SPEAKER_02: I see all my co-worker sessions.
• 1:23:40
  SPEAKER_02: They see all of mine.
• 1:23:41
  SPEAKER_02: It's almost like the benefit that Slack had over email was that you didn't have
• 1:23:47
  SPEAKER_02: to be in every conversation to know what was happening.
• 1:23:50
  SPEAKER_02: You could maintain,
• 1:23:51
  SPEAKER_02: you could see all these channels light up,
• 1:23:52
  SPEAKER_02: you could check on them, you go, okay, I don't care about any of that.
• 1:23:54
  SPEAKER_02: But if you saw a conversation that you cared about, you could jump in on that.
• 1:23:57
  SPEAKER_02: And it's like,
• 1:23:58
  SPEAKER_02: how do we do that for engineering work versus like,
• 1:24:00
  SPEAKER_02: we really had these like very strict,
• 1:24:03
  SPEAKER_02: even when we called it agile, it was very waterfall-y, like PRD.
• 1:24:07
  SPEAKER_02: ARD, tickets,
• 1:24:09
  SPEAKER_02: everyone goes and builds for a day and then you get the PR back and
• 1:24:13
  SPEAKER_02: then one person reviews it.
• 1:24:14
  SPEAKER_02: How do you create this more just like soup and like what is the data model for
• 1:24:19
  SPEAKER_02: that world where you have like agentic traces,
• 1:24:21
  SPEAKER_02: you have documents,
• 1:24:22
  SPEAKER_02: you have tasks and projects that group these things,
• 1:24:25
  SPEAKER_02: you have actual git diffs being streamed everywhere
• 1:24:28
  SPEAKER_02: where it's like why would I review all the code at once when I can just all always,
• 1:24:32
  SPEAKER_02: everybody's work lives in a shared environment that anyone can go interact with.
• 1:24:36
  SPEAKER_01: I mean, what are our minds?
• 1:24:38
  SPEAKER_01: Reminds me of is what You know, githubed at the software.
• 1:24:40
  SPEAKER_01: Software teams.
• 1:24:41
  SPEAKER_01: Before GitHub and its competitors, you might have a tracker somewhere, But most scenes...
• 1:24:48
  SPEAKER_01: teams were just kind of like inside the company.
• 1:24:50
  SPEAKER_01: You didn't know what one team was.
• 1:24:51
  SPEAKER_01: I remember pre-GitHub, like,
• 1:24:52
  SPEAKER_01: you know,
• 1:24:53
  SPEAKER_01: you had individual teams,
• 1:24:54
  SPEAKER_01: some of them had like a board with stickers,
• 1:24:57
  SPEAKER_01: but no one else in the company knew what they were doing.
• 1:24:59
  SPEAKER_01: They were all working in isolation.
• 1:25:00
  SPEAKER_01: And now when you have GitHub or even the internal version of GitHub inside
• 1:25:04
  SPEAKER_01: a company,
• 1:25:05
  SPEAKER_01: you can always see when you go You go to a team, you see the pull request flying.
• 1:25:08
  SPEAKER_01: You can join in.
• 1:25:09
  SPEAKER_01: You have history.
• 1:25:11
  SPEAKER_01: It is all kind of connected and it came together.
• 1:25:13
  SPEAKER_01: And now for a very long time I was like, duh,
• 1:25:16
  SPEAKER_01: you're going to use GitHub or people will copy So do I sense that you're trying
• 1:25:20
  SPEAKER_01: to build something?
• 1:25:21
  SPEAKER_01: This workflow for when you have The Software Factory.
• 1:25:25
  SPEAKER_01: Factories,
• 1:25:26
  SPEAKER_01: which are dark factories and loops at a bunch of places,
• 1:25:29
  SPEAKER_01: how can we have this new way of working,
• 1:25:32
  SPEAKER_01: which will feel natural, but Like, coming up with it.
• 1:25:35
  SPEAKER_01: Like, it's hard.
• 1:25:36
  SPEAKER_01: Hard work.
• 1:25:36
  SPEAKER_01: It's counterintuitive.
• 1:25:38
  SPEAKER_02: How can we do something that accomplishes what GitHub did,
• 1:25:40
  SPEAKER_02: but like 10x better,
• 1:25:41
  SPEAKER_02: like more,
• 1:25:42
  SPEAKER_02: specifically like more continuous and more real time and more collaborative
• 1:25:46
  SPEAKER_02: than like these discrete units of work that is like the pull request.
• 1:25:50
  SPEAKER_01: Well, I now I'm starting to understand why you're saying maybe we should kill
• 1:25:53
  SPEAKER_01: the pull request because pull request was invented by GitHub, right?
• 1:25:55
  SPEAKER_01: Like it's,
• 1:25:56
  SPEAKER_01: it is not part of Git,
• 1:25:57
  SPEAKER_01: but they do that as a way for you to do a code review,
• 1:26:01
  SPEAKER_01: merge before it goes in and be able to modify it or like just reject Rejected it,
• 1:26:05
  SPEAKER_01: et cetera.
• 1:26:05
  SPEAKER_01: Cetera.
• 1:26:06
  SPEAKER_02: And it's probably a lot better than whatever we had before,
• 1:26:08
  SPEAKER_02: which I guess was like emailing your Git patch to Linus and ask him to merge it into
• 1:26:12
  SPEAKER_02: the kernel
• 1:26:12
  SPEAKER_01: or whatever.
• 1:26:13
  SPEAKER_01: They still do it.
• 1:26:14
  SPEAKER_01: It works for them.
• 1:26:15
  SPEAKER_01: That's the point, but it only works
• 1:26:16
  SPEAKER_01: for them.
• 1:26:16
  SPEAKER_01: Yeah.
• 1:26:17
  SPEAKER_02: I don't know anybody else who does that.
• 1:26:18
  SPEAKER_02: I mean, I'm sure even before Get Up for you, you guys had what, like CVS or Subversion?
• 1:26:23
  SPEAKER_01: CVS, TFS, if you have a lot of money for Microsoft.
• 1:26:25
  SPEAKER_02: They made us use Subversion in undergrad because the guy
• 1:26:29
  SPEAKER_02: who invented Subversion Was it UChicago?
• 1:26:31
  SPEAKER_02: UChicago guy.
• 1:26:32
  SPEAKER_02: The year after I graduated, they switched everybody to Git.
• 1:26:34
  SPEAKER_02: And I was like, damn, I learned a useless thing just for somebody's ego.
• 1:26:39
  SPEAKER_01: Specifically for AI startups or startups building on top of AI
• 1:26:42
  SPEAKER_01: or building AI products, how important do you think location and network is?
• 1:26:46
  SPEAKER_01: Especially you are based in the valley.
• 1:26:49
  SPEAKER_01: We see research that AI startups are more frequently funded from here
• 1:26:54
  SPEAKER_01: than normal startups as well.
• 1:26:56
  SPEAKER_01: Do you see this advantage?
• 1:26:57
  SPEAKER_01: And also,
• 1:26:58
  SPEAKER_01: do you see some disadvantages of being a specific,
• 1:27:00
  SPEAKER_01: may that be Silicon Valley or elsewhere?
• 1:27:02
  SPEAKER_02: I don't have really strong opinions on this.
• 1:27:04
  SPEAKER_02: Actually, like Paul Graham gave a talk in Sweden about why SF is cool.
• 1:27:08
  SPEAKER_02: Rather than just regurgitate that, I will forward people onto that one.
• 1:27:11
  SPEAKER_02: We can put it in the show notes or whatever,
• 1:27:13
  SPEAKER_02: but he talks about all of the dynamics Dynamics of silver.
• 1:27:16
  SPEAKER_02: Silicon Valley and the paid forward culture and the like,
• 1:27:19
  SPEAKER_02: People take you way more seriously just because you're based here.
• 1:27:21
  SPEAKER_02: I lived in Chicago for a long time.
• 1:27:23
  SPEAKER_02: I have a lot of really good friends from high school,
• 1:27:25
  SPEAKER_02: from college,
• 1:27:25
  SPEAKER_02: from growing up in LA,
• 1:27:26
  SPEAKER_02: and never before have I felt like So locked in with my people.
• 1:27:31
  SPEAKER_02: Never have I felt more seen, more connected.
• 1:27:35
  SPEAKER_02: There's just so many people here, again,
• 1:27:36
  SPEAKER_02: talking about the founder thing,
• 1:27:37
  SPEAKER_02: people who care deeply,
• 1:27:39
  SPEAKER_02: who are incredibly competent, who we all have all the same types of problems.
• 1:27:43
  SPEAKER_02: We love all the same types of things.
• 1:27:45
  SPEAKER_02: I don't do LAN parties where we play video games,
• 1:27:47
  SPEAKER_02: but all my buddies will come over and we'll sit in the office till 11.
• 1:27:49
  SPEAKER_02: We'll just do co-working and hack on cool, fun projects and stuff.
• 1:27:53
  SPEAKER_02: You can't do that anywhere else.
• 1:27:54
  SPEAKER_02: There's not enough Like Critical math.
• 1:27:58
  SPEAKER_02: Mass for that to just happen organically everywhere you go.
• 1:28:00
  SPEAKER_02: And I absolutely love it.
• 1:28:02
  SPEAKER_02: I wouldn't trade it for anything.
• 1:28:03
  SPEAKER_01: Yeah, I've been a critical masses.
• 1:28:04
  SPEAKER_01: It nails it on the head.
• 1:28:06
  SPEAKER_01: When it comes to hiring, what types of folks are you hiring for specifically?
• 1:28:10
  SPEAKER_01: Because I'm interested in how hiring changes and what a standout engineer means
• 1:28:14
  SPEAKER_01: for you and how So you are...
• 1:28:16
  SPEAKER_01: are trying to You know, confirm that those.
• 1:28:18
  SPEAKER_01: Those traits exist.
• 1:28:19
  SPEAKER_02: In general, we are looking Um, for people who.
• 1:28:22
  SPEAKER_02: Who have really strong software fundamentals.
• 1:28:25
  SPEAKER_02: So understand distributed systems, understand Like the core fundamental.
• 1:28:29
  SPEAKER_02: Fundamentals of CS and operating systems and these kind of things.
• 1:28:32
  SPEAKER_02: I mean, you don't have to be a PhD in freaking kernel design or whatever, but it's a lot easier.
• 1:28:37
  SPEAKER_02: We can We can teach, we can teach.
• 1:28:39
  SPEAKER_02: somebody, I think, to be around.
• 1:28:39
  SPEAKER_02: A really good AI developer in a few months.
• 1:28:41
  SPEAKER_02: You can build enough intuition where you are...
• 1:28:43
  SPEAKER_02: You know, accelerated off the ground and you can go like keep growing there.
• 1:28:47
  SPEAKER_02: It's really hard to teach someone a CS undergrad program in three months.
• 1:28:51
  SPEAKER_01: And what's the problem space that you're excited about in And
• 1:28:55
  SPEAKER_01: in software engineering, we would.
• 1:28:56
  SPEAKER_01: Even product engineering or building products that you think in
• 1:28:58
  SPEAKER_01: the next few years is going to be one of the interesting things that you're going
• 1:29:01
  SPEAKER_01: to be attacking?
• 1:29:02
  SPEAKER_02: My co-founder could talk more about this,
• 1:29:04
  SPEAKER_02: but like there's a lot of interesting things happening in real life.
• 1:29:07
  SPEAKER_02: Real time and cloud and sandboxes and sync and kind
• 1:29:11
  SPEAKER_02: of like using these new building blocks that have gotten really solid in
• 1:29:15
  SPEAKER_02: the last couple of years.
• 1:29:16
  SPEAKER_02: We're big fans of the electric SQL team.
• 1:29:18
  SPEAKER_02: We're users of durable streams.
• 1:29:19
  SPEAKER_02: It was like,
• 1:29:20
  SPEAKER_02: how can you build systems that kind of are a lot more spread out and distributed
• 1:29:25
  SPEAKER_02: and almost like decentralized?
• 1:29:27
  SPEAKER_02: This is really interesting for coding because you want to be able to run coding agents anywhere.
• 1:29:29
  SPEAKER_02: You want to be able to run them.
• 1:29:31
  SPEAKER_02: For a short time,
• 1:29:32
  SPEAKER_02: for a long time,
• 1:29:33
  SPEAKER_02: on demand,
• 1:29:34
  SPEAKER_02: on a schedule,
• 1:29:34
  SPEAKER_02: all these things and have them all be part of this kind of like brain.
• 1:29:38
  SPEAKER_02: So I don't know, parts of what we're doing are really boring, like all our data is in Postgres.
• 1:29:41
  SPEAKER_02: And then parts of what we're doing is really interesting.
• 1:29:43
  SPEAKER_02: But there's a lot of distributed systems problems.
• 1:29:45
  SPEAKER_02: There's a lot of infrastructure problems.
• 1:29:46
  SPEAKER_02: Like we are building tools for AI,
• 1:29:49
  SPEAKER_02: but there's a lot of problems in building collaboration platforms that are really,
• 1:29:53
  SPEAKER_02: really hard.
• 1:29:54
  SPEAKER_02: And there's a lot of new tech that makes it easier and more interesting,
• 1:29:56
  SPEAKER_02: but it's still Well, by far from...
• 1:29:58
  SPEAKER_02: from an easy problem.
• 1:30:00
  SPEAKER_01: It sounds like what you're saying is like infralayers,
• 1:30:02
  SPEAKER_01: Inflayers,
• 1:30:02
  SPEAKER_01: it's some extent a new infralayer is being built and it'll take some time,
• 1:30:06
  SPEAKER_01: but it'll be like Just new, new blocks.
• 1:30:09
  SPEAKER_01: Blocks and it will eventually become the primitives like for cloud.
• 1:30:11
  SPEAKER_01: We have primitives already, but it took freaking decades to get those together or more.
• 1:30:15
  SPEAKER_02: Yeah.
• 1:30:15
  SPEAKER_02: You had AWS in what, like 2006?
• 1:30:18
  SPEAKER_02: 2006.
• 1:30:19
  SPEAKER_02: Yeah.
• 1:30:20
  SPEAKER_02: And then you got Kubernetes a decade later.
• 1:30:22
  SPEAKER_01: Yep.
• 1:30:22
  SPEAKER_01: And as closing, what's a book or reading that you would recommend?
• 1:30:27
  SPEAKER_01: Something that you personally enjoyed?
• 1:30:28
  SPEAKER_02: Nowadays, we talk a lot about Refactoring by Martin Fowler.
• 1:30:32
  SPEAKER_02: Love it.
• 1:30:36
  SPEAKER_02: Well, Dex, thanks so
• 1:30:48
  SPEAKER_02: much. This was fun.
• 1:30:56
  SPEAKER_01: This is
• 1:30:58
  SPEAKER_02: a a blast dude!
• 1:31:00
  SPEAKER_02: Thanks for having me on.
• 1:31:01
  SPEAKER_02: This is great.
• 1:31:02
  SPEAKER_00: I had a lot of fun.
• 1:31:04
  SPEAKER_00: I don't know about you, but I really enjoyed this conversation.
• 1:31:06
  SPEAKER_00: Dex is such a big believer in gender coding,
• 1:31:08
  SPEAKER_00: yet he's the one warning us that if you stop reading the code,
• 1:31:11
  SPEAKER_00: you have about 3 -6 months before your codebase becomes easier to rewrite than
• 1:31:15
  SPEAKER_00: to fix.
• 1:31:16
  SPEAKER_00: And this comes from first.
• 1:31:17
  SPEAKER_00: Furt has an experience.
• 1:31:18
  SPEAKER_00: His team built a light soft software factory, ran it, and then had to shut it down.
• 1:31:23
  SPEAKER_00: I also like the idea of the slow loop.
• 1:31:26
  SPEAKER_00: Loop engineering feels like a somewhat meaningless term to me.
• 1:31:29
  SPEAKER_00: What DEX's team does It's actually pretty...
• 1:31:32
  SPEAKER_00: pretty boring.
• 1:31:33
  SPEAKER_00: A cron job runs every night,
• 1:31:34
  SPEAKER_00: fixes one issue or one anti-pattern, and opens one small pull request.
• 1:31:38
  SPEAKER_00: The team wakes up to a codebase that's a little bit better every morning,
• 1:31:41
  SPEAKER_00: and devs still need to review and approve it.
• 1:31:43
  SPEAKER_00: This is a practice that, honestly, any engineering team could just adopt today.
• 1:31:47
  SPEAKER_00: Finally, I really enjoyed the history lesson.
• 1:31:49
  SPEAKER_00: The term software factory comes from a NATO conference in 1968.
• 1:31:52
  SPEAKER_00: In 1968, the idea of solving a problem was to solve a problem.
• 1:31:54
  SPEAKER_00: Software used to build software with analogies to a factory is more
• 1:31:57
  SPEAKER_00: than 60 years old,
• 1:31:59
  SPEAKER_00: and every generation of our industry has tried to automate more of the loop
• 1:32:03
  SPEAKER_00: of building software.
• 1:32:04
  SPEAKER_00: AI agents are just yet one more attempt, although probably the most successful one.
• 1:32:08
  SPEAKER_00: Do check out show notes below for the related The Pragmatic Engineering Deep Dives
• 1:32:11
  SPEAKER_00: that go even deeper into AI engineering and other related topics
• 1:32:14
  SPEAKER_00: If you've enjoyed this podcast,
• 1:32:15
  SPEAKER_00: please do subscribe on your favorite podcast platform and on YouTube.
• 1:32:19
  SPEAKER_00: A special thank you if you also leave a rating on the show.
• 1:32:21
  SPEAKER_00: Thanks, and see you in the next one.