Latest / Superlinear / Beyond Root-Level Harness Engineering: A Different Environment for Every Task

Transcript

Beyond Root-Level Harness Engineering: A Different Environment for Every Task · Superlinear

• 0:00
  Christine: putting a custom harness in every meaningful part of your workflow and not just at the root of your project, you can squeeze dramatically more intelligence out of your agents.
• 0:11
  Christine: Basically, general-purpose harnesses are built to handle broad use cases and messy realities.
• 0:17
  Christine: They are not optimal for any one task because they need to be decent at all of them.
• 0:22
  Brandon: The next step
• 0:23
  Christine: Yeah. Yeah. Yeah.
• 0:24
  Brandon: is to treat harness engineering as something we do at every meaningful
• 0:26
  Christine: Yeah. Yeah
• 0:27
  Brandon: section of your project and to do it even for every sufficiently hard task.
• 0:32
  Brandon: An agent should not just have its own job to do, it should have its own world.
• 1:02
  Christine: In this episode we will briefly review what we mean when we say harness engineering. We will talk about what if you deliberately built a harness in every meaningful part of your workflow and even between sub-agents in the same prompt and not just on the top of your work tree.
• 1:19
  Christine: We'll also talk about when narrow- when you have to narrow the capabilities of your agents and how to enforce behavior instead of just hoping that the model will follow your instructions.
• 1:31
  Christine: So we've all probably experienced an agent making mistakes, like ignoring instructions or repeatedly producing poor results.
• 1:39
  Christine: And often the way we respond is by adding more instructions.
• 1:43
  Christine: For example, we clarify the prompt, we create a skill or we paste in more context or we write another role in agents dot MD[UM] but and sometimes this all works but it can also turn one general-purpose harness into an increasingly large collection of guidance that every task must carry around.
• 2:04
  Christine: And Brandon, I know you and I like talked for hours the other day about this and you have a lot of thoughts about it.
• 2:08
  Christine: Have you run into this as well
• 2:12
  Brandon: Yeah. We, we have to remember that the models have a certain amount of attention.
• 2:17
  Brandon: So, the more things that you put in context, the more distracted they get.
• 2:22
  Brandon: Right? So,
• 2:24
  Christine: That makes sense.
• 2:25
  Brandon: if you're blindly shoving things in your agents dot MD whenever there's a mistake, if it's something that's rare, then you're distracting the agent when it's doing most of its other work, which is
• 2:36
  Christine: Yeah.
• 2:36
  Brandon: not good,
• 2:37
  Christine: Yeah,
• 2:37
  Brandon: not good.
• 2:37
  Christine: yeah.
• 2:38
  Brandon: I think
• 2:38
  Christine: Yeah,
• 2:39
  Brandon: a lot of people have started to do better. They're they're starting to do progressive disclosure of information. So if you're using skills, for example, then your agent will only pick up the skill if it's about to do something that is related to that skill, unless you, you know, manually include it[UH] and there's other ways to
• 2:58
  Christine: yeah.
• 2:59
  Brandon: do this.
• 2:59
  Christine: Yeah
• 2:59
  Brandon: You can have a directory of different markdown files and pull them in as needed.
• 3:03
  Brandon: So.
• 3:04
  Brandon: But this is only one half of the problem.
• 3:06
  Christine: Mm-hmm.
• 3:06
  Brandon: Cuz as you as you know, listener [laughter] or maybe as as you might remember, agents get their context one from these feedforward guides, markdown that we're inserting ahead of time, but also from sensors, from things that are giving feedback, from tests that are running, from code reviews, things like this. So, these are giving feedback into the agent after they take certain steps. And
• 3:33
  Christine: I
• 3:33
  Brandon: that's also
• 3:33
  Christine: know.
• 3:33
  Brandon: adding to the context.
• 3:35
  Christine: Yeah, I know we talked about this in our episode three, the latest episode, but basically guides are things and context that the agent gets before it starts its task, and the sensors are the things that the agent get after the task and they would give, for example, test lend rules. Right?
• 3:53
  Brandon: Yeah.
• 3:54
  Christine: OK,
• 3:54
  Brandon: after
• 3:54
  Christine: cool.
• 3:54
  Brandon: steps basically yeah. So.
• 3:56
  Christine: OK. It it makes sense
• 3:57
  Brandon: So.
• 3:58
  Christine: what you're
• 3:58
  Brandon: Mm-hmm.
• 3:58
  Christine: saying because [UM][UM] also just [UH] [UH] to remind myself, because if you put for s- for I like how you said like some skills you could only invoke them when the time is right, because
• 4:11
  Brandon: Yeah
• 4:11
  Christine: you know, a a bad example would be to put it in the agents or Claude dot MD because basically every agent or sub-agent that you spawn will also read that markdown file, right?
• 4:21
  Brandon: Exactly.
• 4:22
  Christine: So, so I guess the problem is not, hey the model needs another instruction, but sometimes the model needs a better world, like you said, or an [UH] a better environment.
• 4:31
  Brandon: Right.
• 4:32
  Brandon: Exactly.
• 4:32
  Christine: Awesome.
• 4:33
  Brandon: Instructions are just part of the environment. They're just part of the world.
• 4:35
  Brandon: But there's other parts in the world too
• 4:39
  Christine: Gotcha. I would love to dive into that and and I think to talk about that we need to dive into harness engineering.
• 4:43
  Christine: And I know you are jumping[laughter] to get
• 4:47
  Brandon: [laughter]
• 4:47
  Christine: into harness engineering. Maybe before we like dive into this[UH] [UM] different way of thinking about harness engineering, can we first tell everyone what we mean when we say harness engineering? Like what what are the things exactly that are included in the harness
• 5:04
  Brandon: when we say harness engineering, we mean the coding agent harness, Claude Code, Codex, pi, maybe an SDK, how that's
• 5:16
  Christine: Yeah.
• 5:16
  Brandon: customized globally and a layer on top of that, how it's customized for your particular project. This is how people typically talk about harness engineering.
• 5:28
  Brandon: plus other guides
• 5:29
  Christine: Yeah.
• 5:30
  Brandon: and other sensors.
• 5:31
  Brandon: So the markdown that's part of Agents dot MD, any skills,
• 5:34
  Christine: Mm-hmm.
• 5:36
  Brandon: any other reference files, even things that are automatically generated but inserted ahead of the agent doing its task.
• 5:45
  Brandon: And then the sensors that are relevant for your project or your system, like every time I run git commit, automatically run these tests and if there's a test failure, you know, exit loudly and it redirects to H.
• 6:02
  Christine: Gotcha. So so the harness can be very broad, like it's not only the prompts, not only the project instructions and the skills and even the tools, but I think you're referring to like it's much broader, like tests, oracles, hooks probably as well, maybe profilers,
• 6:19
  Brandon: Plugins.
• 6:20
  Christine: plugins, permissions, sandboxes, like everything in that environment.
• 6:25
  Christine: Is that right
• 6:25
  Brandon: Yeah.
• 6:26
  Brandon: Yeah. So, I think this is important. When we talk about harness engineering, we can unambiguously talk about this whole thing if you just say harness online, different people will interpret it different ways. But the community has come together and there's a consensus definition for harness engineering, which is to mean that all that stuff as part of the harness that we're engineering, which means we're sort of taking care and setting all these things together holistically to build a system that helps us when we're solving our problems.
• 6:57
  Christine: so it sounds like if you do harness engineering really well, you can push performance way higher than what a bare model can do.
• 7:06
  Christine: And I remembered from our earlier conversation, you mentioned that that you listen to an interesting discussion as part of a podcast, another podcast that you listen to.
• 7:16
  Brandon: Yeah.
• 7:17
  Brandon: Yeah, so so you're going to listen to a clip from the Moonshots podcast, one of my favorite podcasts, where[UH] Dave Blundin is discussing that if you engineer your harness in a particular way, you can basically destroy this particular benchmark.
• 8:18
  Brandon: so so Dave used the term scaffold, but he means harness [laughter] in this in this context. So as as we heard, getting your harness right for a particular problem set or a particular focus can push your performance way higher than even an incrementally smarter model can. what we saw was going
• 8:38
  Christine: Yeah.
• 8:38
  Brandon: from Fable to Opus-5 and even Opus-4 to Fable is dwarfed by having your harness set up properly with a with one of the dumber models. So. So what does this mean? I mean basically one, this is why harness engineering matters, and it's why lots of people have been talking about it, OK?
• 9:00
  Christine: Mm-hmm. Yeah, indeed.
• 9:00
  Christine: Because [UM] it was it wa- it was not only the folks on the Moonshot podcast, but we've also seen people [UM] discussing this on X.
• 9:11
  Christine: I saw Dax from Open Code and Anthropic staff[UM] also touching on this topic. Maybe we can
• 9:18
  Brandon: Yeah.
• 9:18
  Christine: pull up the tweets
• 9:19
  Brandon: Yeah, let's pull that up. So, so Dax is quoting something from the Anthropic stuff, so let's pull that up first so basically Jess is saying here this is the relevant statement. I think I'm quite biased, but I also think that it is impossible to get the maximum performance without tying together the harness and the model. Now, it's important to note that she's talking about harness in the context of cloud code and specifically in the context of a general-purpose harness.
• 9:50
  Brandon: So Dax is quoting this, OK?
• 9:54
  Christine: Mm-hmm.
• 9:54
  Brandon: And he's saying people want to dunk on this by saying their weekend harness does better on some dumb one-shot benchmark. But at scale, this is true.
• 10:03
  Brandon: These tools are complicated and must factor in user behavior. And on and on and on.
• 10:10
  Christine: Mm-hmm
• 10:10
  Brandon: So, so what is he saying? If you're trying to build a really good general-purpose harness, it's hard.
• 10:19
  Christine: Mm-hmm.
• 10:20
  Brandon: It's hard and what you get is you get something that is pretty good at everything but it doesn't one-shot benchmarks.
• 10:27
  Christine: Mm-hmm
• 10:28
  Brandon: Now, Dax is saying this saying, oh, you know, people are trying to build these these harnesses and they're thinking they're good because they can one-shot benchmarks.
• 10:36
  Brandon: I I think that's the wrong take. What the other way to look at this is to say it's possible to one-shot benchmarks by building a specific harness. This is the point of harness engineering.
• 10:47
  Brandon: Make your harness specific.
• 10:49
  Christine: Mm-hmm.
• 10:49
  Brandon: Right[UM] there's a big consensus these days around harness engineering [UH] and you know we also talked about it last time.
• 10:57
  Christine: Mm-hmm.
• 10:58
  Brandon: But
• 10:58
  Christine: Mm.
• 10:58
  Brandon: people when they talk about harness engineering they're talking about the harness at the scope of their project. what matters
• 11:04
  Christine: Mm-hmm
• 11:05
  Brandon: is specializing the harness.
• 11:07
  Christine: Mm-hmm
• 11:07
  Brandon: To your work.
• 11:08
  Brandon: To your task.
• 11:09
  Christine: Mm-hmm. Just the way how Dax was describing, if you specify it, if you adapt it to that specific benchmark, it will perform way better.
• 11:19
  Christine: So, why don't we do that for our specific tasks?
• 11:22
  Christine: And I think that definitely makes sense. I can imagine how this could apply in our own workflows.
• 11:28
  Brandon: Mm-hmm.
• 11:29
  Christine: Like, a lot of[UH] projects that we probably a of people work on like a single project can contain very different kinds of work.
• 11:37
  Christine: Maybe there's[throatclearing] a part that's[UM] front-end implementation, there's back-end architecture, there's performance optimization, maybe there's three-D modeling or data migrations.
• 11:49
  Christine: And why should all of them use the same harness, right
• 11:52
  Brandon: Exactly. And I I know if you're like me maybe you've come across these kind of one-off instances through your work and I'll show a couple of them[UM] but it wasn't until this week that I realized you know actually what we should be doing is thinking about harness engineering holistically at these finer grained levels. so in order to maybe help you understand what we're saying, we
• 12:18
  Christine: Yeah.
• 12:18
  Brandon: can pull up some examples. Yeah.
• 12:19
  Christine: Yes, please
• 12:20
  Brandon: OK, so I'm gonna I'm gonna start with a solution to a problem that I've seen people talk about for months. I responded to a tweet even yesterday about this and it's something that you can solve in different ways depending on the kind of work that you're doing. So, what is the problem?
• 12:37
  Brandon: When I want to run multiple
• 12:41
  Christine: Mm-hmm.
• 12:41
  Brandon: sub-agents or agents in parallel that do the same work.
• 12:45
  Brandon: What do I do if that same work is dependent on a shared resource so to be more specific, let's start with iOS I want
• 12:55
  Christine: Yeah.
• 12:55
  Brandon: multiple agents to be testing in an iOS simulator and doing builds at the same time. Well, by default[laughter] that doesn't work because[UH] by default these things are they they conflict with each other. And so so this was a project I did in January in my in my harness I have something that that says that has a configuration[UM]
• 13:22
  Christine: Sorry,
• 13:23
  Brandon: per agent.
• 13:23
  Christine: f- just for the listeners
• 13:24
  Brandon: Mm-hmm.
• 13:24
  Christine: who are not, who don't, who don't, who are listening on Spotify and don't have video, what are
• 13:29
  Brandon: And
• 13:29
  Christine: you
• 13:29
  Brandon: good
• 13:29
  Christine: showing
• 13:29
  Brandon: idea.
• 13:29
  Christine: exactly on the screen?
• 13:30
  Brandon: Yeah, sorry about that so what I've pulled up is a terminal window and I've opened a an iOS project that I was working on. And within that iOS project I'm gonna open up in agents dot MD and just show you at the top of here it says Agent One simulator configuration and then there's a bunch of stuff. And there's a lot of these like hard-coded things in it[UM] but what this is what this is saying is
• 13:55
  Christine: Yeah.
• 13:55
  Brandon: it's giving the agent
• 13:56
  Christine: Yeah,
• 13:56
  Brandon: some specific commands that
• 13:58
  Christine: yeah.
• 13:58
  Brandon: it can run to have
• 13:59
  Christine: yeah.
• 13:59
  Brandon: a conflict-free
• 14:00
  Christine: yeah.
• 14:00
  Brandon: access
• 14:01
  Christine: Yeah.
• 14:01
  Brandon: to simulators. And the way that I set that up
• 14:04
  Christine: Yeah.
• 14:04
  Brandon: is I have a
• 14:05
  Christine: Yeah.
• 14:05
  Brandon: script which
• 14:05
  Christine: Yeah.
• 14:05
  Brandon: I've stored in the scripts folder that is
• 14:08
  Christine: Yeah.
• 14:08
  Brandon: run by every agent to get a unique
• 14:12
  Christine: Yeah.
• 14:12
  Brandon: environment. And when you do that then you're able to run multiple iOS simulators at the same time. So this is an example of how I've set up the harness.
• 14:21
  Brandon: Yes, I've put some stuff in agents dot MD, but I've also created tools to help me isolate my environment so that I can have multiple agents running in parallel to build my iOS app.
• 14:30
  Brandon: [laughter] with my harness
• 14:32
  Christine: Hey.
• 14:32
  Brandon: engineering hat on
• 14:33
  Christine: [laughter]
• 14:34
  Brandon: [UH] I would and if skills existed at the time that I did this I would put this into a skill so
• 14:41
  Christine: Right.
• 14:41
  Brandon: that it doesn't have to pollute the context and waste attention for agents that are not using the simulator
• 14:48
  Christine: Gotcha.
• 14:49
  Brandon: Yeah, I think this example's interesting because a lot of examples are just about context narrowing. So it's like turning off the right tools, turning off certain guides. But the harness that you're building up through harness engineering, it's not just about what goes or doesn't go in in context, but it's about the whole environment.
• 15:09
  Brandon: And in this case, this is about configuring the execution environment.
• 15:13
  Brandon: It's setting up the execution environment to be able to run in parallel.
• 15:16
  Christine: Gotcha. So [UM] I guess this example is more to show what can we set up in that environment to really help this agent to succeed in this specific task.
• 15:26
  Christine: OK. Now I get it.
• 15:26
  Christine: [laughter]
• 15:27
  Brandon: Exactly. Exactly. And I'm gonna just show I can quickly show an example of this same thing on the web. And I only this is exactly the same thing but for a web project Cuz I keep seeing people struggling with this[laughter] if you want to have concurrent sessions with let's
• 15:43
  Christine: Mm-hmm.
• 15:44
  Brandon: say like the Chrome DevTools MCP. So what I've done is I've shared the docs from Chrome DevTools.
• 15:49
  Brandon: There's a section on concurrent sessions. There's a certain amount of flags that you pass in and there's some instructions here. You can feed this to your agent and ask your agent to set you up a harness to set up your environment so that it behaves in this way.
• 16:02
  Christine: Oh, cool.
• 16:04
  Brandon: So
• 16:04
  Christine: Yeah, we will add the link [UM] in the description too, so people who are only listening can find it too and watch later if they want.
• 16:11
  Brandon: Yeah So yeah, so that's, I think. This is an interesting example that came up in the past. For me, that doesn't have to do with context narrowing But context narrowing is a big part of it.
• 16:20
  Christine: Mm-hmm.
• 16:20
  Brandon: So[UH] And we even have a whole section on context narrowing later [laughter]
• 16:27
  Christine: Yeah, this is a very [UM] interesting example where you.
• 16:32
  Christine: Yeah, I like it. I[UM] it's a good example that shows how you can give an agent [UM] the right tools to, specific to that task.
• 16:41
  Christine: I'm wondering [UM] because we when
• 16:43
  Brandon: Yeah,
• 16:43
  Christine: we had our conversation previously, we also had a few other like general examples.
• 16:49
  Christine: Maybe we can show those now too and [laughter] for folks who are only listening, maybe we can just [UM] voice over them just to give people a little bit of [UM][UM] [UH] give give them more sense of how how this could be applied for different kind of use cases
• 17:06
  Brandon: So, what I've done is I've shared a diagram. And this diagram says, front-end implementer environment. What this agent needs in order to build UI well. So this is for tasks that involve the front-end The kinds of things that you're thinking about when you're designing your harness, when you're and you're engineering your harness and you're iterating on your harness, for this specific focus, you wanna understand what the goal is and then you wanna think, OK, what is the context here? What are the things that I need to specify in in files and pull them in In this case, design systems, reference screenshots for example[UM] what are the tools that I need? What are the tools that this harness needs to be able to control? Maybe if you're doing front-end web you need browser tools, you need a way to take screenshots you need a way to write code For and then there's automated checks. These are like your sensors that run in a hook or as a result of certain commands. So for example you might wanna do a visual diff for an accessibility check or you know integration tests.
• 18:08
  Brandon: And then and then you wanna think carefully about your definition of done. The criteria that the agent can that the agent can use to check that it's actually done with this work. So that's
• 18:20
  Christine: Mm-hmm
• 18:21
  Brandon: you can der- derive that from the goal.
• 18:23
  Brandon: For example in this case it's like making sure that there aren't any accessibility issues that the the reference screenshot that you passed in is the same as the screenshot on the web.
• 18:34
  Brandon: And if you specify that, you can add feedback loops into your harness.
• 18:38
  Brandon: That for example, you can say, you know, keep going until the screenshot is in place.
• 18:43
  Brandon: Right? So.
• 18:44
  Christine: Yeah?
• 18:44
  Brandon: So this is this is like a simple front-end example. I'll just really more quickly talk through a few more so
• 18:52
  Christine: Yeah, how would the environment for other kinds of agents look like?
• 18:56
  Brandon: Yeah, so here's for maybe a performance agent. so I'm sharing a new diagram here you know, it's helpful to think in these in these categories. So what's the goal what context do we need, tools, feedback, exit criteria. I'm not gonna talk through all of these specifics, but just I'll pull a little handful context might be a baseline measurement of how fast something is, tools you might want a profiler the feedback might be the benchmark results and the exit criteria is that the benchmark result hits a certain point and then I'll just pull up one more so this is a debugger agent. The goal might be finding the root cause of a bug. Context might be things that you've seen, things that you've tried repro steps[UM] The tools, you might want a a debugger. This is like similar to the thing I talked about last time where I built a custom debugger for debugging my Game Boy projects[laughter]
• 19:57
  Christine: Mm-hmm.
• 19:58
  Brandon: Yeah, and then and so on.
• 19:59
  Brandon: So.
• 20:00
  Christine: This is very helpful.
• 20:00
  Christine: This really [UM] [UH] helps me also see, like, when you mentioned, oh, build a world for your agent specific for to that task, how that could look like[UM] it also just just a a a quick side note, it also emphasize or shows me how a harness is much more than just a skill[UM] like skills are very useful, but a skill is usually just a additive.
• 20:25
  Brandon: Yeah.
• 20:26
  Christine: It gives the agent instructions, knowledge, examples or maybe tools, but the harness environment that you just described and showed is much broader. It can add capabilities but[UM] basically it can also remove tools, restrict permissions[UM] or narrow available contexts.
• 20:44
  Christine: So I guess maybe that brings us to is more capability for the agents always better?
• 20:52
  Christine: so we we talked about what what should the agents have in their environments but also what should they not have.
• 20:59
  Christine: And the reason why I ask is is because I got inspired by Tariq's post.
• 21:02
  Christine: Maybe we can pull that up. Yes. So Tariq like I think it was a week ago when he sent out this [UM] post where he told
• 21:13
  Brandon: Yeah.
• 21:13
  Christine: Claude Code users that he simplified the Claude MD file and I think he cut away like eighty percent of it but we this does not only need to be applied to context engineering you could also think about this from the perspective of harness engineering. Right?
• 21:34
  Christine: What do you think
• 21:36
  Brandon: I think this is a really interesting an interesting thing to think about. So removing parts of your Markdown files is one thing. But [UM] let me pull up another tweet that was inspiring
• 21:49
  Christine: Mm-hmm
• 21:50
  Brandon: that inspired interesting thoughts. So this is a tweet by Matt Pocock the creator of famous skills like grill me, et cetera[UM] [laughter]
• 22:01
  Christine: Not
• 22:01
  Brandon: which
• 22:01
  Christine: not real
• 22:02
  Brandon: not
• 22:02
  Christine: me [laughter]
• 22:03
  Brandon: drill me, no [UM] so [UH] so Matt Matt posted this tweet the other day and[UH] I think the important part is here, the specs my skills create are intended to be deleted immediately, not kept around or treated as source code and he further clarified in a response and there was some discussion on this that this is because it poisons the context of your agent.
• 22:30
  Christine: Mm-hmm
• 22:30
  Brandon: Because so the spec is used to derive the code the first time but as your agent starts iterating on the code as you make changes when you're reviewing as you add more features the code and the spec starts to drift. And you can try to keep them together but but it's tough and and it's not only about being careful not to put that context automatically in your agent's context[laughter] but even having it in your repo in your folder is enough to poison the mind of the agents.
• 23:01
  Brandon: So this is actually a real problem that I've been experiencing [laughter] and
• 23:05
  Christine: Mm-hmm
• 23:05
  Brandon: and [UH] and didn't realize that I should do something about it until this week I have been checking in my my specs and frequently in the reasoning of my agents when I'm working with my agents I see something like, oh I thought the spec says this but when I went in the code and I tried to change it I noticed that actually the code says this other thing so I'll just forget what the spec says. That reasoning trace is polluting the mind of the agent.
• 23:32
  Brandon: It's distracting it from its work.
• 23:36
  Brandon: And so actually thinking carefully about how accessible is that information?
• 23:41
  Brandon: Even if it's not there, if it's accessible by the tools that the agent uses to search around, then you're still at risk of getting your contacts poisoned[UM] and I I think there's one other one other example of this. I mean,
• 23:54
  Christine: Yeah.
• 23:54
  Brandon: the same sort of thing but so I use Beads Beads Rust which is a tool for task management. At some point we'll probably talk about this in more detail in an episode but the same thing. I had a conversation with a friend the other day and he was saying, oh how can I how can I use Beads but not check in the the Beads? And I'm like, hey that's the point. Like you wanna check it in because you you know that information's useful you can do a sort of all sorts of analysis on your data and and these kinds of things but but it's the same problem. If it's available and it's a spec basically it's like a description of a task not the actual result of the task then that can poison the content of your agent.
• 24:34
  Brandon: And that that then leads to this idea of OK well I should have two separate harnesses two separate environments. In one environment I have access to all the history beads, all the historical stacks.
• 24:45
  Brandon: Hopefully [UH] like ideally the reasoning traces of all my chat history. And I can do retros in that environment [UM] and in my environment where I'm actually implementing my work, I need to get rid of all that stuff. It needs to not even be available[UH] and accessible to the agent. This is something that I I only realized this week. So
• 25:05
  Christine: [laughter] [UM] OK, no, that's super helpful.
• 25:09
  Christine: I I could see I could see the context pollution [UM] point[UM] and also how we can how removing them can improve the work itself.
• 25:20
  Christine: Cuz basically, agents have finite context, they have finite attention, they have finite tokens and finite compute. So every irrelevant spec or document loaded into the context will compete with something that is relevant.
• 25:34
  Christine: So and also like every unnecessary tool that we introduce will also add another possible direction.
• 25:40
  Christine: So this every generic instruction the agent must carry through the trajectory will consume attention that could have gone towards solving the actual problem.
• 25:51
  Brandon: Yeah. and you might think, OK OK, I'll I'll I'll keep my specs separate, you know?
• 25:55
  Christine: Mm-hmm.
• 25:56
  Brandon: But it's not just specs, it's code too I wanna show another tweet that came up recently by by Victor Talin the creator of the BEND system, a prolific tweeter
• 26:08
  Christine: [laughter]
• 26:09
  Brandon: [UH] so Talin noticed he has long tweets so let me highlight the the relevant part [UM] He he was begging
• 26:17
  Christine: Mm-hmm.
• 26:17
  Brandon: the AI to remove and re-derive the work.
• 26:19
  Brandon: But notice that this does not work. It has a massive bias to keeping what is there [UM] so he he talks about how he spent so long trying to get Fable to just not [laughter] not keep what's there and and he couldn't.
• 26:32
  Brandon: And then he realized he can't keep something he doesn't see. So what he did is he he adjusted his harness to remove the parts of the code that he wanted Fable to re-derive.
• 26:45
  Brandon: And[UM] and only include a description of what it's meant to do.
• 26:49
  Brandon: And that and that fixed it.
• 26:50
  Christine: Interesting. Do you have any examples, like when would you want to hide parts of the code? Like what parts of the code would you like to
• 26:56
  Brandon: Yeah.
• 26:56
  Christine: hide?
• 26:57
  Brandon: in this particular case[UM] so Victor's working on a a core kernel for his new proof checker in programming language.
• 27:09
  Brandon: And and he's really conscious to keep the code and the complexity as small as possible, cuz the problem is already complex
• 27:17
  Christine: Mm-hmm.
• 27:17
  Brandon: and also when the when the code is simpler it's easier to optimize. So[UM] so so he had he has a part of that code that he felt is too messy and he wants it to be cleaner and so so I guess like in general if you're refactoring if you have some part of your system where the code is messy and that messiness is making it hard to understand both for you and for your agents it's making it hard for new features to get added in it's making it hard to optimize this is a technique, a practice that you can do to to convince your agents to rewrite it. It's just get
• 27:52
  Christine: [laughter]
• 27:53
  Brandon: rid of it [laughter]
• 27:54
  Christine: [laughter]
• 27:55
  Brandon: and only include a s- like a placeholder rather than having it there and telling the agent that you need to to redo it more
• 28:01
  Christine: mhm.
• 28:01
  Brandon: thing, one more
• 28:02
  Christine: Yeah.
• 28:02
  Brandon: thing, but before we move off of this so we talked about, like, specs and plans, how that can pollute your, your environment if it's included in the wrong places. We talked about how code in the wrong places can ruin your environment for certain tasks
• 28:16
  Christine: Mm-hmm.
• 28:17
  Brandon: but even sensors,
• 28:18
  Christine: Mm-hmm.
• 28:19
  Brandon: even like tools that run automatically, tests that run automatically can pollute your environment[UM] and how is this?
• 28:25
  Brandon: Well, every unnecessary sensor slows
• 28:29
  Christine: Mm-hmm.
• 28:30
  Brandon: down your agent. And
• 28:31
  Christine: Yeah
• 28:31
  Brandon: as as our agents are getting faster, as we're willing to accept dumber models that run faster, we are controlled by the grips of of Amdahl's law.
• 28:46
  Brandon: OK?
• 28:46
  Christine: [laughter]
• 28:47
  Brandon: So I found the article.
• 28:47
  Brandon: It was not a tweet, it was an article. so this was on the front page of Lobsters today[UH] it's a a post by Martin Alderson.
• 28:55
  Brandon: The title is I'm in parentheses mostly picking models on speed now, not intelligence.
• 29:02
  Brandon: So the point, I guess the whole point of his article is that he is reaching for dumber models that run faster. But here's the interesting part. There's a nice diagram that he built which is an illustration of, of Amdahl's law let's talk about this diagram first and then I'll explain Amdahl's law.
• 29:19
  Christine: Mm-hmm.
• 29:19
  Brandon: OK? So So what is this diagram of on the bottom, on the X-axis, you see seconds per agent turn.
• 29:26
  Christine: Mm-hmm.
• 29:26
  Brandon: And the Y is just a bar chart, so there's two bars
• 29:30
  Christine: Mm-hmm
• 29:31
  Brandon: At the top is a bar that's fifty tokens per second.
• 29:34
  Brandon: And you see this this bar chart, it's broken up into sub-bars At the bottom is inference, and then tool tool calls, and then you[UM] So inference is the time that it takes to do inference, tool calls is the time it takes to wait for a tool to run and then get a result back.
• 29:51
  Brandon: And then U is the time it takes your brain to think about what happened and then decide what to do next.
• 29:59
  Brandon: And [UM] the the interesting thing here that's counterintuitive which is a consequence of what is called Amdahl's is if you if you compare the bar chart at the top, the fifty tokens per second one,
• 30:10
  Christine: Mm-hmm.
• 30:10
  Brandon: and the bar chart at
• 30:11
  Christine: Mm-hmm.
• 30:11
  Brandon: the bottom,
• 30:11
  Christine: Mm. Mm-hmm
• 30:11
  Brandon: the two fifty tokens per
• 30:12
  Christine: Mm-hmm.
• 30:12
  Brandon: second one
• 30:13
  Christine: Mm-hmm.
• 30:13
  Brandon: [UM]
• 30:13
  Christine: Mm-hmm
• 30:14
  Brandon: this is a rate of inference speed.
• 30:17
  Brandon: And so in this diagram the author shows inference take forty seconds at the top and eight seconds on the bottom. And that's a five X speedup
• 30:25
  Christine: Mm-hmm
• 30:25
  Brandon: in the model speed.
• 30:27
  Brandon: But
• 30:27
  Christine: Mm-hmm
• 30:28
  Brandon: the tool call speed and how fast your brain thinks does not change.
• 30:32
  Brandon: And so on net, your agent only speeds up by two X.
• 30:37
  Brandon: So a five X increase in speed in tokens per second is only a two X increase in your agency. And this
• 30:44
  Christine: Mm-hmm
• 30:44
  Brandon: is Amdahl's law in action. So Amdahl's law I'm not gonna explain it in detail but essentially it's [UM] it relates the how fast your speed up of some section of your work is to how fast you can realize that speed up in the whole system. Because basically
• 31:04
  Christine: Right.
• 31:04
  Brandon: if you have some fixed amount of time for other parts, that's
• 31:07
  Christine: Mm-hmm
• 31:07
  Brandon: staying the same, but it's becoming a larger percentage of the total time.
• 31:11
  Christine: Mm-hmm
• 31:12
  Brandon: So[laughter] so what's the point of this? How is it relevant to what we're we've been talking about?
• 31:17
  Christine: Yeah,
• 31:17
  Brandon: Well,
• 31:17
  Christine: because we were talking about having a lot
• 31:19
  Brandon: Mm-hmm.
• 31:19
  Christine: of tests,
• 31:21
  Brandon: Yes. That's exactly how it's relevant. so so so yeah, as the models are getting faster, as you're willing to pick up faster models, if you actually want your agent to be faster, you have to make sure to shrink the tool call time as well. so and that's not just getting your your your tools to be faster, but it's choosing which tools you need at any specific point. That's the point.
• 31:42
  Christine: Mm-hmm.
• 31:43
  Christine: So that so you're so[UM] going back to how we started talking about this topic, about the [laughter] Amdahl's law. when there are so many tests in your own projects that have been accumulating over time while you were building it, you shouldn't keep all of them. Is that what you're or
• 32:00
  Brandon: Exactly.
• 32:00
  Christine: you shouldn't
• 32:01
  Brandon: I mean
• 32:01
  Christine: run
• 32:01
  Brandon: this
• 32:01
  Christine: you
• 32:01
  Brandon: is.
• 32:01
  Christine: shouldn't run them all the time for every
• 32:03
  Brandon: Yeah.
• 32:04
  Christine: agent
• 32:04
  Brandon: So this is and this is just one kind of sensor, this is one kind of feedback that you have.
• 32:08
  Brandon: But let's use this as an example to ground ourselves
• 32:12
  Christine: Mm-hmm.
• 32:13
  Brandon: yes.
• 32:14
  Brandon: what's happened to me several times in my projects is I have my agents write tests as they go and I have all the tests run automatically after each task so that there's no regressions
• 32:25
  Christine: Mm-hmm.
• 32:25
  Brandon: but what happens is the the test time starts building up and up and up and it's slow
• 32:29
  Christine: Yeah.
• 32:30
  Brandon: and and this is a problem that people might have run into in the olden days before agents as well it's just more acute now because the tests are running more often and so it's important to to take a step back and saying, OK, which test do I actually need for this part of my work? Which test can I run in parallel which tests are actually duplicates of each other [UH] so it's worth like going back and and that's part of the effort of like tending to your harness engineering garbage that you should be doing all the time.
• 32:57
  Brandon: And I
• 32:58
  Christine: So,
• 32:58
  Brandon: I
• 32:58
  Christine: mhm.
• 32:59
  Brandon: guess in the context of our conversation now, you should think about these environments distinctly even within a project.
• 33:07
  Christine: Mm-hmm.
• 33:08
  Christine: could you actually implement that in practice in your workflow? How can you [UM] for example in my current workflows I also have like almost four thousand tests.
• 33:18
  Christine: How can I tell my agents what is the best way to tell my agents how to group them or filter them?
• 33:23
  Christine: And how do I even like hide certain files from some agents
• 33:29
  Brandon: So for the for the first part, for for grouping tests[UM] that's that's just a prompt. So open a fresh prompt and in a fresh harness you know [UM] tell your tell your agent, OK look through my tests[UH] benchmark them, tell me if there's any that are duplicated, parallelized where you can.
• 33:52
  Brandon: And and you can just work with your agent and and fix that[UM] because that's applicable for everything.
• 33:58
  Brandon: So that's that's one step that's easy.
• 34:01
  Christine: Mm-hmm.
• 34:03
  Brandon: The second part, actually removing them.
• 34:05
  Brandon: This is there's a few ways I can think of to do this. It's not something I've done yet myself since
• 34:13
  Christine: Oh, wait.
• 34:13
  Brandon: we
• 34:13
  Christine: Before
• 34:13
  Brandon: just
• 34:13
  Christine: before
• 34:13
  Brandon: discovered it. Mm-hmm
• 34:15
  Christine: removing them [UM] when[UM] when we ask the agent to help us categorize or filter the tests.
• 34:23
  Christine: Probably it's not only the duplicates, right?
• 34:26
  Christine: Because for example, some tests are more tests that touches front-end, some others tech- touches the back-end architecture.
• 34:33
  Christine: So how would you how would you tell the agent to make a distinction there
• 34:37
  Brandon: Yeah.
• 34:38
  Brandon: It's funny, this is actually something that I spent a lot of my engineering career solving in different companies [UH] and there's lots of different ways to do it.
• 34:49
  Brandon: And there's different like levels of complexity you can go in.
• 34:52
  Brandon: So[UM] so there's so like we said this is a this is a problem that has existed for a while and so there's there's solutions[UM] so So the a simple thing you can do is for each test you can assign which part of the code base it's associated with[UM] and so in somewhere where
• 35:15
  Christine: Yeah.
• 35:15
  Brandon: you're registering tests in in your strip that's running the test. You can work with your agent and say, OK, you know, match this prefix[UH] for these tests.
• 35:24
  Brandon: And if that prefix doesn't match of of like files or folders that were touched in this change, then you do don't have to run those tests.
• 35:31
  Brandon: So that's that's a simple thing that you can do that your agent can probably one-shot[UM] Does that make sense? So, let let me be specific.
• 35:38
  Brandon: You could say [UM] if you have a front-end directory and a back-end directory,
• 35:42
  Christine: Mm-hmm.
• 35:43
  Brandon: in your in your test runner, your agent would change it for you to register[UH] tests that say like, OK, these are all the tests that run if a file is touched under the front-end folder. And these are all the tests that run if a file is touched under the back-end folder. And if you touch a file in both the front-end and back-end folder, then all those tests are gonna be run.
• 36:02
  Brandon: And that's exactly
• 36:02
  Christine: Mm-hmm.
• 36:02
  Brandon: what you want
• 36:03
  Christine: And you you
• 36:04
  Brandon: [UM]
• 36:04
  Christine: ask the agents to make the decision which test needs to be run when[UH] code in a certain folder is touched.
• 36:15
  Brandon: Yeah. So you can you can work with your agent and and decide, OK, these are tests that are important for front-end and these are tests that are important for back-end. Then you tell your agent, OK, label them and put it into my test runner so that they run automatically in those situations.
• 36:28
  Brandon: So that's that's a really simple thing you can do.
• 36:31
  Brandon: There's a problem with that approach though, which is it's it's something that you assign that you make a judgment on
• 36:39
  Christine: Mm-hmm.
• 36:39
  Brandon: and your judgment could be wrong [UH]
• 36:41
  Christine: [laughter]
• 36:42
  Brandon: or
• 36:42
  Christine: Mm-hmm.
• 36:42
  Brandon: it or it can go out of date.
• 36:44
  Brandon: And so
• 36:44
  Christine: Mm-hmm.
• 36:45
  Brandon: [UH] so so that that's where this breaks down. So the other way to do it, which is much more effort, but
• 36:51
  Christine: Mm-hmm
• 36:51
  Brandon: there's a bunch of test test runners that[UH] go the other way around where[UM] where the your whole build and test system is built around
• 37:05
  Christine: Mm-hmm.
• 37:06
  Brandon: the idea
• 37:06
  Christine: Mm-hmm.
• 37:06
  Brandon: of understanding
• 37:07
  Christine: Mm-hmm
• 37:08
  Brandon: the the boundaries and the surface area of all of your code files.
• 37:12
  Brandon: And so when your build system understands that, it natively knows which which tests are associated with which modules. And this is something that[UH] there's tools like Bazel or its derivatives, Buck[UM] these things are sort of you have to buy buy into them opt into it [UM] if you use to set up your project that's another tool to look into but these are very heavy weight and you'll have to invest a lot of energy in or you'll have to change the structure of your project
• 37:48
  Christine: Mm-hmm.
• 37:48
  Brandon: agents can do that for you so it's not that much energy[UM] but but[UH] it's also not always possible if you have if
• 37:55
  Christine: Mm-hmm.
• 37:55
  Brandon: you have in your project code that has a lot of different programming languages and you're doing lots of physical custom things, it can be really hard to fit them into these frameworks.
• 38:08
  Brandon: So.
• 38:09
  Christine: It sounds like it might be better if you used a second method, that you just build start building the project in that way instead of trying to[UM] implement those later
• 38:19
  Brandon: Yeah. Yeah. Yeah, that's something[UM] that's something you can do up front [UH] and with the agents it's not even that hard to change in the
• 38:27
  Christine: [laughter]
• 38:27
  Brandon: middle [laughter] to be honest [UM] but it's
• 38:30
  Christine: I'm just
• 38:30
  Brandon: [UH]
• 38:30
  Christine: worried about mistakes or,
• 38:32
  Brandon: I
• 38:32
  Christine: you
• 38:32
  Brandon: would.
• 38:32
  Christine: know, misses and gaps
• 38:34
  Brandon: Yeah, I the thing that I would that I would discuss with your agent when you're trying to dig into this[UM] if you're listening and and you wanna play with it[UM] just talk to your agent about the trade-offs of of these different approaches[UM] because yeah, you'll you'll be limiting what is possible[UM] when you have your your tests set up that way, where it's
• 38:58
  Christine: Mm-hmm.
• 38:58
  Brandon: like all tracked by the build system tracks it. But the benefit is it's perfect. Like,
• 39:04
  Christine: [laughter]
• 39:04
  Brandon: if you change this line of code, it knows exactly what tests are associated with it and nothing else will run.
• 39:11
  Christine: Awesome.
• 39:11
  Brandon: So, yeah, so so that's how to do th- that that side of it.
• 39:16
  Christine: OK,
• 39:16
  Brandon: And then,
• 39:17
  Christine: I I could [UM]
• 39:18
  Brandon: m-
• 39:18
  Christine: also imagine if [UH] the second option, I don't know, is is not an option for some people and they go with option number one.
• 39:26
  Christine: You could also add a retro step and run a loop to [UH] make sure that any test that you have missed the first time will be captured afterwards.
• 39:35
  Brandon: Yes.
• 39:36
  Brandon: Yeah. Yeah, yeah. Yeah, you can have a, you can have a review agent[UM] run periodically to make sure that things are still aligned.
• 39:44
  Christine: Mm-hmm.
• 39:45
  Christine: Awesome.
• 39:46
  Brandon: Yeah [UM] and I guess, and then there's one more piece, which is, you know, we we talked about actually making it impossible for the harness to see the code
• 39:56
  Christine: Yeah.
• 39:57
  Brandon: or the files or the context.
• 39:59
  Brandon: How can we do that[UM][laughter] I don't have a great solution[UM] I I think[UH] I think to do this, so to to do it kind of in a in an ad-hoc way is easy, right? You just move the file[laughter] You
• 40:19
  Christine: [UM]
• 40:19
  Brandon: know [UM] and
• 40:20
  Christine: [UH]
• 40:20
  Brandon: and and that's that can be enough a lot of the time. I mean, the example I showed with with Victor with Victor Taelin, he he just deleted that part of the code and replaced it with a comment.
• 40:32
  Brandon: And and
• 40:32
  Christine: Mm-hmm.
• 40:32
  Brandon: and that can be enough
• 40:33
  Christine: [laughter]
• 40:33
  Brandon: [UM] I think there's an opportunity to build some kind of tooling around automatically[UM] showing and hiding or exposing and and removing from your s- execution environment, your sandbox[UM] the stacks and and plans and tasks, issue tracker, that that kind of thing[UH] I don't know of a tool that exists for that. So for now, what I've been doing so far is just manually moving files around
• 41:04
  Christine: Mm-hmm. But if you're[UM] like in our examples that we mentioned during this episode we we for
• 41:11
  Brandon: Mm-hmm.
• 41:11
  Christine: example we talked about front-end agents and agents that work on maybe the back-end architecture.
• 41:17
  Brandon: Yeah.
• 41:18
  Christine: I could imagine that[UM] you want to give the front-end agents maybe the style guide
• 41:24
  Brandon: Mm-hmm
• 41:24
  Christine: [UM] but when the back-end agent is working you might not want to remove the file, right? You you do want to like [UM] you don't want it to pollute the back-end agent's contacts, but you don't want to remove it and then hopefully you won't forget to re-add it again when you tell your front-end
• 41:41
  Brandon: Well, yeah.
• 41:41
  Christine: agent to do work.
• 41:43
  Brandon: This is the, I mean, this is the thing to set up. It is like, we saw some examples where you, you do need to remove it. Cuz even it existing is gonna distract your agents because they'll, they'll pick it up when they're searching for files. So[UM] you know, more concretely, like a simple way to do that, if you're already using worktrees[UH] so if you use Claude or Codex, it's just a check box [laughter] when you, when you start a session[UM] if you're using the the app[UM] [UH] the you'll have a fresh work tree and you can just delete the files and [UH] and then you have to remember or you tell your agent like, hey don't commit these[UM]
• 42:20
  Christine: Mm-hmm.
• 42:20
  Brandon: But yeah, even then they're they're sort of visible. So I think there's I think there's an opportunity to to to build tooling to automate this more[UM] But I, yeah.
• 42:35
  Christine: So even if you spawn a sub-agent and you specify exactly what what files should be shared with the sub-agent, you think maybe since they have [UM] access to the files, like what Taelin said, Victor Taelin said[UM] they they might still access it in some way.
• 42:55
  Brandon: I guess[UH] in Victor's case it was in the same file in [UM] in the example with Matt Pocock.
• 43:03
  Brandon: The
• 43:03
  Christine: Mm-hmm.
• 43:03
  Brandon: specs were just living in different directories and I've seen it. I've seen it happen. The agent, when you when you tell it change the start of the l- code, it looks and finds a spec that describes it and then it loads into context even if you didn't tell it to.
• 43:16
  Christine: Mm-hmm.
• 43:16
  Brandon: And so, yeah, it existing is is a problem. I think [UM] I think you can probably.
• 43:23
  Brandon: You know, you can hack your way around it for now. You can just delete delete the files and maybe add something in your prompt that says like don't look at them.
• 43:31
  Brandon: But but that's that's not a good solution
• 43:34
  Christine: Yeah, it's it's a good [UH] an interesting insight and practice to think about though [UM] because sometimes, you know, after building it maybe you don't need the spec anymore and and then the way how you described it, move it to a different directory that only the retro agent or some other agent who really needs it can access it that that makes sense.
• 43:52
  Brandon: Duh
• 43:52
  Christine: Yeah, definitely [UH] worth thinking about even though, you know, there's not not a mature product or tool built yet.
• 44:13
  Christine: Is limiting the blast radius.
• 44:16
  Christine: That could also be another reason why to narrow the scope or capabilities of your agent, right? Could you maybe, yeah.
• 44:24
  Brandon: Yeah, yeah. [UM] when we say blast radius, we
• 44:27
  Christine: [laughter]
• 44:27
  Brandon: mean like when we have an ask, how much of the world gets[laughter] gets destroyed by the blast of the bomb that's sort of the metaphor. So like how how many files are touched, how much stuff is changed.
• 44:39
  Brandon: And so yeah, it's good practice to limit the blast radius of your actions when you're using multiple agents in parallel, or just in general because you have less change to review and there's less chance of something getting away.
• 44:49
  Brandon: And so the best way to limit your blast radius is to remove the parts of your environment from the world and then the bomb cannot touch them because they're not even there.
• 45:00
  Christine: Right, right. So for example, if [UM] if I was building an agent who would check my CRM to decide which clients I have to contact or reach out to, then give it only read access. Don't give it well[laughter][UM] Yeah, only give it read access. Don't let it edit the database.
• 45:20
  Brandon: yeah, this is I mean, that's an example of a of like the danger [laughter]
• 45:24
  Christine: Mm-hmm.
• 45:25
  Brandon: the danger argument, which is also something, I mean, we haven't really touched on that it is something that a lot of people talk about I suppose, but it's another part of the harness so yeah, as you're saying, like we've seen people accidentally lose their database delete their hard drive.
• 45:42
  Brandon: These are examples of a harness that's not carefully guarded
• 45:47
  Christine: Yeah. And maybe, you know, don't even give them the tools that they don't need, like don't give that CRM agent the tool to send out emails if you only want them to propose which people to email
• 45:58
  Brandon: Yeah.
• 45:59
  Brandon: Exactly
• 45:59
  Christine: Cool. So yeah, I guess it's [UM] a narrowing, so I guess when designing a harness it's good to ask yourself what information does this agent genuinely need?
• 46:11
  Christine: What tools will really help it complete its task?
• 46:15
  Christine: What feedback or tests should it run automatically? What should it never be able to do?
• 46:21
  Christine: And then [UM] the goal of this all is not to create the most powerful possible agent that can do everything, but it's to create the agent that most likely can succeed at a very specific task.
• 46:35
  Christine: Awesome.
• 46:37
  Brandon: Perfect
• 46:37
  Christine: OK. [laughter] Cool.
• 46:37
  Christine: Then maybe this is a good point to also move to our[UM] one of the last things that we also want to touch on and it's also very important and interesting to touch on.
• 46:48
  Christine: We talked about how narrowing the harness can help the agents make the agents work better but sometimes mistakes still happen.
• 46:58
  Christine: And there are probably also different ways of enforcements that we can apply.
• 47:05
  Christine: and that reminds me of something that I was trying to do with OpenClaw in January[UM] I was building this personal agent and I wanted to it to help me schedule my day based on my calendar and my priority list.
• 47:19
  Christine: And [UM] but when I first tried to build it with a simple prompt I I learned that it sometimes does not follow the instructions or missed things.
• 47:30
  Christine: And I remember that we talked back then as well and[UM] after [UM] chatting and discussing it, I realized there are actually different layers of reliability.
• 47:43
  Brandon: Yeah. Tell us, tell us about the layers.
• 47:45
  Christine: So there are four different enforcement levels for agents.
• 47:50
  Christine: That is, level one is prompting, level two is using templates, level three is using scripts, and level four is cron and heartbeat triggers.
• 48:02
  Christine: So, level one is natural language instructions.
• 48:07
  Christine: For example, wind down work by eight thirty PM.
• 48:10
  Christine: And this is fine for tone and communication style, but it's not reliable for anything where accuracy matters because the agent will only follow it follow it most of the time.
• 48:22
  Christine: And
• 48:22
  Brandon: Right,
• 48:22
  Christine: that's not enough
• 48:22
  Brandon: right.
• 48:23
  Brandon: Yeah.
• 48:24
  Brandon: OK, and then what about level two
• 48:26
  Christine: [UM] level two, using templates.
• 48:28
  Christine: So, you could force your agent to use a structure that he n- that it needs to fill in before acting.
• 48:35
  Christine: For example, I told my agents, check the calendar and fill out this section. And the section is calendar output and then blank section.
• 48:44
  Christine: And if that section is empty, then the plan does not get sent to me because then it means the agent did not do a good job at checking the the calendar before sending it to me.
• 48:54
  Christine: So that's a
• 48:55
  Brandon: Right.
• 48:55
  Christine: forcing function to[UM] make sure the the the failure becomes visible.
• 49:01
  Brandon: Right. And then the and then OpenClaw would get that feedback loop. If that makes sense. And then And then what about [UH] what about the next layer?
• 49:26
  Christine: So then the agent can read the output instead of interpreting raw API data.
• 49:32
  Christine: And a script cannot misread a meeting time or forget to check[UM] so that is another enforcement level that's even more reliable.
• 49:43
  Brandon: Great. And then what's the last one
• 49:52
  Christine: Fire on a schedule, then the agent does not need to remember at all to, for example, plan your day or ping you, send you a reminder at some[UH] [UH] at a certain hour of the day. The cron job will just fire and force it. And the heartbeat, you could also have a heartbeat that pulls every thirty minutes or something.
• 50:11
  Christine: And I found that[UM] after I[UM] used these different layers for different things that I wanted my personal agent to do, finally it started [laughter] to be reliable enough[UM] for the task that I gave it.
• 50:27
  Christine: And I could also imagine that [UM] these kind of enforcement levels could also be applied to harness engineering.
• 50:35
  Brandon: Yeah. I think so. The[UH] this like nomenclature I think is really interesting the there was a there was a post by John DeGoes this week. I'll just show it quickly.
• 50:48
  Brandon: Your old agent architecture is dead.
• 50:50
  Christine: about
• 50:50
  Brandon: I mean,
• 50:50
  Christine: that?
• 50:50
  Brandon: it's replacement John DeGoes is the CEO of Ziverge Tech he's the creator of ZIO, which is an inspiration to Effect and just in general like a functional programming guru and anyway this this post is interesting but in the very beginning he introduces this concept of of hope and and then how to fix it with enforcement. And I think that's like the best lens in which to look at what you were sharing regarding OpenClaw.
• 51:23
  Brandon: So.
• 51:23
  Christine: Cool. How would you describe the different enforcement levels when people are building harnesses
• 51:28
  Brandon: So [UH] so there's you
• 51:31
  Christine: and when
• 51:31
  Brandon: can think of it
• 51:32
  Christine: people are
• 51:32
  Brandon: like a sliding scale
• 51:33
  Christine: building
• 51:33
  Brandon: between
• 51:33
  Christine: harnesses
• 51:33
  Brandon: hope and enforced. So so at all the way on the hope level, it's all in markdown it's all context driven, it's all non-deterministic, it's all the what will happen will be generally what you want but maybe not exact, and depending on how how complex your ask is.
• 51:49
  Brandon: And this is what you are seeing with layer one so and then as each of those layers that you described is moving us more and more towards enforcement, towards determinism, towards[UH] software one point O
• 52:04
  Christine: a good
• 52:04
  Brandon: that does
• 52:04
  Christine: point.
• 52:04
  Brandon: exactly what we want and ultimately the best designed harnesses are doing as much as possible in the determinism side, essentially, with speculative evaluation, with branch prediction I'm just using metaphors here, but with also guidance on the hope side. So this is exactly what we talked about in the last episode But just briefly to say it again
• 52:29
  Christine: [laughter]
• 52:29
  Brandon: [UM] if you only have the determinism, if you only have your rules that yell at you if you do something wrong, then the agent is likely to do it wrong first and then fix itself [UM] if you only have the guidance in Agents dot MD, a lot of the time it'll do what you want the first time, and sometimes it'll fail. And so we
• 52:50
  Christine: Yeah.
• 52:50
  Brandon: we want the best of both. We want to most of the time do the correct thing, and in the cases where we make a mistake, we correct it so that covers us to the layer two, I think. And then that
• 53:03
  Christine: Yeah,
• 53:03
  Brandon: last
• 53:03
  Christine: I think
• 53:03
  Brandon: layer that you talked about,
• 53:04
  Christine: that's
• 53:05
  Brandon: the trigger,
• 53:05
  Christine: a
• 53:06
  Brandon: this is just another part of the harness that we haven't talked about yet so how the how the agent actually starts doing its work is is another thing that you can control either non-deterministically by an orchestrator or if possible deterministically with a cron for example that runs on a timer and automatically deterministically always does that task at a certain time.
• 53:32
  Christine: I definitely noticed when [UM] you go the deterministic route, your success rate is much higher[laughter] than staying on level one
• 53:42
  Brandon: And and this is this is exactly why context engineering is is not enough. And we need harness engineering.
• 53:49
  Brandon: Because if you're thinking in terms of context engineering, you're only in that hope world. you're[UM] you're changing your agent's dot MD. Maybe you're making lots of scales, you're making lots of different markdown files. But all of these are hopes. All of these are non-deterministic. All of these work sometimes but not all the time.
• 54:05
  Brandon: And you need that you know old school code.
• 54:08
  Brandon: You need that determinism. You need real software that's checking things and doing things and triggering things and that's a really important part of the harness and we can't leave that out
• 54:20
  Christine: Yeah.
• 54:21
  Christine: that hundred percent makes sense. I think I learned that when the cost of failures is higher[laughter] then the m- system should depend less on hoping that the model remembers a certain sentence[UM] in one of the[UH] markdown files [UM] I could also imagine that maybe sometimes not every smaller task needs a custom environment. Yeah, I I could imagine that maybe for a one-line copy change[UH] building a a a whole environment is not would be absurd
• 54:54
  Brandon: Yeah. Yeah. Yeah. And
• 54:56
  Christine: Yeah,
• 54:56
  Brandon: it.
• 54:56
  Christine: are there how would you describe like the type of
• 54:58
  Brandon: Mm-hmm.
• 54:59
  Christine: situations to spend more
• 55:01
  Brandon: Yeah.
• 55:02
  Christine: time on designing
• 55:03
  Brandon: Mm-hmm.
• 55:03
  Christine: the harness
• 55:04
  Brandon: Yeah Yeah, I think this is also a scale.
• 55:07
  Brandon: so let's talk through the different endpoints and then the middle.
• 55:10
  Brandon: So
• 55:10
  Christine: Mm-hmm.
• 55:11
  Brandon: like you mentioned, on one end if it's, you know, a one-line copy change, don't waste your time with this.
• 55:16
  Brandon: It's low stakes and the cost of the mistake is minimal as on the other side the complexity grows, safety becomes more paramount, more important,
• 55:27
  Christine: Mm-hmm.
• 55:27
  Brandon: and And I guess in that world
• 55:30
  Christine: Mm-hmm.
• 55:31
  Brandon: it's worth spending a
• 55:32
  Christine: Mm-hmm
• 55:32
  Brandon: lot of energy and a lot of effort on your harness and actually taking the time building it, thinking carefully, going through loops, the human agent feedback loop and getting something really good before you start your test And then and then there's stuff in the middle. So, something that[UH] actually John mentioned in his article[UM] that [UH] I started playing with[UH] and and it seems to work. know, you don't have to design the harness yourself. You can just tell your agents to do it.
• 56:00
  Brandon: So, you can for example create a[UH] a nice sub-agent graph where the first step is to have a bootstrap agent build a harness for you for your task.
• 56:10
  Brandon: And and that that I found
• 56:11
  Christine: Mm-hmm
• 56:11
  Brandon: works well for these in the middle cases. So you have a you have a reasonably complex problem you're trying to solve. Maybe the stakes of making a mistake are not too high, like it doesn't send an email to your boss [laughter] with something rude
• 56:25
  Christine: Mm-hmm.
• 56:25
  Brandon: in it but you know, you're changing some code.
• 56:28
  Brandon: You're you're doing some, you know, large task and just asking your agent to do this for you
• 56:34
  Christine: Mm-hmm.
• 56:35
  Brandon: is something that's helpful.
• 56:36
  Christine: so it sounds like the amount of harness engineering should scale with factors like the difficulty of the task, the cost of failure[UM] maybe the number of different failure modes.
• 56:49
  Christine: And probably I could imagine that if it's [UM] a recurring class of task, you might want to invest more time on building a reusable harness.
• 56:59
  Brandon: Yeah, that's a really good point
• 57:00
  Christine: Yeah, so it sounds like that not every subtask needs the general-purpose harness and the elaborate infrastructure[UM] building the right harness that is better adapted performs really well on a specific task or bottleneck. So that is where the large productivity gains can come from.
• 57:19
  Christine: So I guess the takeaway is match the harness investments to the task.
• 57:24
  Christine: Is that right?
• 57:25
  Brandon: Exactly.
• 57:25
  Christine: Alright.
• 57:26
  Christine: OK. Well maybe to wrap things up, when setting up the environment for our workflows, what we often do is we set up the tools, the skills, the contacts, even hooks[UM] for our h- entire workflow. And we will have all our agents and sub-agents reuse all of them.
• 57:44
  Christine: But hopefully, after listening to this episode, you're asking yourself also another question.
• 57:51
  Christine: What harness could we build that is adaptive to this specific task and makes the outcome better?
• 57:58
  Christine: Or what environment would make it hard for that agent to do wrong?
• 58:03
  Christine: So, that question might change how you think about doing harness engineering from here on out and how you focus your harness on a problem you're working on.
• 58:12
  Christine: Well, this was episode four of Superlinear.
• 58:15
  Christine: We hope that you got something out of this episode.
• 58:18
  Christine: Have a lot of fun with harness engineering. Please tell us in the comments about what you build and what you notice in your approach. Brandon and me are learning as well.
• 58:27
  Christine: And we'd love to hear about what you experienced in your workflows and don't forget to subscribe so you don't miss out on our next episode.
• 58:37
  Christine: Thank you all and see you next time.
• 58:39
  Brandon: Bye everyone.