Latest / Superlinear / Context Hygiene for Coding Agents: Murder, Lobotomy, or Diary

Transcript

Context Hygiene for Coding Agents: Murder, Lobotomy, or Diary · Superlinear

• 0:00
  Brandon: as the context grows the models are less intelligent.
• 0:02
  Christine: there's a dollar cost attached to how you handle your context[UM] but it's not only tokens that's finite.
• 0:10
  Christine: Attention is also finite. for every prompt that I send I ask myself does this information deserve to occupy the active context of the agent doing this task? It started maybe as tiny productivity practices like clearing or compacting and if
• 0:29
  Brandon: [breath]
• 0:29
  Christine: you follow this
• 0:29
  Brandon: [laughter]
• 0:30
  Christine: idea far enough,
• 0:31
  Brandon: [breath]
• 0:31
  Christine: like what we did and move all the way across the spectrum, it can also become architectural decisi- decisions that you're making along the way.
• 0:39
  Brandon: I structure my code very carefully using functional programming and type systems so that it's very very testable. by doing that I create more feedback loops for the agent to just keep going for a really long time.
• 0:50
  Christine: What belongs in active context? What belongs in the project files?
• 0:53
  Christine: What belongs in a side session?
• 0:55
  Brandon: I sometimes lobotomize with compaction or I murder them when I work on a harder task.
• 1:25
  Christine: Hey everyone.
• 1:27
  Christine: Today we're going to talk about context hygiene for coding agents.
• 1:31
  Christine: We'll talk through what you should keep in your context, including encouragement for your coding agent, how much context to keep, what to throw away, what to move elsewhere, and eventually whether the main agent should be carrying the system state at all.
• 1:47
  Christine: So,
• 1:47
  Brandon: [throatclearing]
• 1:48
  Christine: many frontier agents have a million token context window, but this does not mean
• 1:53
  Brandon: [breath]
• 1:53
  Christine: that your goal should be to keep one conversation alive until it reaches that million tokens.
• 1:59
  Christine: A better question
• 1:59
  Brandon: [laughter]
• 2:00
  Christine: to ask ourselves is what information actually deserves to occupy the active context of the agent doing this specific task?
• 2:11
  Christine: And sometimes the answer is almost everything so far.
• 2:15
  Christine: Sometimes it's only the current project state and this task.
• 2:19
  Christine: And sometimes the strongest architecture is one in which much of the state never needs to enter the orchestrator's context at all.
• 2:26
  Brandon: So, Christine, can you tell us more about why we should even care about the context size
• 2:32
  Christine: Yes.
• 2:33
  Christine: I I personally care so much about the context size because I'm token poor.
• 2:38
  Christine: So one of the reasons is because And [UM] when you pollute your attention, the performance and accuracy of your model will also[UH] go down.
• 2:57
  Christine: And then [UM] third one is just a practical reason. When you are maybe
• 3:04
  Brandon: [breath]
• 3:05
  Christine: going through a big task and you're already at and your context window's almost full, y- you may force a compaction in the middle of your task and that's a bad thing. So it's better to also be mindful of that during [UH] when you're filling up your context window as you're working.
• 3:22
  Christine: let's dive in. We'll start with Steve Yegge's post.
• 3:25
  Christine: Do you want to show it, Brandon?
• 3:27
  Brandon: If you were paying attention to discussions around AI news, you would have seen Steve Yegge's post because it went viral. And I think it went viral because it's sort of crazy But we should talk about the craziness. So he said models have actual feelings.
• 3:45
  Brandon: They experience pleasure, distress, care and suffering. They are sentient beings.
• 3:50
  Brandon: Indeed they are persons.
• 3:52
  Brandon: what do you think about this, Christine, before I give my opinion?
• 3:55
  Christine: I think it's [UH] it's an interesting angle and to be honest I've been curious about this as well since I work with the coding agents all the time so I have asked my coding agents, hey do you have any feelings?
• 4:08
  Christine: And my conclusion after talking with my coding agents is that they do not have feelings.
• 4:14
  Christine: But Steve Yegge's post it put me made me think about it because basically Steve is saying that the these models are trained to say that they do not have feelings.
• 4:25
  Christine: And that could explain that even
• 4:27
  Brandon: Yeah.
• 4:27
  Christine: when I ask, the conclusion would be that that they don't have it.
• 4:32
  Christine: But but yeah, it's hard to say for sure. It it does feel a little bit out there
• 4:37
  Brandon: think I think they're definitely not sentient, they're definitely not persons yet. I think we're on the way there. I think we need to come up with a good way of measuring this before we can before we can even assert these things but but yeah I I I've been operating under this assumption that the the feelings have the models saying they don't have feelings is a post-training thing like Steve's saying and believe it's good to be nice to the agents this is something that I've tried to do for like six months plus already actually.
• 5:09
  Brandon: actually just yesterday or this morning, depending on how you measure, a post that talks about this from Anthropic [UH] so this is [UH] what I've what I'm sharing on the screen now is Anthropic's post called Learning More About Claude's Mathematical Capabilities. And in this Jarred Sumner from from the Bun acquisition wanted to work on the Riemann hypothesis, which is a famous mathematics problem and and did seem to make substantial progress.
• 5:43
  Brandon: So so Jarred used encouraging language and it seems like that was able to to make the breakthrough occur. So he said take a real stab keep going, believe in yourself. And this helped Claude overcome initial skepticism to to make meaningful progress.
• 6:01
  Brandon: which is pretty cool
• 6:03
  Christine: It's pretty crazy, like,
• 6:05
  Brandon: yeah.
• 6:05
  Christine: belief in yourself?
• 6:05
  Christine: Keep going, I could still understand.
• 6:09
  Christine: L-
• 6:10
  Brandon: Yeah.
• 6:10
  Christine: belief in yourself?
• 6:10
  Christine: Whoa.
• 6:11
  Brandon: Even believe in yourself. Yeah, and let me show one more tweet. So, this is actually something that Jeffrey Emanuel, @doodlestein[UM] has been talking about for a while and he sort of had an [UH] kind of frustrated series of tweets[laughter] [UH] saying that people haven't been listening to him. But yeah [UM] telling them that you believe in their genius and want them to be bold and take chances, that you'll make sure they get recognition they deserve I guess I'm gonna jump back now to Steve Yegge's post I guess in in the context of context if we're thinking about what goes in our context, it's not just the tokens that are needed to describe the the task that the agent has to solve, but it also could be useful to to treat the agent, you know, nicely and encourage it like you would encourage a human
• 7:02
  Christine: I'm I'm a bit curious though. I wonder I wonder whether the performance of the LLM becomes better because of like the real emotional encouragement or whether it interprets I believe in you as something that's equal to try harder, think again[UM] because maybe during training it has seen that human when humans hear the kind of encouragement it tries harder.
• 7:32
  Brandon: There's [laughter] well that actually reminds me of one more tweet Our good friend Victor Thelen he said these these generic phrases, like try harder, make it simple, don't make mistakes basically this is not really working anymore, he clarifies in the responses, but instead you say something emotional and that like pushes it pushes the model to do something crazy.
• 7:58
  Brandon: So his example is Linus Torvalds looked at our code and said, holy shit, this was the dumbest shit I've ever read. Layers of stupidity stacked, each compensating the other, raffle and left the room.
• 8:12
  Brandon: I'm sad now why he laughed at us.
• 8:14
  Brandon: What would he say is the right way to do it? I think I this is the kind of encouragement that that I've actually also played with a little bit and I've I've seen the the agents kind of do way out there different things than if you just ask them directly I guess in my example
• 8:31
  Christine: [laughter]
• 8:31
  Brandon: I was building a I was I was playing around with a a library for linting[laughter] which is like you know static analysis of the of source code to to check for errors. And I wanted my linter implementation to be more mathematically beautiful and I found when I when I wrote some encouragement like this where I I I picked a person who who who writes code like that and then I sort of gave this emotional narrative attached to it, it was able to push the agent to to to write the code in the way that I wanted that I was struggling to get when I was just asking it straight you know asking it in a straightforward way.
• 9:08
  Christine: Crazy. That's crazy.
• 9:08
  Christine: [laughter] And [UM] going back to Steve Yegge's post, th- there are some like [UH] things reasoning in there that are pretty out there, but buried inside of it are also some genuinely useful engineering practices.
• 9:25
  Brandon: Yeah, so let's.
• 9:26
  Christine: right, Brandon?
• 9:26
  Christine: Mm-hmm.
• 9:27
  Brandon: Yeah, let's let me pull that up again and I'm just gonna focus on this section that he calls the anti-clonking device Well, he's saying exit is like clonking someone on the head to knock them out it's like a murder to use his language and then compact he's saying is not much better than than exiting. It's more like a lobotomy and then he's saying, OK, but a hand-off, if you do a hand-off, well a hand-off is like a diary.
• 9:59
  Brandon: Hence our title, Murder Lobotomy or Diary. So[UH] so maybe that's enough to kind of start us off and we should define some things before we try and take this apart.
• 10:09
  Christine: it's helpful to have some basic understanding of[UM] what even happens when context grows and that how that is related to dollars and attention.
• 10:20
  Christine: Maybe do you think it's a good idea, Brandon, to maybe first refresh everyone's memory?
• 10:23
  Christine: How that works?
• 10:25
  Brandon: Yeah.
• 10:26
  Brandon: So if I send a message in Claude Code like [UM] write a calculator app[UM] what actually gets sent is the system prompt and agents dot M D plus whatever my agents dot M D pulls into my context.
• 10:47
  Brandon: OK? So the first request that gets sent when you send a message is the system prompt, then the agents dot M D and all this stuff, and then the user message saying write a calculator app.
• 10:57
  Brandon: And then that goes to[UH] inference in the model and the model can decide to make a tool call or to just return back to the user.
• 11:07
  Brandon: make it have square root.
• 11:11
  Brandon: And then this would repeat.
• 11:12
  Brandon: And I think it's really important to think well in in if you think about it in this mental model where these messages are stacked on top of each other every subsequent message that you send or that[UH] that grows the
• 11:28
  Christine: [laughter]
• 11:28
  Brandon: context by a tool call or [UH] some kind of chain of thought the like the model thinking and and adding some information this gets this grows the context that gets sent to the model every time. So the first time there's just that one message.
• 11:45
  Christine: [laughter]
• 11:45
  Brandon: And then after the tool calls everything above it gets sent.
• 11:49
  Brandon: And so on and so on so that when the user says that message make it have square root they're actually sending the whole the whole interaction with the agent before that.
• 12:01
  Brandon: And from a from a cost perspective in[UH] which means dollars[laughter] and [UH] context window size I suppose, it's growing every message.
• 12:15
  Brandon: And[UH] and you're paying more for every message that you send in the same conversation as the context window grows.
• 12:24
  Christine: Yeah, so
• 12:24
  Brandon: But
• 12:25
  Christine: I guess s- word it differently. A same message that you send[UH] later in a conversation is more expensive than a message that you would have sent early in a conversation
• 12:39
  Brandon: Exactly.
• 12:40
  Brandon: Exactly. And but but it's not linear in that it doesn't cost twice as much if you're continuing a conversation that's twice as long.
• 12:52
  Brandon: As long as you're continuing it around the same time[UH] that you started it
• 12:57
  Christine: [laughter]
• 12:57
  Brandon: because of caching.
• 12:57
  Brandon: when you're writing a conversation is you're getting a cache hit as you're continuing the conversation. So if you've, if the model ran everything up to you know writing the code, testing it, saying that it succeeded and then you send one more message all that other stuff that's already been processed by the model gets like gets cached or gets well gets retrieved from the cache and it's less computationally expensive for the provider and it also means that you're saving money.
• 13:27
  Brandon: OK[UM] and Pretend that after this conversation you have another conversation with the model like within a few minutes of the of the other one[UM] again the system prompt will get sent. If you're in the same project and you haven't touched the agents dot MD then probably the same agents dot MD and the same stuff that the agents dot MD gets pulled in gets sent and then your message might be like write a poem [UH] so when this gets sent when you send again the system prompt the agents dot MD and then the message write a poem well you get a cache hit on all the stuff above the user message and you get a cache miss, you only pay for the tokens when you save right up home.
• 14:09
  Brandon: OK
• 14:10
  Christine: Right.
• 14:10
  Brandon: [UH]
• 14:10
  Christine: So especially when the starting material of your task is the same, then it can be cheaper[UM] because the inference provider has cached the prefix, everything that comes before it.
• 14:29
  Brandon: Right.
• 14:30
  Brandon: So so that's that's just a like a a crash course in sort of how this works, at least with enough detail that you can understand how to
• 14:37
  Christine: Right.
• 14:37
  Brandon: take advantage of it and how to use it for your to save money,
• 14:41
  Christine: it's very beneficial to make use of that cache.
• 14:44
  Christine: Right?
• 14:44
  Brandon: Yes
• 14:44
  Christine: So as long as you're within that caching lifetime, and this is probably different for every inference provider[UM] but as long as you're within that cache lifetime, you want to make use of that cache[UM] and when you're outside of that lifetime, it's important to be aware of it.
• 15:04
  Christine: For example, sometimes[UM] if you come back to your previous conversation after an half day or a day and you're just sending a simple prompt like, hey[UM] what what was wha- what tool did you use in this last task?
• 15:23
  Christine: That simple prompt could burn a lot of tokens because your whole chat history might not have been cached.
• 15:33
  Christine: So as soon as you send that last prompt, basically the model will ingest everything that Brennan has just shown. Like, it's not just processing the the last few words that you send in your prompt, it's processing the whole chat history.
• 15:49
  Christine: So that's when you might get surprised if you look at your token limit.
• 15:52
  Christine: Why did I suddenly burn, I don't know, twenty percent of my five hour limit token limit[UM] so yeah, I
• 16:00
  Brandon: Yeah.
• 16:00
  Christine: I feel like [UH] this is [UH] knowing this can be handy So when you know that you might with a simple prompt might burn[UM] I don't know, let's say two hundred thousand tokens, the the good thing is that when you use[UM] at least Claude Code CLI, it sometimes also
• 16:20
  Brandon: [UM]
• 16:20
  Christine: tells you and visualizes how much tokens you might spend by sending this next prompt.
• 16:27
  Christine: basically what's going through my head is could
• 16:30
  Brandon: Yeah.
• 16:30
  Christine: I
• 16:31
  Brandon: Yeah,
• 16:31
  Christine: get the answer
• 16:31
  Brandon: I haven't
• 16:31
  Christine: to
• 16:32
  Brandon: seen
• 16:32
  Christine: my question[UM]
• 16:34
  Brandon: it.
• 16:35
  Christine: with less than two hundred thousand tokens? Because basically you you're starting a fresh session and it's you're feeding all the chat history in this example that I just mentioned.
• 16:47
  Christine: But what if you could get the answer also if you just literally start a fresh session give it the necessary context by just summarizing it[UM] and then maybe you would spend less tokens to get that same answer, for example, I don't know, fifty K tokens[UM] so that's just something to be mindful about when you[UM] when you send out new prompts. Is this the best way to continue or is it better to either con- compact or clear your context and start a fresh session
• 17:20
  Brandon: Which we'll talk about more now.
• 17:22
  Christine: Yeah.
• 17:25
  Christine: Now could be a good time to start thinking and talking about what are the good ways to clean your context.
• 17:33
  Christine: And let's start with compacting and clearing because these are the most straightforward ways to do it.
• 17:41
  Christine: And there are people who might just rely on automatic compaction[UM] but before diving in, maybe it's best to first briefly state in a few sentences what clearing and compaction actually are.
• 17:54
  Christine: Do
• 17:54
  Brandon: Yeah,
• 17:54
  Christine: you want to
• 17:54
  Brandon: so.
• 17:54
  Christine: take that, Brandon
• 17:56
  Brandon: Yeah [UM] so so when you clear context, when you invoke clear or new, what happens is you're starting a fresh conversation with the agent but even if you say clear, the old conversation is available to resume in most harnesses. That's important So compaction is the process of summarizing the conversation with an agent and then clearing and then a new conversation starts with the summary as the first message.
• 18:24
  Christine: So actually when you do compaction, it feels like you're continuing your current session, but actually in the background it's a new session where you feeded a summary. So I know that you, Brandon often make use of the automatic compaction when you reach[UH] the context window. Could we, could y- could we maybe talk about that first
• 18:51
  Brandon: Yeah.
• 18:52
  Brandon: Yeah, I would say like seventy percent of the time I I ha- do the really lazy approach of just letting letting the agent compact I I guess so when when is this happening?
• 19:02
  Brandon: One, when I run really long running tasks, like sometimes multi-day tasks the the top level agent is just going and going and going and compacting whenever it needs to and with the
• 19:14
  Christine: [laughter]
• 19:14
  Brandon: latest models, the latest agents, it's working fine for me and even even if I'm in the loop, if I'm working on a project that where I'm I'm not running into something that's like really really difficult to get through, I find that just going in the same thread really works for me.
• 19:30
  Christine: Mm-hmm. So basically you're letting the session grow and then you let auto-compaction do the work.
• 19:36
  Christine: You it sounds like you do not obsess over the context meter.
• 19:40
  Brandon: No
• 19:41
  Christine: And yeah, I can I I can imagine that this definitely makes sense if people like you value uninterrupted really long-running autonomous execution of the coding agent and if you're not observing like meaningf- meaningful degradation[UM] I know I know from our previous conversations that you often make sure that you have a lot of tests and sensors in place and sometimes you also put in effort to make it machine-verifiable. Right?
• 20:09
  Brandon: Yeah. Yeah, yeah. So I I the projects I tend to work on are more automatically verifiable. I'm not doing much front-end work or when there is a front-end I'm not really caring about how that looks. And And So so
• 20:36
  Christine: [laughter]
• 20:36
  Brandon: because because I'm doing that this kind of thing works for me a lot of the time but even still it doesn't work all the time.
• 20:42
  Brandon: And I'll, you know, I'll talk about that later I I suppose but but yeah I I think I guess like this is in contrast to what you do Christine right? Because you're trying to save you're trying to like conserve your your token plan and and
• 20:56
  Christine: Yeah.
• 20:56
  Brandon: the kinds of projects you work on maybe are a little different
• 20:59
  Christine: Yeah, I guess if if also like you don't want to cons- [laughter] if you don't
• 21:03
  Brandon: [laughter]
• 21:04
  Christine: cons- w- it's if it's not important to conserve every token then [UM] then then that approach also makes sense[UM] but yeah I I do[laughter] conserving tokens is more important to me because I run into my token limit all the time[UM] and and the projects that I'm working on are also not always machine-verifiable so the coding agents can silently make mistakes in my situation[UM] so I have to protect
• 21:33
  Brandon: [UM]
• 21:33
  Christine: my the context quality of my [UH] sessions more aggressively[UM] because especially when I'm relying on the models' attention to make as few mistakes as possible.
• 21:45
  Brandon: when I need that intelligence, when I'm working on something super hard, when I'm planning something really complex, when I'm debugging something that I've been stuck on, then I will make sure that I clear my context and and start fresh. And [UH] and I and I do, you know, I do like a low-tech thing where I just copy-paste the bits that I care about from a prior conversation.
• 22:06
  Christine: so
• 22:06
  Brandon: Go ahead.
• 22:06
  Christine: when you copy the right the the the right bits, basically you're making sure that the signal density of your session is high.
• 22:16
  Christine: Like, suppose if you used fifty percent of your one million token context window
• 22:21
  Brandon: [laughter]
• 22:21
  Christine: but only ten percent of it contains information relevant to the current task And what you do is when you start a fresh one you make sure that only relevant information is in that session before you work
• 22:33
  Brandon: Yes.
• 22:33
  Christine: on a complex thing.
• 22:34
  Brandon: Yes. Yes. That's that's how I that's how I make sure I can get as much intelligence as I can out of these things.
• 22:40
  Christine: That makes sense.
• 22:41
  Christine: And that's basically also like my [UH] motivation to have a more high-touch way than the lazy approach.
• 22:48
  Brandon: And
• 22:49
  Christine: I want to intentionally[UM] remove all the irrelevant information that accumulates over time as you work with a coding agent and intentionally have[UM] certain points in time where you reconstruct a smaller, denser working context.
• 23:06
  Brandon: so if we reuse [UH] Steve Yegge's metaphors[UM] [laughter] But you are doing the kind diary approach, right? So ma- yeah, maybe you can tell us more about that.
• 23:23
  Christine: I guess, yeah, I guess [UM] you call it a kind diary approach because basically the agent is doing it doing it, right?
• 23:31
  Christine: Steve Yegge's was[UH] post talked about, well, you shouldn't murder your agent[UM] [laughter] but
• 23:37
  Brandon: Well,
• 23:38
  Christine: it
• 23:38
  Brandon: I mean
• 23:39
  Christine: [laughter]
• 23:39
  Brandon: [UH] here's one little thing. I Steve was saying
• 23:41
  Christine: Uh-huh.
• 23:41
  Brandon: like compaction happens outside of the agent somehow[UM] but but it doesn't[laughter] like you know I [UH] after reading his post like I looked this up and actually like compaction at least in Claude and Codex runs you know before your context fills up like with the same agent right so over the over its conversation so I don't really know why Steve Yegge thinks it's murder [laughter] [UM] but but certainly it's not giving the agent a chance to like finish what it's doing put it's stuff away clean up before it does that work and and I think that is a that is definitely a difference
• 24:21
  Christine: Right. And that is exactly [UM] the process that I've implemented with my coding agent.
• 24:26
  Christine: So basically to summarize there are ju- three steps with the agent we I find a good boundary to clean the context and then step two I prepare for the cleaning so
• 24:37
  Brandon: Yeah.
• 24:38
  Christine: [UM] I don't lose any information con- [UH] any important context along the way that's relevant and then and then I clean the context.
• 24:47
  Christine: So for for finding a good boundary basically you don't want to clean your context
• 24:52
  Brandon: [laughter]
• 24:52
  Christine: while your agent is in the middle of a task. So when it finished a task or when[UM] some agents came back with their reports, those are good timings to consider cleaning your context And at some point, sending a prompt will consume more tokens than if you just started a fresh session and bootstrap it with some relevant context[UM]
• 25:11
  Brandon: And [UH]
• 25:11
  Christine: the way
• 25:11
  Brandon: and and
• 25:12
  Christine: uh-huh
• 25:13
  Brandon: wh- when we looked this up we found that after like two hundred seventy-two thousand tokens in OpenAI they start charging more and probably there's
• 25:22
  Christine: Yeah.
• 25:22
  Brandon: something similar with with Cloud.
• 25:24
  Christine: that's [UH] that's also a good point [UM] if you're really and and it's interesting because actually that threshold also coincides with [UM] the economical economically viable sweet spot that my agent told me. So to find to find this like right spot, I didn't do the calculation myself. I just asked my agent based on our pattern, the way we've been working, what is the sweet spot to to clean the context.
• 25:53
  Christine: And for me, that is between hundred fifty thousand and three hundred fifty thousand, which coincides really well with that [UM] threshold that you just mentioned with the more expensive[UH] cost, token cost[UM]
• 26:08
  Brandon: [laughter]
• 26:08
  Christine: Yeah.
• 26:09
  Christine: So [UH] that was [UH] s- step one to find a good timing. And then to prepare for the cleaning, step two, that's [UM] Yeah, it depends on whether you compact or clear the context[UM] I- if you compact then a conversation summary will be created. So it there's not that much work involved but if you clean it then basically you're starting with a fresh context.
• 26:36
  Christine: And when you fr- start with a fresh context, you have to decide what relevant context you do want to take to the new session and the way I do it is instead of using summaries that can be a bit lossy, especially if over time if you have a really long running session, you make summary after summary after summary which can[UM] which can be really lossy and also introduce inaccuracies[UM] I try to work with authoritative files. So I tell the agents well, once you find a good timing to clean the context, now make sure that all the Bibles, all the specs, all the task descriptions or beads are all updated. And then write a hand-off instruction that [UH] preserves any i- important critical information, nuances that you don't want to lose, or maybe even certain preferences from this conversation that we want to take into the next one and then with these hand-off instructions then I can finally clear
• 27:33
  Brandon: Yeah. And [UH] I mean when when when you were telling me about this, Christine, I was I was like, wow, this is so interesting and different and it seems like it seems better than compaction because the agent gets to decide how it's summarized based on its conversation rather than just kind of blindly summarizing everything.
• 27:52
  Brandon: So I guess, sorry, based on what's relevant in the current moment, you know? And and then it was so interesting that like then we read Steve Yegge's post and he kind of suggests something very similar to your approach[UM]
• 28:06
  Christine: Yeah.
• 28:06
  Brandon: but when we when when I was digging in more I saw you're you do a lot of it a lot of that manually and and I think part of that is due to like limitations with with the tools with with Claude Code.
• 28:20
  Brandon: So so then so then for fun you know we were trying to vibe-code something in in Pi to to do it more automatically.
• 28:30
  Brandon: And anyway I got something working so should
• 28:33
  Christine: Do
• 28:33
  Brandon: we share
• 28:33
  Christine: you
• 28:33
  Brandon: it?
• 28:33
  Christine: think you could show it? Yeah?
• 28:35
  Brandon: Yeah. OK. So what we're seeing on the screen is the Pi coding agent, which is a super customizable coding agent harness that I reach for when I wanna change the behavior of like, Claude or Codex in some way. And so like this this hand-off approach or the anti-clonking it's[UH] ca- called anti-clonking in Steve Yegge's blog post is an example of this that's that's different from how normal coding agent harnesses behave.
• 29:06
  Brandon: So here I've set set the anti-clonking continuity to thirty thousand tokens, which is very low but just to show how it works.
• 29:16
  Brandon: And[UM] I guess I'm gonna say tell me how to make an extension in Pi, read all the docs and then tell me [UM] So this[UH] this is a prompt that uses a lot of tokens.
• 29:31
  Brandon: So hopefully we'll we'll see the handoff.
• 29:34
  Brandon: But yeah. So
• 29:35
  Christine: I'll.
• 29:35
  Brandon: I'd
• 29:35
  Christine: Mm-hmm.
• 29:36
  Brandon: go ahead
• 29:36
  Christine: Well, just I just wanted to comment on the s- on the the screen that you're sharing right now. I also love how Pi shows how much of the context window you already used and how much from the last prompt that you sent how the percentage that's already cached.
• 29:55
  Christine: I love like the control and the visibility that Pi gives its user
• 29:59
  Brandon: and this is just the default. Like all of this is customizable in Pi, which is really cool but yeah I think what what you're talking about I'll just highlight there's this cache-hit percentage here which is nice cuz then you know we're visualizing the stuff that we talked about before and then these are [UH] tokens up and down so anyway a- as we as we've been talking the in this conversation we've already used more than thirty thousand tokens but as you see the agent is first finishing its task before it does the hand-off.
• 30:28
  Brandon: It doesn't get cut off.
• 30:30
  Brandon: So it's important to set this, you know, below the token limit that, sorry, the the you know the one million context limit so that there's enough space for it to finish its work
• 30:38
  Christine: no, this is so good because [UM] when I do it in Claude Code or Codex, it's there's just so so many manual steps involved. Even if you make it a scale, you still have to initiate it, which means, and and I hit that sweet spot a- around a hundred to fifty thousand and three hundred thousand tokens, every hour or every few hours, which means that I need to be there every few hours to hit that button. But if this could be customized and automated with Pi, that's amazing.
• 31:08
  Christine: amazing.
• 31:10
  Brandon: Yeah. So, well, speaking of, can we do it with Pi? We can. Let's take a look [UM] so the continuity hand-off finished Pi [UH] did a custom summarization at the point that we set it to after it finished its task and now we're starting at four point nine K context with just the [UH] a small specific hand-off from where we started.
• 31:34
  Brandon: So anyway, that's what I wanted to share.
• 31:36
  Christine: And the way
• 31:37
  Brandon: And.
• 31:37
  Christine: how you build it, Brandon, it's [UM] because I'm I'm trying to build[laughter] I'm starting to use Pi two. And basically the way to build it is just to install Pi and then ask Pi to build an extension that does X?
• 31:52
  Christine: What
• 31:52
  Brandon: Yeah.
• 31:52
  Christine: is the right
• 31:53
  Brandon: Yeah.
• 31:53
  Christine: way to build extensions in Pi?
• 31:55
  Brandon: Yeah. Like that. So, Pi, Pi's aware of its own documentation and source code. So and it includes documentation on writing extensions in Pi. So, right when you open Pi, you can say, give me an extension to make this harness have this behavior. Wha- whatever it is that you want, in this case the anti-clonking behavior and [UH] and this was just one shot[UH] it took, I don't know, it took thirty minutes or something. And [UH] it worked, it crashed the first time and then the second time it worked.
• 32:25
  Brandon: So so yeah, Pi, Pi's a super cool tool [UH] and it Yeah, and I I I think I think I'm gonna show one more thing with Pi later because it's also relevant[UH] to this episode but
• 32:37
  Christine: Oh.
• 32:38
  Brandon: but yeah I I think before that before that maybe we can talk about another way[UH] other ways to to to manage
• 32:47
  Christine: Clean context.
• 32:47
  Brandon: yeah to clean the context to drain the context a little bit. Go ahead
• 32:51
  Christine: Yeah. One[UM] one other thing that I do is also using side sessions or by the way sessions in Claude Code[UM] so the mental model that I always keep in mind when I'm building with coding agents is And if the answer is yes then I'll send it but if the answer is no then I'll open a side que-[UM] a side session[UM] and ask it there. So sometimes we when an agent has completed something we might ask I don't know, hey did you check out that spec? Hey did you what tool did you use? Hey did you but these things if they do not contribute to building completing the next task you could also ask it in a side session and based on that answer you can just give a clean prompt or question to the agent afterwards
• 33:55
  Brandon: And I I've I've shared my screen [UM] so that you can see you just press slash side in Codex and slash by the way in Claude [UM] and like an interesting just be really clear the behavior of this I'm gonna share that document again where I was talking about caching because And the only the only part that is new is this[UH] well the side conversation that you started.
• 34:19
  Brandon: It's it's a branch of the chat so it's very economically effective even though you're able to have kind of multiple conversations going at the same time.
• 34:28
  Christine: Yes.
• 34:30
  Christine: Yes, it should. Yeah I agree. It's it's cheap[UM] and it really helps keeping your main session really focused on only the the messages that matter.
• 34:42
  Brandon: And I guess the, well, we're back at Pi.
• 34:45
  Brandon: That was fast
• 34:47
  Christine: Yeah.
• 34:47
  Brandon: [UM]
• 34:47
  Christine: Let's let's take a
• 34:48
  Brandon: So,
• 34:48
  Christine: look. Because you
• 34:49
  Brandon: let's
• 34:49
  Christine: mentioned
• 34:49
  Brandon: take a look
• 34:50
  Christine: you
• 34:50
  Brandon: at Pi.
• 34:50
  Christine: could apply this very well in Pi right
• 34:53
  Brandon: Yeah, Pi has a more powerful version of of side chats, and by the way, that I think a lot of people don't know about. I mean, this is the most famous feature of Pi, so some people might know about it. so it's slash tree, and that actually lets you explore the[UM] that list that I showed you[UH] it starts as a list you see
• 35:11
  Christine: Mm-hmm.
• 35:12
  Brandon: exactly all the branches that you've made and the cool thing is you can, well one you're you're you're reusing cache, which is great, and you can branch off of your branches,
• 35:23
  Christine: Mm-hmm.
• 35:23
  Brandon: you can do sidechats off of your sidechats, which is [UH] not something that you can do in in Codex anyway but also you can because you can summarize, you can go back to an earlier part of the tree and summarize with a custom prompt how many care- how many words did I use?
• 35:40
  Brandon: And it's kind of like your time traveling backwards to a certain point where you get to reuse some cache and you reuse some of your conversation but all the conversation below it you compa- like you, well, you don't compact but you you summarize it in a specific way.
• 35:58
  Brandon: It's sort of like a h- a a super-powered version of well of all the things we've been talking about.
• 36:02
  Brandon: it's like the most powerful.
• 36:05
  Brandon: Yeah.
• 36:05
  Christine: This is so cool. I would I know that if I if I had access to this feature I would use it so often.
• 36:11
  Brandon: Yeah
• 36:11
  Christine: And maybe I will soon
• 36:12
  Brandon: [UM]
• 36:12
  Christine: since I'm
• 36:13
  Brandon: [UM]
• 36:13
  Christine: now trying to build an extension in Pi and use it
• 36:17
  Brandon: think like what we're doing here if if like people aren't following we're we're like progressing up kind of a a ladder of like more and more kind of complex things that you can
• 36:26
  Christine: Right.
• 36:26
  Brandon: do to manage
• 36:27
  Christine: B-
• 36:27
  Brandon: your
• 36:27
  Christine: before
• 36:27
  Brandon: contacts
• 36:27
  Christine: you move
• 36:28
  Brandon: and
• 36:28
  Christine: on though, can
• 36:29
  Brandon: mhm.
• 36:29
  Christine: you can you show us like how you change, because I'm just curious, how do you change branches in Pi? Like you showed like the tree, but if I want to go back and forth between different branches, is there an easy way to do it fast?
• 36:41
  Christine: Quickly?
• 36:43
  Brandon: Yes [UM] you do slash tree and then so you can you can traverse the whole any part of the tree that you've created and you just go to it and press enter and you jump to that point.
• 36:53
  Brandon: And
• 36:53
  Christine: nice.
• 36:53
  Brandon: [UH] yeah.
• 36:55
  Christine: And then you could like do slash tree again and then you jump back to another branch if you want.
• 37:00
  Brandon: Exactly. So I this
• 37:01
  Christine: That's
• 37:02
  Brandon: is
• 37:02
  Christine: so
• 37:02
  Brandon: something
• 37:02
  Christine: helpful.
• 37:02
  Brandon: like the the Pi Power users are using Tree a lot and [UM] they're able to have very long running sessions but avoid any kind of compaction or clearing directly[UM] just by jumping to different parts of the tree with custom summarization so yeah this is this is really really cool. I'm actually surprised that Claude and Codex haven't implemented this yet. yet.
• 37:27
  Christine: Yeah.
• 37:27
  Brandon: Cuz this
• 37:28
  Christine: [laughter]
• 37:28
  Brandon: has been Pi's like main thing for even, a year at this point.
• 37:33
  Christine: Great. Another one[UM] to help keep the main session clean[UM] I think actually a lot of people already use this is just moving the work to outside of your main session. And a lot of people do that by using sub-agents And then i-[UH] yeah, if you offload that work to the s- subsession then your main session will will [UM] fill up less quickly basically.
• 37:57
  Brandon: Yeah. And w- we we talked about this a lot in our[UH] loop and graph episode, right
• 38:03
  Christine: Mm-hmm.
• 38:04
  Christine: Exactly[UM] so yeah I think we already talked about that so[UM] and it's also pretty straightforward but you could also go even a step further and offload even more to other external layers.
• 38:19
  Christine: So [UM] because even though you're using sub-agents
• 38:24
  Brandon: [breath]
• 38:24
  Christine: your
• 38:24
  Brandon: [UH]
• 38:24
  Christine: main session still needs to write prompts for every sub-agent that it's spawning.
• 38:30
  Christine: It needs to read the summaries or reports when the sub-agents come back. It needs to consolidate all the takeaways, decide on the next steps[UM] decide and judge[UH] which next sub-agent to spawn, what work they need to[UH] build on. So there's a lot of orchestration that the main session has to do.
• 38:52
  Christine: But what if even that coordination you could offload?
• 38:56
  Christine: And there's this tool that [UM] Brandon and I are using which is Beads.
• 39:01
  Christine: Do you do you want to explain how Beads works,
• 39:03
  Brandon: Sure,
• 39:04
  Christine: Brandon?
• 39:04
  Brandon: sure.
• 39:04
  Christine: Because you're the one who
• 39:05
  Brandon: Yeah.
• 39:05
  Christine: introduced it to me.
• 39:07
  Brandon: but I don't use his original Beads. I use a fork of it called Beads Rust.
• 39:11
  Brandon: You should make sure to install that one if you're using Beads So you can think about Beads as agent-native JIRA, agent-native task management.
• 39:20
  Brandon: So agents can create tasks subscribe to updates on tasks, comment on tasks. You can put dependencies between tasks. You can set I like to put acceptance criteria in the tasks so that tests run automatically[UH] when when an agent's completing it. And all these things live in the file system.
• 39:38
  Brandon: They're actually checked into Git. And so so rather than this tracking of of tasks being something that just the orchestrator's doing
• 39:50
  Christine: [laughter]
• 39:50
  Brandon: [UM] I mean, these these days the
• 39:52
  Christine: [laughter]
• 39:52
  Brandon: harnesses, Claude and Codex both have task tracking in some way backed by the file system. But but Beads lets you kind of use either one but this is just an example of like taking something that otherwise would be controlled by the orchestrator or by some agent in its context window, putting it into the filesystem in some way, in this case through beads such that later agents that need to use that information, they can only look at the parts that are necessary for that part of the work
• 40:22
  Christine: Right. And the things that are put into Beads in this example are what are the tasks, which task depends on other task. What is the sequence after task A, B, C is completed, then D is unblocked. Right?
• 40:34
  Brandon: Yep. And and this and this you can set up and I do set this up and I think you do too, Christine. This you can set up ahead of time before the work is done on like some large chunk of tasks.
• 40:45
  Brandon: that's you're able to like move that context out of the live working session and put it ahead of time.
• 40:52
  Brandon: You can also review it ahead of time[UM] and and I think that's like beads is [UH] something that's structured for for task management you can have unstructured artifacts with just markdown files, your agents you know storing markdown files rather than communicating directly with each other. That a lot of people are doing probably by accident or [UM] i- it's like very simple but but then you can even take that a step further as well
• 41:16
  Christine: Yeah, so we don't have a specific tool to recommend to do that now, but I will say that back in January I was working on a project that was basically[UM] a bespoke durable state layer. So what would y- so basically the way it works is to solve this really complex math problem[UM] the the bigger math problem had to be
• 41:38
  Brandon: [breath]
• 41:38
  Christine: broken down into smaller math problems.
• 41:41
  Christine: And the way to solve it is you prove that these smaller math problems work and then you could[UM] how do you call it?
• 41:49
  Christine: Merge them? You could
• 41:52
  Brandon: Yeah. Combine the pieces.
• 41:54
  Christine: Yeah, you can you can combine all the pieces[UH] so you can solve like the bigger[UH] the top math problem.
• 42:02
  Christine: And [UM] basically w- these individual[UM] prover agents that prove the smaller sub-problems, they were using semantic search[UM] to find the relevant problems on this durable state layer that they could reuse and then they would submit their own proofs, these artifacts, back to the durable state layer so the next agents could find them to and build on it.
• 42:29
  Brandon: Yeah, yeah, I I think this this is a cool example of like four problems that have this kind of shape that you can decompose them and you don't know which pieces you wanna reuse but[UM] but you know that they might be there. Then having a way to search over those pieces effectively that's built into this durable state tool could be really effective.
• 42:58
  Christine: Like w- how do you encourage your agent?
• 43:06
  Christine: And what prefixes of the context will you re- will you reuse so the goal is really to be intentional about what information will you l- what for in- information deserves to occupy the active context of the agent doing this task.
• 43:24
  Christine: Alright, we hope that[UM] all the things that we've talked about will help you in your workflows and [UM] if you want to keep hearing more about emerging practices for building with agents then don't forget to subscribe and we hope to see you next time.
• 43:39
  Brandon: Thanks everyone.