Latest / Superlinear / Understanding Loops and Graphs in Modern Coding Agents

Transcript

Understanding Loops and Graphs in Modern Coding Agents · Superlinear

• 0:00
  Brandon Kase: When you ask Fable or GPT 5.6 Soul to complete a task, it feels like you're talking to one agent.
• 0:05
  Brandon Kase: But most of the time, several subagents are spawned to help.
• 0:08
  Christine Yip: Behind that single conversation, Claude and Codex may be spawning multiple subagents, giving each one its own context window, running them in parallel, and passing their results between one another.
• 0:19
  Christine Yip: In other words, you gave it one prompt and it'd build a graph.
• 0:22
  Brandon Kase: That graph may be one of the reasons the latest coding agents feel so capable.
• 0:26
  Brandon Kase: It also may be one of the reasons your tokens disappear so quickly.
• 0:30
  Christine Yip: Welcome to Super Linear, a podcast about emerging practices for building with AI agents.
• 0:36
  Brandon Kase: Each episode, we cut through the noise to find one new angle worth trying.
• 0:40
  Brandon Kase: Then we unpack why it matters, where it works, and how you can apply it in your own workflows so you can build more powerful things.
• 0:47
  Christine Yip: We're your hosts, Christine Yip and Brandon Case.
• 0:50
  Christine Yip: Hello everyone! Today we're talking about the hidden graphs behind Fable and GPT 5.6 Soul and why they might be one of the reasons you burn through your token limits so quickly.
• 1:03
  Christine Yip: So by the end of this episode, you'll understand what people mean when they talk about loops and graphs in a gentic engineering.
• 1:10
  Christine Yip: And more importantly, you'll understand what Fable and GPT 5.6 Soul are actually doing for you behind the scenes and how you can trade off token spend for performance.
• 1:19
  Brandon Kase: Before we explain what these graphs are, we're first unpacking how they build upon the familiar agent loop.
• 1:24
  Brandon Kase: We'll show you how modern harnesses turn those loops into graphs of subagents for you.
• 1:29
  Brandon Kase: And finally, show you how to think through these and other derivative graphs because reasoning with them can save you when your agents get slow, expensive, or stuck.
• 1:38
  Christine Yip: All right, let's dive into it.
• 1:40
  Christine Yip: I think maybe we can first start with how you and I came up with this topic.
• 1:46
  Brandon Kase: Yeah.
• 1:46
  Brandon Kase: So basically we saw a a tweet from Peter, Peter Steinberger, and I'm sure a lot of people saw this.
• 1:53
  Brandon Kase: Peter said, are we still talking loops or did we shift to graphs yet?
• 1:58
  Christine Yip: Yeah.
• 1:59
  Christine Yip: And it it is funny because uh it's funny how Peter can post one cryptic sentence and everyone immediately starts trying to decipher it.
• 2:08
  Christine Yip: But to be honest, I was curious as well.
• 2:10
  Christine Yip: So since Brandon is one of the most active people I know building with agents, I asked him what he thought Peter meant.
• 2:18
  Christine Yip: And that's how this conversation started.
• 2:20
  Christine Yip: So we do not know what Peter means.
• 2:22
  Christine Yip: But once we dived into this rabbit hole, we found it's pretty interesting how graphs actually describe the behaviors that much more sophisticated agent users have already been exploring for a while.
• 2:34
  Christine Yip: But what changed is that newer models and harnesses like Fable and GPT 5.6 Sol, it can increasingly create these structures automatically.
• 2:44
  Christine Yip: So previously, and the trans user might manually specify the workflow, but now Fable and GPT 5.6 Sol, they may decide on their own to create the sub-agents and organize the work as a graph.
• 2:55
  Christine Yip: To understand why this is a graph, first we need to understand the smallest unit it is built from, and that is the agent loop.
• 3:02
  Christine Yip: If you already know the agent loop well, feel free to skip ahead.
• 3:05
  Christine Yip: If not, then let's dive into loops.
• 3:07
  Brandon Kase: Yeah, so I I guess this section will try and go fast because a lot of you might already be familiar with this.
• 3:14
  Brandon Kase: So what is an agent loop?
• 3:21
  Brandon Kase: Well, uh, let's start on the left here.
• 3:25
  Brandon Kase: When you're interacting with cloud code, let's say, you type a message in, that's your prompt.
• 3:32
  Brandon Kase: Um that gets combined with a system prompt, maybe.
• 3:35
  Brandon Kase: And this gets sent to the actual LLM, the model under the hood, and then inference happens.
• 3:42
  Brandon Kase: And when the when inference occurs, either there will be just a simple result back and that gets sent to you, and that's the end.
• 3:52
  Brandon Kase: That's that's what we call the end of the turn.
• 3:54
  Brandon Kase: And then you get, you know, you can type something else.
• 3:58
  Brandon Kase: But sometimes, or many times, the LLM says, Hey, I need to get some extra information from the real world.
• 4:06
  Brandon Kase: And to do that, you make what's called a tool call.
• 4:09
  Brandon Kase: And literally what happens is the result of inference is something like, hey, I want to make a tool call that does X.
• 4:17
  Brandon Kase: This could be calling an MCP, it could be some tool that's built into the agent, like uh run something in bash, or it could be, well, I guess anything really.
• 4:29
  Brandon Kase: And and then that result uh produces some context.
• 4:34
  Brandon Kase: So like if you run a command in bash, there will be some output.
• 4:38
  Brandon Kase: If you if you do a web search, there will be the you know results of that search.
• 4:45
  Brandon Kase: And that context gets pushed to the end of the conversation that you've had so far, and then all of that gets fed back in as the prompt and the process repeats over and over again.
• 4:58
  Brandon Kase: Does that make sense?
• 4:59
  Brandon Kase: Yeah.
• 5:00
  Christine Yip: Perfect.
• 5:02
  Brandon Kase: Cool.
• 5:03
  Brandon Kase: So that's the agent loop.
• 5:06
  Brandon Kase: And I think it's important to note that uh this is not loops, you know.
• 5:13
  SPEAKER_00: What do you mean with loops?
• 5:15
  SPEAKER_00: This is an agent loop.
• 5:17
  Brandon Kase: So when when Peter says, are we still talking loops or did we shift to graphs yet?
• 5:22
  Brandon Kase: He didn't mean agent loops.
• 5:24
  Christine Yip: What did he mean then?
• 5:26
  Brandon Kase: He meant a loop on top of the loop.
• 5:29
  Brandon Kase: So uh so here's here's a few examples of that.
• 5:35
  Brandon Kase: First, we have the goal loop.
• 5:37
  Brandon Kase: This uh is also known as a Ralph loop.
• 5:39
  Brandon Kase: And you can see one of the steps is the agent loop over here.
• 5:45
  Brandon Kase: And then and then after the agent loop is done, so after there's the prompt, inference, tool calls, and eventually end of turn, before actually returning to you, um, there's a goal check that occurs.
• 5:59
  Brandon Kase: In in a traditional Ralph loop, that goal check happens in a script.
• 6:04
  Brandon Kase: In modern harnesses, you can do slash goal, and the the you know, Claude or Codex harnesses will keep track of this for you and and and do this check, which runs in a separate, uh like a separate inference call.
• 6:19
  Brandon Kase: And and that goal check either verifies, okay, this thing's really, really done, or it says, hey, even though the agent loop said we were done, actually we're not.
• 6:31
  Brandon Kase: So we need to keep going.
• 6:35
  Christine Yip: I see.
• 6:36
  Christine Yip: I guess I'm trying to understand why would why do we need the gold check?
• 6:42
  Christine Yip: Because if you gave it in your initial prompt in the agent loop, shouldn't shouldn't it what why would gold check work better than the prompt that you give in the agent loop?
• 6:51
  Brandon Kase: Yeah, that's a good question.
• 6:54
  Brandon Kase: So until recently, at least, the models they they give up pretty easily.
• 7:01
  Brandon Kase: They sometimes they cheat.
• 7:04
  Brandon Kase: So for example, if you say, okay, build me, build me a working calculator and make sure it works and don't stop until you're done.
• 7:13
  Brandon Kase: But then what would happen is at the end of the agent loop, sometimes the the agent would say, Okay, I did it.
• 7:21
  Brandon Kase: And it used to be that as a human, you had to be like, hey, this actually isn't working.
• 7:25
  Brandon Kase: Try again.
• 7:26
  Brandon Kase: It's not quite right.
• 7:27
  Brandon Kase: You lied.
• 7:28
  Brandon Kase: Keep going, keep going, keep going.
• 7:30
  Brandon Kase: If you remember, if if people were using codecs last maybe last year, towards the end of last year, there was a common pattern where we would type keep going in the in the prompt and cue them up, and you just say keep going, keep going, keep going, maybe 10 times before going to bed.
• 7:48
  Brandon Kase: Because uh effectively the agent loop would just stop and it wouldn't actually do what you wanted.
• 7:54
  Brandon Kase: So so this goal command and these Ralph loops are ways to automate that nudging to to make the agent keep going.
• 8:04
  Brandon Kase: And yeah, do we need this anymore?
• 8:08
  Brandon Kase: I don't think so.
• 8:09
  Brandon Kase: I think Fable and Soul, if if you're using the frontier, uh, the frontier models, they are actually pretty driven.
• 8:18
  Brandon Kase: And even without goal, they'll keep going for hours or days if they need to.
• 8:24
  Christine Yip: Interesting.
• 8:25
  Christine Yip: Okay, so this was the goal loop.
• 8:30
  Brandon Kase: So so this is this is one type of loop that you probably don't need too much anymore, but it's good to know about it.
• 8:37
  Brandon Kase: But there's another kind of loop that is still useful.
• 8:42
  Brandon Kase: And this kind of loop is called the auto research loop.
• 8:46
  Brandon Kase: And it's something that Christine knows a lot about because she built auto research at home when she was at NSU.
• 8:53
  Brandon Kase: Christine, you can tell us about this.
• 8:57
  Christine Yip: Yeah, yeah.
• 8:58
  Christine Yip: So uh indeed, as part of uh the previous startup that I was part of, we launched Auto Research at Home.
• 9:05
  Christine Yip: So Auto Research is a project that Andre Karpathi released, where you give an AI agent a machine learning experiment and you let it keep trying to improve the result on its own.
• 9:17
  Christine Yip: But what's especially interesting is that the same loop can be applied far beyond machine learning.
• 9:23
  Christine Yip: And the basic structure is actually pretty simple.
• 9:26
  Christine Yip: So first you choose a metric that you want to optimize.
• 9:29
  Christine Yip: In Karpathi's original experiment, that metric was validation loss, but it could also be software performance or throughput latency, and even things like newsletter signups or ad conversion.
• 9:44
  Christine Yip: So after you picked the metric that you want to optimize, you define the hyperparameters.
• 9:50
  Christine Yip: And basically, these are the knobs and levers that the agent is allowed to change.
• 9:54
  Christine Yip: In machine learning experiments, those might be model or training parameters.
• 9:58
  Christine Yip: On a website, they could be the headline or the layout, the call to action, the copy or the placement of the buttons.
• 10:06
  Christine Yip: And basically, the agents will come up with ideas and change those hyperparameters or a combination of them and then run an experiment and measure the result.
• 10:18
  Christine Yip: Based on the changes that the agent made, did the final metric improve or get worse?
• 10:23
  Christine Yip: And if it improved, the agent will keep the change.
• 10:26
  Christine Yip: If it didn't, then it will discard it.
• 10:29
  Christine Yip: And then it will come up with another idea and keep experimenting and continuing optimizing that metric that you chose.
• 10:36
  Christine Yip: And this is super helpful because basically sometimes when there are a million things that you can try, we humans only have limited time and energy to try them.
• 10:47
  Christine Yip: But an agent could basically keep trying it the whole time overnight and faster than us.
• 10:53
  Christine Yip: And then the agent can explore the search space much more faster than us.
• 10:57
  Christine Yip: So yeah, this is a little bit different from the goal loop that Brandon just described.
• 11:00
  Christine Yip: With a goal loop, you have one defined task and the agent keeps working until the task is finished.
• 11:05
  Christine Yip: With auto research, there's not really a final answer.
• 11:09
  Christine Yip: The agent is repeatedly trying to optimize that metric that you chose.
• 11:14
  Brandon Kase: So if we were to think about how this like it's almost so similar to the goal loop, right?
• 11:20
  Brandon Kase: So it's like there's an agent loop at the core.
• 11:23
  Brandon Kase: And then at the end of the turn, there's just one extra step.
• 11:27
  Brandon Kase: But in auto research, that extra step is like checking something and and kind of inventing the next step.
• 11:35
  Brandon Kase: Whereas in goal, we're just repeating the same thing that we did before.
• 11:39
  Brandon Kase: Is that is that right?
• 11:41
  Christine Yip: Exactly.
• 11:42
  Christine Yip: Exactly.
• 11:42
  Christine Yip: The agent loop, the agent loop comes up with the idea, what to try, and then the outer loop measures the results of executing that idea and then decides whether to keep it or discard it.
• 11:57
  Christine Yip: And then you go back to the agent loop where the agent comes up with a new idea again.
• 12:02
  Christine Yip: It's uh it's actually a really useful loop.
• 12:04
  Christine Yip: I have seen like so many people doing very interesting things with this.
• 12:08
  Christine Yip: For example, Toby, the CEO from Shopify, he ran the auto research loop on the liquid code base on his own machine, and he found a 53% speed up.
• 12:18
  Christine Yip: Oh, maybe we should show the tweet too.
• 12:20
  Christine Yip: He tweeted about it.
• 12:22
  Brandon Kase: All right, here's the tweet from Toby.
• 12:25
  Christine Yip: Exactly, this one.
• 12:27
  Christine Yip: So here he he shows us that he ran the auto research loop on the liquid code base from Shopify on his own machine, and he found a 53% speed up.
• 12:37
  Christine Yip: And I I don't know how he did it, but the way he presented it was super casual.
• 12:41
  Christine Yip: So you can do pretty cool stuff with the auto research loop.
• 12:45
  Christine Yip: And uh my previous startup and Sue partner with this research lab called Optimal Intellect to auto-research ways to optimize inference on Apple's neural engine.
• 12:55
  Christine Yip: And that process produced an approach that was more than six times faster than Apple's official baseline.
• 13:01
  Christine Yip: So there's actually, well, when Karpathi shared the auto research repo, uh, everyone went crazy with it.
• 13:09
  Christine Yip: Everyone was optimizing different things, and it was pretty impressive what you could do with it.
• 13:14
  Christine Yip: Yeah, including the auto research at home, of course, an experiment where that we ran during my time at NSU.
• 13:20
  Christine Yip: And we had more than 100 agents who collectively ran over 3,000 experiments.
• 13:24
  Christine Yip: So, yeah, it's not just a niche loop for training models, it's uh it's a general pattern for letting agents explore a large space of possible improvements.
• 13:33
  Brandon Kase: Awesome.
• 13:34
  Brandon Kase: So I think now we understand enough about agent loops and then loops, which are loops over agent loops.
• 13:41
  Brandon Kase: And so we can finally talk about graphs.
• 13:43
  Brandon Kase: So what's a graph?
• 13:46
  Brandon Kase: A graph is a collection of nodes and edges where edges can connect two nodes together.
• 13:54
  Brandon Kase: That's about it.
• 13:55
  Brandon Kase: It's a very abstract mathematical structure.
• 13:58
  Brandon Kase: So in this diagram, I've erased the words and we'll add it back later.
• 14:04
  Brandon Kase: But just so you can see here, there's you know a node at the top, there's nodes at the bottom, and then there's edges that connect the top node to the bottom nodes.
• 14:12
  Brandon Kase: And and here the edges are directed.
• 14:14
  Brandon Kase: That means that they have an arrow pointing in one direction.
• 14:18
  Brandon Kase: So, so when Peter said, are we still talking loops, or did we shift to graphs yet?
• 14:25
  Brandon Kase: What came to my mind is hey, these loops, they're graphs.
• 14:32
  Brandon Kase: I mean, look at them, right?
• 14:33
  Brandon Kase: There's a node, there's a node, there's edges.
• 14:37
  Brandon Kase: Another way to view this is it's a graph of agents interacting with each other.
• 14:43
  Brandon Kase: So there's an agent doing the agent loop, and there's another agent doing the goal check, and there's arrows that represent the interaction between the agents.
• 14:53
  Brandon Kase: So the agent loop when it's done, you know, sends some kind of signal to the goal check, the goal check sends some kind of signal back to the agent loop to tell it to keep going or to stop.
• 15:07
  SPEAKER_00: All right.
• 15:08
  SPEAKER_00: Now, if we we can also What can we also do?
• 15:16
  Brandon Kase: Well, we should talk about subagents, actually.
• 15:18
  Christine Yip: When I was when I read Peter's graph, uh uh Peter's tweet about graphs, I think the first thing that came up to my mind was subagents.
• 15:29
  Brandon Kase: So what is a subagent?
• 15:32
  Brandon Kase: So if we look at the agent loop, there's a prompt, there's inference, there's maybe tool calls.
• 15:40
  Brandon Kase: A subagent is an agent loop that's spawned from a tool call.
• 15:46
  Brandon Kase: So the tool call might be spawn subagent, for example.
• 15:51
  Brandon Kase: And and then what happens is another agent loop runs with fresh context until the end of its turn, and then that information can get sent back to its parent.
• 16:05
  Brandon Kase: And in modern subagents and subagent workflows, the the parent agent can actually look at the subagent as it's running.
• 16:12
  Brandon Kase: But the important thing is it's a fresh agent loop that spawns with a new context window.
• 16:17
  Brandon Kase: So starting from scratch.
• 16:19
  Christine Yip: That's the definition.
• 16:21
  Christine Yip: You know, that's what we mean when we say subagents.
• 16:23
  Brandon Kase: Yes.
• 16:24
  Christine Yip: New session with new context window.
• 16:27
  Brandon Kase: Yes.
• 16:28
  Brandon Kase: But but somehow, you know, connected to the the parent agent loop that spawned it.
• 16:34
  Brandon Kase: There's a couple reasons that um subagents are useful.
• 16:38
  SPEAKER_00: Yeah.
• 16:41
  Brandon Kase: I guess we can we can go through those.
• 16:44
  Christine Yip: Yeah, I was about to ask you.
• 16:46
  Christine Yip: Like maybe before we continue, what what maybe we can just pause a little bit and why subagents?
• 16:53
  Brandon Kase: Why subagents?
• 16:54
  Brandon Kase: Why subagents?
• 16:57
  Brandon Kase: The the context window is finite.
• 17:00
  Brandon Kase: And by that I mean there's only so so much that you can put in your prompt.
• 17:05
  Brandon Kase: And as the agent loop progresses, the prompt keeps getting bigger and bigger and bigger.
• 17:10
  Brandon Kase: So that's that's one reason.
• 17:11
  Brandon Kase: And even even as as context windows grow and uh we can have one million tokens in the context, uh there's this uh phenomenon where the models get dumber the more the context window is full.
• 17:26
  Christine Yip: So context pollution.
• 17:29
  Brandon Kase: Yeah, this is sometimes called context pollution, uh where you can think of it like the agents getting distracted.
• 17:37
  Brandon Kase: And so uh it's nice to be able to basically erase some things from working memory temporarily and have some focused attention on a single task.
• 17:51
  Brandon Kase: And and that's that's one thing that that subagents enable us to do.
• 17:55
  Christine Yip: Oh and what what is the downside of subagents?
• 17:59
  Brandon Kase: There's this there's this concept of of caching when you're making requests to inference providers where it's it's cheap to extend a conversation.
• 18:10
  Brandon Kase: And if you start from scratch, it's more expensive.
• 18:12
  Brandon Kase: So so you'll you'll spend you'll spend more tokens if you do the same work in subagents that would be otherwise fine to just do in one agent loop.
• 18:22
  Christine Yip: Because you have to build that context from scratch in that new session of the subagent.
• 18:28
  Brandon Kase: Yes, exactly.
• 18:31
  Brandon Kase: So so that's that's one issue.
• 18:33
  Brandon Kase: Another is, I guess another that's another thing that's a benefit, but also potentially a negative, is that you can run multiple subagents at the same time.
• 18:43
  Brandon Kase: So this is a benefit because you can parallelize and in the same wall clock time, if there's a task that makes sense that you can break into pieces and do concurrently, um, it actually happens concurrently.
• 18:56
  Brandon Kase: The downside is these concurrent subagents aren't sharing context.
• 19:01
  Brandon Kase: So to the extent that they need the same information, you're paying for those tokens more than once.
• 19:06
  Brandon Kase: And essentially, you know, if you're running five subagents in parallel, you're paying five times as much, or you're spending your plan five times faster.
• 19:13
  Christine Yip: That can add up pretty quickly if they all have to rebuild that context from scratch.
• 19:19
  Brandon Kase: Yes, exactly.
• 19:22
  Brandon Kase: Now that we understand graphs, we understand subagents, we can show an example of a subagent graph.
• 19:32
  Brandon Kase: A simple one.
• 19:35
  Brandon Kase: So this is a research, uh, a research fan-out graph.
• 19:40
  Brandon Kase: This is uh like if you said, Hey, Claude, spawn five researcher subagents to focus on a particular part of how to make my game, and uh and then come back to me with synthesized results.
• 19:59
  Brandon Kase: So, what will happen is the orchestrator will figure out, okay, I'll have one researcher look for the game engine, one researcher will look at game design, one researcher will look at anti-cheating, one researcher will look at multiplayer.
• 20:12
  Brandon Kase: These researchers all run in parallel, all with their own separate context.
• 20:17
  Brandon Kase: Uh, they might search the web, they'll fill up their context potentially, and then at the end synthesize some result and send it back.
• 20:28
  Brandon Kase: Does that make sense?
• 20:30
  SPEAKER_00: Yeah.
• 20:32
  Brandon Kase: And the the point here is like, what is what did we do, right?
• 20:37
  Brandon Kase: Like this graph, it's a way to visualize and understand what's happening when we are working with subagents doing something.
• 20:48
  Brandon Kase: Because it very quickly gets confusing.
• 20:52
  Brandon Kase: This is a simple example, but you'll we'll we'll show a couple more examples where without without this graph structure, without being able to draw this out, either like by either actually looking at it or thinking about it in our heads, it's hard to reason about what's happening.
• 21:07
  Christine Yip: When we were talking about context window, you mentioned that the context window is one million, but in practice, you don't really use it all, right?
• 21:17
  Christine Yip: So when I'm just I'm just curious, like when do you clear the context?
• 21:23
  Christine Yip: When do you just let it run?
• 21:27
  Christine Yip: Do you just let the context window get filled all the way up and then let it automatically compact and then you go on?
• 21:33
  Christine Yip: Or do you sometimes manually compact before it reaches 100%?
• 21:39
  Christine Yip: Just curious about your approach.
• 21:41
  SPEAKER_00: What do you do there?
• 21:48
  Brandon Kase: I will if if I'm working on some really hard task and I see the agent get dumb, then I'll I'll stop and clear the context.
• 21:59
  Brandon Kase: But my Most of the time these days, the latest models and their harnesses are really good at compacting.
• 22:06
  Brandon Kase: And so I actually just don't think about it.
• 22:09
  Christine Yip: Oh wow.
• 22:10
  unknown: Yeah.
• 22:10
  Christine Yip: What a luxury.
• 22:12
  Christine Yip: You must you must be token rich.
• 22:14
  Christine Yip: You're definitely not token poor.
• 22:16
  Brandon Kase: Yeah.
• 22:16
  Brandon Kase: It also spends it also spends the plan faster.
• 22:20
  SPEAKER_00: Yeah.
• 22:22
  Brandon Kase: The codex plan is much more generous than the clawed one.
• 22:25
  Brandon Kase: And I mostly use codex.
• 22:27
  Christine Yip: Oh, right.
• 22:28
  Christine Yip: I I generally use clawed code because I just have my memory system building clawed code.
• 22:34
  Christine Yip: And I just notice how fast I run through that five hour limit.
• 22:40
  Christine Yip: But I I noticed uh when I cleared a context window, like even when it's just between 150 and 200k, when I start, when there's a good cutoff point where all my tasks are done.
• 22:53
  Christine Yip: And when I clear my context window around that point, I run through my tokens much, much slower.
• 23:01
  SPEAKER_00: Yeah.
• 23:02
  Christine Yip: So uh and yeah, I don't really have proof, but I did see that other builders in the space are also clearing their context window around that point.
• 23:11
  Christine Yip: So that was interesting to see that I wasn't the only one.
• 23:13
  Christine Yip: But yeah, that could be yeah, I guess if you're token rich, you don't need to worry about it.
• 23:18
  Christine Yip: If you're a token four like me, it could be uh a good way to run out of run out of tokens.
• 23:24
  Brandon Kase: Yeah.
• 23:25
  Christine Yip: Last fast slower.
• 23:28
  Brandon Kase: It depends.
• 23:29
  Brandon Kase: Like uh if I'm doing something that's really really hard, then I will I will clear context at boundaries just to have like the sharp fresh attention and to prevent distraction, prevent the context pollution.
• 23:46
  Brandon Kase: But I I haven't been doing that to save save tokens.
• 23:50
  Brandon Kase: I haven't needed it.
• 23:52
  Christine Yip: But you know We definitely also notice it's much better.
• 23:55
  Christine Yip: It makes less uh mistakes.
• 23:57
  Brandon Kase: Yeah.
• 23:58
  Christine Yip: And it's more likely to do that aging loop right and know when the NF turn is.
• 24:03
  Christine Yip: The longer it is, uh more often it stops while it's not good enough.
• 24:08
  Brandon Kase: We we came across this this tweet by by Sichi.
• 24:12
  Brandon Kase: My favorite new way of using Fable, ask it to write a plan, and then please get second opinions from Codex CLI and Kimi CLI.
• 24:20
  Brandon Kase: Revise your plan with findings, etc.
• 24:23
  Brandon Kase: So this is similar to what I do in my workflow.
• 24:27
  Brandon Kase: I I have in order to save tokens in my codex plan, I have codex ask claude for reviews.
• 24:34
  Brandon Kase: So it's a way to spread if you have if you have plans on Claude and Codex, it's a way to use both of them on hard problems.
• 24:40
  Christine Yip: Why are you using Claude for plans?
• 24:42
  Christine Yip: That's interesting because I'm doing it the other way around.
• 24:45
  Christine Yip: I'm asking Claude to use Codex for tasks for for building.
• 24:52
  Brandon Kase: Yeah.
• 24:52
  Brandon Kase: So sorry, I was using uh, I guess in this, in Sichi's example, he's talking about a plan and then reviewing a plan.
• 24:59
  Brandon Kase: I was saying when I do my engineering and I review my engineering, I had the same kind of pattern.
• 25:06
  Brandon Kase: For planning, I'll also mostly use Claude as well.
• 25:10
  Brandon Kase: But you know, maybe maybe it's hard for people to visualize what we're talking about, right?
• 25:14
  Brandon Kase: Because we're we're just saying things and and that's the point of graphs.
• 25:18
  Brandon Kase: You know, look, we can draw it.
• 25:20
  Brandon Kase: This is a drawing, this is a a graph of the the setup that C Chi talked about.
• 25:27
  Brandon Kase: So at the top here you see the planner agent.
• 25:30
  Brandon Kase: The planner agent delegates to Kimmy and GPT.
• 25:34
  Christine Yip: Yeah, I would imagine an arrow back from GPT soul to the main plan agent.
• 25:38
  Christine Yip: Yeah, I guess that's what better, yeah.
• 25:42
  Brandon Kase: But anyway, this is this is this is the point here.
• 25:45
  Brandon Kase: It's like um we can draw pictures to help understand what's going on.
• 25:48
  Brandon Kase: And again, this one maybe is is is pretty easy to follow, you know.
• 25:54
  Brandon Kase: But let's talk about another one that's harder to follow.
• 25:59
  Brandon Kase: And this is this is one that I use in my in my workflow when I'm building that that Game Boy LLM.
• 26:04
  Brandon Kase: Can I can I show it?
• 26:06
  Brandon Kase: Yeah, of course.
• 26:08
  Brandon Kase: All right, so see, this is complicated.
• 26:10
  Brandon Kase: So let me explain.
• 26:12
  Brandon Kase: Let me explain.
• 26:13
  Brandon Kase: So there's a manager that orchestrates the work.
• 26:16
  Brandon Kase: And and then what I do is I have an engineer that always is making progress on uh some big chunk of work.
• 26:23
  Brandon Kase: So I I do work in in RFC chunks.
• 26:27
  Brandon Kase: So you think of it like an epic or a project, and that can be broken down into maybe like 20 to 40 tasks.
• 26:34
  Brandon Kase: So the engineer is just working through those tasks one at a time and it's not stopping.
• 26:39
  Brandon Kase: It's always picking up the next task.
• 26:41
  Brandon Kase: I use beads, so it's pick up the next bead, and it never stops.
• 26:45
  Brandon Kase: Whenever, whenever a bead is done, the reviewers are notified and the manager manages the reviewers.
• 26:52
  Brandon Kase: The reviewers, like I said, use different models, different agents.
• 26:57
  Brandon Kase: So Codex, Claude, and Kimmy.
• 26:59
  Brandon Kase: And and and they they are actually slower than the engineering progress, typically.
• 27:05
  Brandon Kase: So they give review feedback in a queue.
• 27:08
  Brandon Kase: And that feedback eventually makes it to the engineer and it'll interrupt the engineer.
• 27:13
  Brandon Kase: If the engineer has review feedback, it'll go and do that feedback.
• 27:16
  Brandon Kase: But otherwise, it'll just keep making progress.
• 27:18
  Brandon Kase: And the reviewers will they'll say, okay, the engineer did three tasks.
• 27:21
  Brandon Kase: I'll review all three of those tasks.
• 27:23
  Brandon Kase: And then eventually, all the work is done, all the work is reviewed.
• 27:28
  Brandon Kase: We open a pull request.
• 27:30
  Brandon Kase: The human in the loop can actually go and look at it at this point.
• 27:33
  Brandon Kase: This is where I decided to be the human.
• 27:36
  Brandon Kase: Like I didn't want to see every detailed tiny piece of work, but I wanted to see at the epic level, at this project level.
• 27:43
  Brandon Kase: And then there can be some retro or learning that improves this workflow for the agent.md or the, you know, any of the guides or sensors in the harness that we uh in the harness engineering sense, this is something we talked about in our episode zero.
• 28:02
  Christine Yip: Right.
• 28:02
  Christine Yip: Shifting left for agents.
• 28:04
  SPEAKER_00: Mm-hmm.
• 28:06
  unknown: Yeah.
• 28:06
  Christine Yip: If anyone's wondering what we're talking about, go check out episode zero.
• 28:10
  Christine Yip: That one is also really interesting.
• 28:13
  Brandon Kase: Yeah.
• 28:13
  Brandon Kase: Thanks, Christine.
• 28:14
  Christine Yip: So are you saying like the the reviewers are working in parallel while the engineer agents are building?
• 28:22
  Christine Yip: Do do the reviewers wait for your engineer?
• 28:25
  Brandon Kase: No.
• 28:25
  Christine Yip: Do they review every task, every bead?
• 28:28
  Brandon Kase: Uh well, the review in in my project anyway, reviewing is slower than engineering.
• 28:34
  Brandon Kase: So what happens is uh the reviewers will they'll they'll do all of the beads that have been done, all of the tasks that have been done since the last time they reviewed, all at once.
• 28:46
  SPEAKER_00: All at once.
• 28:47
  Brandon Kase: I I found that was a good balance between spending tokens and getting enough feedback uh to the engineer so that the the implementation I was working on is actually good.
• 29:00
  Christine Yip: Right.
• 29:01
  Christine Yip: So sometimes the engineer agent would already be at working on task four and the reviewers are still reviewing task one.
• 29:08
  Christine Yip: Is that what I'm hearing?
• 29:10
  Brandon Kase: Exactly.
• 29:10
  Brandon Kase: And then and then maybe when the reviewers are done, they would say task two and three are already done.
• 29:17
  Brandon Kase: So I'll start working on those while the engineer is fixing my feedback from task one.
• 29:21
  Christine Yip: Right.
• 29:22
  Christine Yip: Is it a concern at all that that maybe the engineer agents need to do a lot of rework because of feedback?
• 29:29
  Christine Yip: Uh because of feedback on task one while they already like did two, three, and four.
• 29:36
  Brandon Kase: You know, I found because I also do the the planning, planning iteration cycle, where I write a plan, I work with the agents, I get reviews from different agents, um I iterate a lot.
• 29:50
  Brandon Kase: This is the time where major architectural changes and and like really big critical changes are caught.
• 29:58
  Brandon Kase: And by the time that the engineering happens, the issues that the reviewers are catching are still important, but they're not, they don't invalidate all the work.
• 30:09
  SPEAKER_00: Right.
• 30:10
  Brandon Kase: In the sense that like Okay.
• 30:12
  Christine Yip: Maybe they're edge cases.
• 30:14
  Brandon Kase: Exactly.
• 30:15
  Brandon Kase: Exactly.
• 30:15
  Brandon Kase: Edge cases or a security issue or something like that.
• 30:19
  Christine Yip: That's interesting.
• 30:20
  Christine Yip: So because you have a workflow, because your in your workflow you spend a lot of time on planning, it makes sense for you to design or build your agent graph differently because you know that you will probably not find like huge errors.
• 30:36
  Christine Yip: So you could build faster by letting your engineer agents run faster than your Vuhio agents.
• 30:41
  Brandon Kase: Exactly.
• 30:42
  Brandon Kase: And very interesting.
• 30:44
  Brandon Kase: And you know, you you might look at this and say, like, wow, this is so complicated.
• 30:48
  Brandon Kase: Like, how the heck does Brandon set this up?
• 30:52
  Brandon Kase: Like, what what magic do we need to do to make this happen?
• 30:55
  Brandon Kase: And it's so, it's not magic.
• 30:58
  Brandon Kase: I'll just show you.
• 30:59
  Brandon Kase: You literally just tell, you just tell your harness.
• 31:04
  Brandon Kase: Codex and Cloud both are good at this.
• 31:07
  Brandon Kase: Well, I'll I'll run you through an example.
• 31:10
  Brandon Kase: We are looking at, and then I paste the RFC.
• 31:13
  Brandon Kase: You are the manager, you orchestrate the engineers and reviewers.
• 31:17
  Brandon Kase: Your objective is to complete all the tasks and then open a PR.
• 31:21
  Brandon Kase: Keep going until it's done.
• 31:23
  Brandon Kase: You should always have an engineer subagent running in the background making progress.
• 31:28
  Brandon Kase: As soon as it completes a bead, it should pick up pending review feedback, et cetera.
• 31:33
  Brandon Kase: Reviewers chase the engineer.
• 31:35
  Brandon Kase: Reviewers can review up to three beads at a time, depending how far behind the engineer we're getting.
• 31:40
  Brandon Kase: It's okay if the reviewers are slower than the engineer, uh, et cetera.
• 31:43
  Brandon Kase: So you you just describe what you want.
• 31:46
  Brandon Kase: And the point is that the harnesses are smart enough to orchestrate these subagents and create this graph.
• 31:52
  Brandon Kase: And it might help for you to draw a picture or for you to ask your harness to draw you a picture as you're iterating and finding a workflow that works for you.
• 32:01
  Brandon Kase: But these these harnesses are so powerful.
• 32:03
  Brandon Kase: They they use, at least Claude uses dynamic workflows now.
• 32:07
  Brandon Kase: And I I think that that codex does as well.
• 32:10
  Brandon Kase: Uh, it uses something similar, which is like a code mode mechanism for uh working with subagents.
• 32:17
  Brandon Kase: And this is also something we talked about in episode zero.
• 32:20
  Christine Yip: Very cool.
• 32:20
  Christine Yip: So basically, uh you just told your agents in plain English how you wanted your graph to look like, and then it just build it for you.
• 32:28
  Brandon Kase: Exactly.
• 32:29
  Christine Yip: I think you also sent out a tweet recently where you were like, Oh, I'm so surprised Fable and GBT Soul can do it on their own now.
• 32:39
  Christine Yip: So, what does that mean?
• 32:40
  Christine Yip: Like, do we do we not need to tell GBT 5.6 Soul and Fable to set up these graphs anymore?
• 32:48
  Brandon Kase: I tweeted something on this topic.
• 32:51
  Brandon Kase: Um basically after I tried Soul Ultra, I noticed that Soul Ultra and Codex and Fable and Claud Code both orchestrate subagent workflows automatically for you.
• 33:04
  Brandon Kase: And they seem to do it really well.
• 33:06
  Brandon Kase: And I thought, you know, maybe maybe we don't need to do as much orchestration ourselves.
• 33:12
  Brandon Kase: And if we do, maybe it'll interfere with the harnesses choices that they're making that they've been trained on.
• 33:17
  Brandon Kase: So yeah, and you know, it's it's it's hard to tell, right?
• 33:21
  Brandon Kase: I I think I think by default, if you're using the most frontier models in their native harnesses, you should, you should try the subagent graphs that they produce for you first.
• 33:32
  Brandon Kase: But you know, as we talked about before, if you want to tune, if if if you care about your token spend or the parallelization because you want to get something done faster and you want to spend more money, then you should know that this is a knob that you're able to turn yourself.
• 33:46
  Christine Yip: You mentioned that Fable and GBT 5.6 Soul can now spawn these subagents themselves.
• 33:53
  Christine Yip: Do we need to worry about our token burn?
• 33:56
  Christine Yip: And is so is that a good thing or a bad thing?
• 33:59
  Brandon Kase: We've seen evidence that lots of people are seeing their plans get spent really fast.
• 34:05
  Brandon Kase: And I think a big reason this is happening is because the models are more aggressively spawning subagents.
• 34:10
  Brandon Kase: And they're doing this because they're more effective.
• 34:14
  Brandon Kase: Like when it works.
• 34:15
  Brandon Kase: Yeah.
• 34:16
  Brandon Kase: So so it's a trade-off.
• 34:17
  Brandon Kase: If you're if you're finding that your plan's going down too fast, maybe you can accept worse performance, you know?
• 34:24
  Christine Yip: Yeah.
• 34:25
  Christine Yip: One thing that's coming to my mind is there's also an opportunity when you spawn sub-agents to use cheaper models for certain tasks.
• 34:32
  Christine Yip: For example, for planning and strategizing, you might want to use Fable or GPT 5.6 soul.
• 34:39
  Christine Yip: But if an another for another task where Fable already planned the whole, the whole task, the whole spec, everything is spec'd out, then you could give it to Opus or even Sonnet to execute it.
• 34:51
  Brandon Kase: And now I want to show one more knob.
• 34:57
  Brandon Kase: Or one more way that you can decide how to turn your knobs.
• 35:02
  Brandon Kase: And that's with an execution trace.
• 35:06
  Brandon Kase: Okay.
• 35:07
  Brandon Kase: So an execution trace is another kind of graph.
• 35:10
  Brandon Kase: And so this is another tool that you can have in your toolbox for helping you and your agent understand what's going on and making decisions and making changes.
• 35:18
  Brandon Kase: What is an execution trace?
• 35:20
  Brandon Kase: It's a it's a visualization of what actually happened in the subagent graph.
• 35:27
  Brandon Kase: So if I go back to the subagent graph, for example, this shows how different pieces are connected together.
• 35:34
  Brandon Kase: And it's a template that you can kind of imagine is stamped every time that a new project is ran.
• 35:40
  Brandon Kase: But it doesn't tell us, like in this particular project, the engineer did three tasks and then the review went and the review ran into an issue.
• 35:49
  Brandon Kase: But something, an execution trace.
• 35:52
  Brandon Kase: This is a this is like a simple diagram just to get your head in the to understand what we're talking about.
• 35:57
  Brandon Kase: Don't actually look at the details too much here.
• 35:59
  Brandon Kase: But something like this, where you have time at the bottom.
• 36:03
  Brandon Kase: So what happens over time?
• 36:04
  Brandon Kase: And then here on the on the y-axis, we're showing like what each of the different agents are doing at that time.
• 36:11
  Brandon Kase: Or, you know, it's sort of one agent's view of the world.
• 36:14
  Brandon Kase: And then in the middle, it's showing who's doing what at what time.
• 36:18
  Brandon Kase: In in practice, you might want to make this an HTML page with your agent so that you can like click on the different nodes and see what's happening.
• 36:25
  Brandon Kase: You you might want to use there's a tool from from Doodle Stein called C A S S CAS, which you can ask your agent to use to mine your session history to uh reproduce these, to sort of build these uh traces for you.
• 36:41
  Brandon Kase: But yeah, this is this is a a tool you can use to learn more about what's happening and then decide to adjust your graph.
• 36:51
  Christine Yip: For example, like since it's since it's uh since you see what subagents are doing over time, I guess you could see what each subagent is doing and where the bottlenecks are, which one, which task is taking longer than the other.
• 37:07
  Christine Yip: Is it how you would use it?
• 37:10
  Brandon Kase: Exactly.
• 37:10
  Brandon Kase: Exactly.
• 37:11
  Brandon Kase: Which which tasks are using more tokens than we expect, which tasks are failing.
• 37:17
  Brandon Kase: All of this kind of information you you can surface and and from that you can mine learnings and insights with your agent to then apply to improve your harness and shift things more left.
• 37:30
  Christine Yip: Yeah.
• 37:30
  Christine Yip: I think even without like really plotting this uh graph, if just having this mental model for yourself, knowing which subagents are active, what they are working on, and what their relationships are could help a lot with building or customizing your agent graph more.
• 37:46
  Brandon Kase: Peter asked, are we still talking loops or did we shift to graphs?
• 37:50
  Brandon Kase: I guess you know, loops, they they didn't really disappear, but they became part of graphs, of this mental model of graphs that we had.
• 37:57
  Brandon Kase: And this is a a tool that we need.
• 38:00
  Brandon Kase: This is something we need to think about in our heads because modern harnesses are increasingly connecting all these different subagents in intricate ways with loops potentially into graphs.
• 38:11
  Christine Yip: Right.
• 38:12
  Christine Yip: So the way that they're creating structures is becoming more complex.
• 38:15
  Christine Yip: Previously, you could just describe it very easily and simply with a loop.
• 38:19
  Christine Yip: And now they're creating, they're working with multiple agents, their dependencies, branches, parallel work, reviews, and now it's uh much easier to describe it with a graph, or you have to describe it with a graph.
• 38:31
  Christine Yip: Interesting.
• 38:31
  Christine Yip: Well, so I guess uh the takeaway is when you send one prompt, the harness may create an entire graph behind the interface.
• 38:39
  Christine Yip: That graph is part of why Fable and GPT 5.6 Soul can handle much more ambitious work, but it is also one place where your tokens are going.
• 38:49
  Christine Yip: You do not need to manually design every graph, but when a run becomes slow, expensive, or stuck, like the examples that Brendan mentioned, understanding the graph helps you identify bottlenecks, judge where extra inference is buying useful performance, or maybe remove unnecessary work, and then you can ask the agent to redesign the workflow.
• 39:10
  Christine Yip: All right.
• 39:10
  Christine Yip: Well, we hope you got something out of this episode.
• 39:13
  Christine Yip: We hope that you can you now understand what loops and graphs are.
• 39:17
  Christine Yip: You understand what Fable and Soul, uh, GPT 5.6 Soul are actually doing for you behind the scenes.
• 39:23
  Christine Yip: And thank you for listening.
• 39:25
  Christine Yip: I'll see you next time.
• 39:26
  Brandon Kase: Thanks.
• 39:27
  Brandon Kase: Bye.