Latest / Superlinear / 4 Emerging Practices in Agentic Engineering

Transcript

4 Emerging Practices in Agentic Engineering · Superlinear

• 0:00
  Brandon Kase: Essentially, LLMs are experts in everything, but by default, you get the average.
• 0:08
  Christine Yip: People building things in a space and they're like, I just built this in two days using AI.
• 0:13
  Christine Yip: Did they build it from scratch or did they use an Oracle for it?
• 0:16
  Christine Yip: Using an Oracle could be a shortcut.
• 0:18
  Brandon Kase: If something is triggering feedback, a self-correction, then it means that the model did something wrong.
• 0:24
  Brandon Kase: It's better if the model just doesn't do it wrong in the first place.
• 0:27
  Christine Yip: If you want to have good taste, you definitely also want to have deeper understanding.
• 0:33
  Christine Yip: Welcome to Super Linear, a podcast about emerging practices for building with AI agents.
• 0:40
  Brandon Kase: Each episode, we cut through the noise to find one new angle worth trying.
• 0:44
  Brandon Kase: Then we unpack why it matters, where it works, and how you can apply it in your own workflows so you can build more powerful things.
• 0:52
  Christine Yip: We're your hosts, Christine Yet and Brandon Case.
• 0:57
  Christine Yip: Hey everyone.
• 0:58
  Christine Yip: Today we're breaking down four emerging practices that Brandon and I have noticed across the AI space.
• 1:05
  Christine Yip: By the end of this episode, you'll start thinking of your harness more as an evolving system, one that is constantly shifting left.
• 1:13
  Christine Yip: We'll explain what that means later in this episode.
• 1:16
  Christine Yip: You'll also have a greater appreciation for using precise expert language.
• 1:21
  Christine Yip: You'll think carefully before defaulting to MCP, and you will understand when you should reach for an Oracle as a tool for building more effectively.
• 1:31
  Christine Yip: We re-recorded these takeaways right after the most recent AI native DEF CON in London.
• 1:37
  Christine Yip: And we hope they give you a few new ideas to try in your own workflows.
• 1:42
  Christine Yip: So the first emerging practice that we're going to talk about today was mentioned by Ryan Lopopolo from OpenAI at the AI Native DEF CON where you were last week in London.
• 1:52
  Christine Yip: So, and that was leverage shifting left for agents.
• 1:56
  Christine Yip: Yeah.
• 1:56
  Christine Yip: Is that right?
• 1:57
  Christine Yip: Okay.
• 1:57
  Christine Yip: Can you tell us a little bit more about this emerging practice related to shifting left for agents?
• 2:03
  Brandon Kase: Yeah.
• 2:03
  Brandon Kase: So I I think this is interesting.
• 2:05
  Brandon Kase: This was one line that Ryan said sort of offhand during his presentation, which surprised me, and I had to ask a question about it at the end.
• 2:14
  Brandon Kase: So I want to take you all through the story of this point because I think it's interesting.
• 2:20
  Christine Yip: What is the sentence that he mentioned?
• 2:22
  Brandon Kase: Well, let me share my screen because I I tweeted it so I can uh just show you.
• 2:29
  Brandon Kase: This wasn't his exact words, and maybe I misquoted him a little bit.
• 2:32
  Brandon Kase: But but the the sentence was something like you don't need to shift left as much, which was a good practice in software engineering, because harness engineering, which is like this new thing that a few people have talked about, which we can define a little better.
• 2:47
  Christine Yip: Did that caught your attention because it's very controversial?
• 2:50
  Christine Yip: Was shifting left like super important in software engineering?
• 2:54
  Brandon Kase: Yeah.
• 2:54
  Brandon Kase: Shifting left is um generally seen as a good practice.
• 2:59
  Brandon Kase: And in particular, it's a practice that I thought that Ryan really took advantage of in the stuff that he talks about in his presentations and and his blog posts.
• 3:12
  Brandon Kase: Um so that's that's in particular why this was very surprising to me.
• 3:16
  Brandon Kase: Um and also I I didn't believe it uh until I until I dug in and and sort of understood what what he was saying.
• 3:23
  Christine Yip: Interesting.
• 3:24
  Christine Yip: Before we dive into it, could you like explain maybe like in a few sentences what is shift left?
• 3:30
  Christine Yip: So we're all on the same page.
• 3:31
  Brandon Kase: Yeah, so I'm gonna pull up this nice image.
• 3:34
  Brandon Kase: Shifting left, uh, let's see.
• 3:36
  Brandon Kase: Well, think about time on the x-axis, and uh what we're measuring is the time across the software or product development lifecycle.
• 3:45
  Brandon Kase: So, you know, you start with understanding the user's needs, requirements, gathering, architecture, design, coding, and then there's all sorts of testing, and then the product is released.
• 3:56
  Brandon Kase: The idea of shifting left is taking tests or errors that happen on the right hand side and push them as left as possible.
• 4:05
  Brandon Kase: And in the context of modern software engineering practices that I guess I believe, and and I think Ryan exemplifies in his some of the content that he's created, it's maybe focused on the testing side and it's moving tests that are slower and that run less often to tests that can run really quickly and at very close to the point that you're writing the software.
• 4:28
  Brandon Kase: You know, this intuitively is important with coding agents because coding agents are writing a lot more code and they're doing it fast.
• 4:35
  Brandon Kase: And so the, you know, you'd you'd think, you'd expect, and it's generally good practice, to get feedback to those coding agents as quickly as possible.
• 4:45
  Christine Yip: Because then they don't go down the wrong route for too long and waste tokens.
• 4:50
  Christine Yip: Exactly.
• 4:50
  Christine Yip: Okay.
• 4:50
  Christine Yip: But why does Ryan say like that's not as important anymore?
• 4:54
  Christine Yip: It still makes sense to me to do it.
• 4:56
  Brandon Kase: Yeah.
• 4:56
  Brandon Kase: So so so Ryan is talking about this in the context of of harness engineering, which is a term that he wrote a long blog post about in February, and this went kind of viral, which kind of described how his team is creating software with coding agents.
• 5:11
  Brandon Kase: And I think this was again further described and elaborated upon Bergitta Burkler, who also gave a keynote presentation at the conference last week on the same topic.
• 5:22
  Brandon Kase: And so I guess I want to just describe harness engineering a little bit to give some context to help understand why Ryan claims that shifting left is less important.
• 5:32
  Brandon Kase: Okay.
• 5:33
  Brandon Kase: Yeah.
• 5:34
  Brandon Kase: So harness engineering, um, well, it's about engineering the harness.
• 5:38
  Brandon Kase: And the harness refers to everything that's not the model in the context of the coding agent.
• 5:44
  Brandon Kase: So the model is the thing that we're doing inference on.
• 5:47
  Brandon Kase: Then part of the harness is the actual software that you're using to control the coding agent, and maybe it's plugins.
• 5:53
  Brandon Kase: So for example, Cloud Code or Codex or Pi or Open Code.
• 5:57
  Brandon Kase: Uh, and it's also, you know, which plugins you have installed, what your system prompt is, these kinds of things.
• 6:03
  Brandon Kase: And then it also encompasses what uh Brigitte calls the user harness, which are made up of, here's another chart, are made up of feed forward and feedback controls, which she calls guides and sensors.
• 6:16
  Brandon Kase: So the guides are things that you put in markdown files that feed into the context of the agent ahead of time, or or maybe the the code of your project.
• 6:26
  Brandon Kase: So it's anything that goes in the context before the coding agent starts writing code.
• 6:31
  Christine Yip: So like uh agents.md, for example.
• 6:35
  Brandon Kase: Yes.
• 6:35
  Brandon Kase: Agents.md, or if your agents.md tells the coding agent to read other markdown files, those would also be included.
• 6:41
  Brandon Kase: And then sensors are the things that give feedback to the coding agent as it's working and trying to perform its task.
• 6:49
  Brandon Kase: So these could be tests that run automatically every time the code is edited by a hook or because the agent decides to run it.
• 6:58
  Brandon Kase: And these these sensors, when when a test fails, for example, the the output of that test failure uh gets included in the agent's inference the next time um the next time the agent runs its loop.
• 7:19
  Brandon Kase: And so so it's a way to just in time inject the context into the coding agent, which which helps it correct itself if it's going down the wrong path.
• 7:28
  Christine Yip: Yeah.
• 7:28
  Christine Yip: So the agent is basically a loop and it just goes on until it it thinks it fulfilled its task.
• 7:35
  Christine Yip: And in this feedback step, it gives itself more context.
• 7:39
  Christine Yip: So in the next loop, it will uh give a better output.
• 7:42
  SPEAKER_00: Yeah.
• 7:44
  Brandon Kase: So this is this is the harness, and this is uh harness engineering is the idea of thinking carefully and engineering the ways in which feedforward guides and feedback sensors are supplied for whatever project you're building or whatever task you're doing.
• 8:00
  Brandon Kase: That's I think that's like a high-level, high-level explanation.
• 8:04
  Brandon Kase: So I think we can go back to this point now.
• 8:07
  Brandon Kase: So I guess, okay, so why is it surprising?
• 8:10
  Brandon Kase: Let's start with that.
• 8:11
  Brandon Kase: It's surprising to me because harness engineering is about carefully engineering your sensors, for example, um, and and shifting left uh is one way shifting a test left from a slow test that runs once in a while or a pull request review by a human and shifting that into um a lint rule that runs every time the code is changed, it's much more efficient for coding agents to get that feedback and correct themselves.
• 8:41
  Brandon Kase: And so you are more productive.
• 8:42
  Christine Yip: So if we're saying like shifting left is not as important anymore since we have hardness engineering, does that mean like, well, because we have these feedback steps, the self-correcting feedback steps, the coding agent will self-correct it anyway, so shifting left becomes less important?
• 9:00
  Brandon Kase: Um well, shifting left is the is the practice of introducing more of these feedback sensors um uh into your into your harness.
• 9:14
  Brandon Kase: So uh or at least that's one interpretation, and I think that's the interpretation that that Ryan was using when he was saying that he didn't like it as much.
• 9:24
  Brandon Kase: Uh and yeah, and and I think intuitively for me that was surprising because it seems it seems valuable, right?
• 9:34
  Brandon Kase: But I guess his his point when I asked for clarification is well, if something is triggering feedback, a self-correction, then it means that the model did something wrong.
• 9:44
  Brandon Kase: It's better if the model just doesn't do it wrong in the first place.
• 9:48
  Brandon Kase: And um and so rather than relying so much on lots and lots of sensors that get triggered all the time, it's better to put more information in the guides, more information in the markdown files that get fed into the the coding agent before it starts its work so that your sensors are triggered less often.
• 10:06
  Brandon Kase: Because whenever your sensor is triggered, you're spending a little bit of tokens.
• 10:09
  Brandon Kase: And then there's less tokens for focusing on whatever the task is at hand.
• 10:13
  Brandon Kase: Does that does that make sense?
• 10:14
  Christine Yip: Yeah, yeah, that makes sense.
• 10:16
  Christine Yip: So uh maybe sometimes when during when sensors pick up things, maybe some users are like, yes, it found something, good that it it caught it, but Ryan is like, no, if it caught something, then actually it's a failure because your guide wasn't good enough.
• 10:30
  Christine Yip: You know, it would it would have been better, of course, if if you one-shotted it.
• 10:33
  Christine Yip: But it almost, I don't know, maybe I'm just confused, but um it almost sounds like if you want the guide to be good from the start, that's even more left in the whole process, right?
• 10:46
  Christine Yip: So that to me, it also sounds like shifting left.
• 10:49
  Brandon Kase: Yeah, yeah, yeah.
• 10:50
  Brandon Kase: I think I think you're right.
• 10:52
  Brandon Kase: I think, I mean, if we go back to this picture in this particular picture of shifting left, certainly um, I guess you would say the agents.md is kind of like your requirements.
• 11:03
  Brandon Kase: So in that sense, it is further left.
• 11:05
  Brandon Kase: But I think so so in some in some sense, yeah, it's actually shifting further left.
• 11:09
  Brandon Kase: But in another sense, I think like metaphorically, putting information in your guides is like having a conversation with your coworker or like telling them a rule, but it's telling them in a way that is fuzzy.
• 11:20
  Brandon Kase: It's like they have to remember that rule.
• 11:22
  Brandon Kase: And so it's important to still have sensors because a human or an LLM can still make mistakes from that information.
• 11:29
  Brandon Kase: So it's it's sort of, but you know, you are right though.
• 11:32
  Brandon Kase: It is, it is kind of another form of shifting left.
• 11:34
  Brandon Kase: And I wish that I could ask Ryan if what what he thinks about this or something or if we're just misunderstanding what he was saying.
• 11:42
  Brandon Kase: Yeah.
• 11:43
  Brandon Kase: Um, but but I I I think like the whole, even if this is whether it's whether it is precisely shifting left or not, I I think it's just really interesting to as a point around, you want to make sure that your your system is learning.
• 11:58
  Brandon Kase: So whenever the coding agent makes a mistake, even if the mistake is self-corrected, it should still you should try and sort of teach the model or sorry, teach your harness by putting that information in the guide.
• 12:09
  Brandon Kase: I think I think that's that's the the main insight here.
• 12:12
  Brandon Kase: So this is the this is what the emerging practice is, I suppose.
• 12:15
  Christine Yip: Awesome.
• 12:16
  Christine Yip: Cool.
• 12:16
  Christine Yip: Yeah, thanks for sharing this.
• 12:18
  Christine Yip: Also, very curious to listen to Ryan's talk.
• 12:21
  Christine Yip: And if we interpreted what Ryan said wrongly, feel free to let us know.
• 12:26
  Christine Yip: At least it did set us thinking and gave us a lot of insights.
• 12:30
  Christine Yip: And so, yeah, the I believe the talks from ai native.com will be online eventually, right?
• 12:36
  Christine Yip: Do you know Brendan?
• 12:37
  Brandon Kase: Yeah, some of some of the talks, the talks on the main stage are are going to be online, and this one was.
• 12:42
  Brandon Kase: Both both Ryan's and and Brigitte's were on the main stage.
• 12:45
  Brandon Kase: So both of those will be online.
• 12:47
  Christine Yip: We can also share the links as we upload this recording.
• 12:50
  Christine Yip: We'll share it in a description.
• 12:52
  Christine Yip: All right.
• 12:52
  Christine Yip: Thanks, Brandon.
• 12:53
  Christine Yip: Then anything else you want to share before we go to emerging practice number two?
• 12:57
  SPEAKER_00: No, let's move on.
• 12:58
  Christine Yip: All right.
• 12:58
  Christine Yip: Moving on.
• 12:59
  Christine Yip: Emerging practice number two.
• 13:01
  Christine Yip: Compress intent with expert language.
• 13:04
  Christine Yip: What does this mean?
• 13:05
  Christine Yip: And how does what how did this come up for you?
• 13:08
  Brandon Kase: Yeah, so this came from a tweet.
• 13:11
  Brandon Kase: So uh I'll pull up that tweet.
• 13:13
  Brandon Kase: Um, so so there's a a tweet, I guess it was a retweet, a quote retweet.
• 13:18
  Brandon Kase: So the original tweet was from Emil Kowalski from from Linear.
• 13:22
  Brandon Kase: And Emil, well, I'll just click on that.
• 13:24
  Brandon Kase: Emil said, to get good animations from an AI, you need to get good at telling it what you want.
• 13:29
  Brandon Kase: Stagger this list of items, make this animation direction aware, spatial consistency, crossfade, layout animation.
• 13:36
  Brandon Kase: I made a motion vocabulary for this.
• 13:38
  Brandon Kase: And then there's a link to a page.
• 13:40
  Brandon Kase: So so that was that was Emil's post.
• 13:41
  Brandon Kase: And then and then uh Guillermo Rausch from Vercell retweeted and said, This is what education in the age of AI looks like.
• 13:48
  Brandon Kase: Start with the language.
• 13:49
  Brandon Kase: The linguistic surface is your roadmap.
• 13:51
  Brandon Kase: Much like an SDK has a set of function definitions, human language is the new API to the world.
• 13:57
  Brandon Kase: It also always has been.
• 13:58
  Brandon Kase: There's never been an expert in any field without mastery over its language.
• 14:02
  Brandon Kase: The distinction is that English alone could not produce tangible things.
• 14:05
  Brandon Kase: You had to translate it into machine instruction through learning or delegation to others.
• 14:10
  Brandon Kase: You can now go direct.
• 14:11
  Brandon Kase: What a beautiful tweet.
• 14:12
  Christine Yip: Yeah, and it it makes sense too.
• 14:14
  Brandon Kase: Yeah, so exactly.
• 14:15
  Brandon Kase: It does it does make sense.
• 14:16
  Brandon Kase: It seems it seems sort of obvious, but I think this really resonated with me in particular because I sort of studied this for like 10 years before LLMs came out in a slightly different context.
• 14:30
  Brandon Kase: Um so so when I when I whenever I read something about learning how to communicate about a topic with with precise language by learning carefully the language of experts in that in that field, it really gets my attention.
• 14:46
  Christine Yip: Yeah, because I guess um also the the really good thing here is if you can convey your intents in a very succinct way to an LLM, then the output from the LLM will also be better while you use less tokens, right?
• 15:01
  Christine Yip: So that's I guess what uh Emil is saying here.
• 15:04
  Christine Yip: Like uh if you're going to build animations with AI, make sure you know the terminology related to animations.
• 15:10
  Christine Yip: Um and I guess this this is for every other domain of any use case that people are building.
• 15:16
  Christine Yip: For example, if you would be working on front end or building a website, you could either say, I want a pop-up window or a panel that appears on top of the current page.
• 15:26
  Christine Yip: It should temporarily take focus, the rest of the page should be darkened or blurred, and visitors should not interact with the page behind it, blah, blah, blah, blah.
• 15:35
  Christine Yip: Or you could just say to the LLM, build a modal for me.
• 15:39
  Christine Yip: So you just have to know what the term is for the thing that you're trying to build.
• 15:43
  Christine Yip: Or if you're building something related to legal or law, then you need to know the language of lawyers.
• 15:48
  Christine Yip: And I guess the beautiful thing is LLMs know everything, right?
• 15:51
  Christine Yip: They are an expert in many different domains.
• 15:53
  Brandon Kase: Yeah, exactly.
• 15:54
  Brandon Kase: So you don't really need to teach it.
• 15:56
  Brandon Kase: I mean, okay, I suppose if if you're doing something really, really on the cutting edge, maybe you need to give it context with some research papers or something.
• 16:04
  Brandon Kase: But but essentially, LLMs are experts in everything, but by default, you get the average.
• 16:12
  Brandon Kase: You get the average of everything.
• 16:13
  Brandon Kase: You get the average person who doesn't know much about all these things in its in its output.
• 16:17
  Brandon Kase: So not only are you saving tokens by being precise and using proper language, but you're also coercing the latent space of the model towards uh the expertise in whatever field you're you're talking to it with that correct language.
• 16:31
  Christine Yip: Interesting.
• 16:32
  Christine Yip: So basically, it's so interesting you're saying the latent space.
• 16:35
  Christine Yip: So basically, it almost appears as an everyday coworker, but actually it knows so much more, but it's not really showing it as apparently.
• 16:44
  SPEAKER_00: Yes.
• 16:46
  Christine Yip: One other thing I thought was super interesting is that in our previous discussion, you went even a step further.
• 16:53
  Christine Yip: In the examples that we just talked about, um, about ML's animation and and building a website.
• 16:59
  Christine Yip: Basically, it was just about using the domain language related to the thing that you're trying to build.
• 17:05
  Christine Yip: But when we talked previously, you pulled in also another domain when you were trying to build something.
• 17:11
  Christine Yip: Could we dive a little bit into that?
• 17:13
  Brandon Kase: Yeah, yeah.
• 17:14
  Brandon Kase: So this is this is the thing that I was uh obsessed about for a long time, for many years.
• 17:20
  Brandon Kase: Um so and it's I guess I wrote about it.
• 17:23
  Brandon Kase: I'll just pull up.
• 17:24
  Brandon Kase: I have a a blog post from from 2020, and and there's just a a a paragraph here.
• 17:30
  Brandon Kase: Well, okay, so just quickly, the idea is using algebraic structures, uh, thinking about algebra when designing software.
• 17:38
  Brandon Kase: And um, I think it's interesting.
• 17:40
  Brandon Kase: In in 2020, in my intro, I I I I'll just read this out.
• 17:44
  Brandon Kase: I wrote, these structures give names to incredibly abstract notions, notions that we otherwise as humans would have a hard time discussing.
• 17:51
  Brandon Kase: When something has a name, our brains can reason about them.
• 17:54
  Brandon Kase: Shared vocabulary means more productivity for teams.
• 17:57
  Brandon Kase: Moreover, using these proper names introduces about a hundred years of mathematics and computer science content for further study.
• 18:04
  Christine Yip: That's such a great way to phrase it.
• 18:07
  Christine Yip: Basically, with LLMs, because they're so smart, you're trying, there's more opportunity to find shared vocabulary that you and LLM have.
• 18:14
  Christine Yip: And because LLMs have such a they're an expert in everything, the shared vocabulary is huge, could be huge.
• 18:21
  Christine Yip: So we could exploit more interesting.
• 18:23
  Brandon Kase: So in like something that I explored is hey, if we use algebra when building software, we can solve hard problems, build good libraries, communicate with our teams effectively, and all these things.
• 18:34
  Brandon Kase: And this is like doubly or triple E tree with LMs.
• 18:37
  Brandon Kase: Um, so just to like, I think it's interesting.
• 18:40
  Brandon Kase: One example that I used like a long time ago is also animations.
• 18:44
  Brandon Kase: So I thought it'd be interesting for me to just show that quickly.
• 18:47
  Christine Yip: So the shared vocabulary that you're using here is not only about animation, but also related to algebra.
• 18:54
  Christine Yip: So you're combining terms from both domains in this example.
• 18:58
  Brandon Kase: Yes.
• 18:58
  Brandon Kase: Although I will say I I'm definitely not an expert in the animation domain, and I did not study that too well in particular, but certainly from an algebra sense, this example I think does a good job.
• 19:10
  Christine Yip: So I'm very curious how we could use algebra for building something else that initially when I hear it, it doesn't like ring a bell for me that it's related to algebra.
• 19:21
  Christine Yip: So very curious about this example.
• 19:22
  Brandon Kase: Yeah.
• 19:23
  Brandon Kase: So just really quickly, I don't want to spend too much time on it.
• 19:26
  Brandon Kase: Um I guess anytime you can um think about a problem in terms of uh a small list of primitives that can be combined together in uh in a few ways, um, but you can just keep recombining them to build something really complex, you can take advantage of algebra.
• 19:49
  Brandon Kase: I guess the metaphor first is like Legos, you know, there's a few different Lego blocks, but even with just a few blocks, if you build them in different ways, you can build cities and towns and castles and all these things.
• 20:01
  Brandon Kase: So so in this case, let's do the same thing with animations.
• 20:04
  Brandon Kase: So let's say an animation is our core object, and we can just say an animation is something that happens over time.
• 20:10
  Brandon Kase: And and uh, you know, but it can be anything that happens over time.
• 20:13
  Brandon Kase: And then you can combine animations together.
• 20:15
  Brandon Kase: So this is an exploration of choreographed animations.
• 20:18
  Brandon Kase: So it's animations that you configure ahead of time in kind of like a movie.
• 20:22
  Brandon Kase: Um, and and so for example, if you have an animation for growing a circle and you have a different animation for fading out a circle, well, you can run them at the same time, for example.
• 20:32
  Brandon Kase: And we can call that addition.
• 20:34
  Brandon Kase: And if you run those two animations at the same time, you get what's on the video here, if you're watching the video, which is a circle that grows and fades out at the same time.
• 20:43
  Brandon Kase: Okay, makes sense.
• 20:44
  Brandon Kase: And you could also run animations one after another rather than at the same time, you could have one complete and then start the other one and then complete that.
• 20:52
  Brandon Kase: And that's sequencing one animation after another, we could call that multiplication.
• 20:56
  Brandon Kase: And it turns out if you choose multiplication for sequencing and addition for parallel, a lot of the algebraic laws that are true on integers, like addition commutes, one plus two is the same as two plus one, that multiplication distributes over addition, uh, which is like something you might have learned in in school, these things are true for animations as well.
• 21:17
  Brandon Kase: And what that does at the end of the day is in just a few operations, you can build really complex things and gives give a lot of power to users of software libraries.
• 21:28
  Brandon Kase: And and this is was true for humans, but it's true for LLMs as well.
• 21:33
  Brandon Kase: If you can describe a software surface area in less tokens, then you have more context to spend on you know doing whatever you need to do.
• 21:41
  Brandon Kase: So it's it's I think it's interesting as an example of something that you can apply to, like you were saying, something else.
• 21:48
  Brandon Kase: So it's animations, and I'm writing software with algebra.
• 21:51
  Brandon Kase: You could, for example, write software using logic and apply that to law, you know?
• 21:58
  Brandon Kase: So you can you can mix and match and pull.
• 22:00
  Brandon Kase: In language from different fields and use them at the same time, and you get uh you get to tap into the expertise of the LLM, uh you know, wearing different sort of expert hats at the same time, which is cool.
• 22:14
  Christine Yip: It sounds like what you're doing in this example is like what you're asking is basically what are the algebraic properties that also work in animation?
• 22:22
  Christine Yip: And here you've shown like, okay, addition is commutative.
• 22:25
  Christine Yip: So no matter what sequence, you get the same results.
• 22:28
  Christine Yip: So basically, what here on screen I see grow circle plus fade out.
• 22:33
  Christine Yip: And then the LLM knows like, okay, these are the algebraic properties that also should be applied here for the animation.
• 22:39
  Christine Yip: That's that's actually such a smart way to communicate um with an LLM.
• 22:43
  Christine Yip: And yeah.
• 22:44
  Brandon Kase: You know, experimentally I have haven't done much with animations personally with with LLMs in my projects, but I have I have explored using the language of algebra to communicate with the LLMs to design libraries that are easy to use.
• 22:59
  Brandon Kase: And ultimately, it in my experience, it does pull the LLMs to build nice libraries that I think are nice to use.
• 23:07
  Brandon Kase: So that's just my opinion.
• 23:08
  Christine Yip: That's so cool.
• 23:09
  Christine Yip: So basically, what we're describing here is we're even going beyond what Guillermo has said here.
• 23:13
  Christine Yip: So it's not only like, okay, try to communicate super succinctly but in a powerful way what your intent is by using the domain language of the thing that you're trying to build in in a specific domain, but you could even go beyond and also use terms from another domain.
• 23:29
  Christine Yip: This is so interesting.
• 23:30
  Christine Yip: Do any examples come to mind where you've seen other buildings doing this too?
• 23:36
  Christine Yip: Where they go beyond one domain and combine it with terminology or libraries that are related to other domains?
• 23:42
  Brandon Kase: Yeah.
• 23:43
  Brandon Kase: Well, the uh it turns out using this like algebraic approach, it works really well when you're doing things that are very abstract.
• 23:52
  Brandon Kase: It's called abstract algebra, so it makes sense that when you're doing abstract things with software, it's useful.
• 23:57
  Brandon Kase: So like handling errors, just like the notion of handling errors, all the different ways you might want to handle errors, dealing with things running at the same time concurrently.
• 24:04
  Brandon Kase: That's another like abstract thing that works well with this kind of algebraic approach.
• 24:09
  Brandon Kase: Dependency management.
• 24:10
  Brandon Kase: So if you depend on certain resources and they need to load in certain orders and all of that.
• 24:16
  Brandon Kase: And so there's actually a library in TypeScript that has become very popular recently since the since the coding agent boom, which is called Effect TS.
• 24:28
  Brandon Kase: And Effect literally is just the combination of error handling, concurrency management, and dependency management treated very algebraically.
• 24:37
  Brandon Kase: And it's something that it's a little bit scary to developers at first, like this kind of thinking.
• 24:43
  Brandon Kase: I don't know people who've worked with Effect directly, but with this kind of approach, I've found resistance in you know different, different teams where I've tried to introduce this kind of programming before.
• 24:54
  Brandon Kase: But you get so much benefit out of it that it's kind of, in my opinion, it's worth getting through that resistance.
• 24:59
  Brandon Kase: Well, the the coding agents, they are experts in everything.
• 25:02
  Brandon Kase: So there's no resistance anymore.
• 25:04
  Brandon Kase: And and turns out, you know, doing things algebraically is a good idea.
• 25:08
  Brandon Kase: And so Effect TS suddenly is this super powerful library that people are finding empirically that when you tell your coding agent to use effect when it's building your TypeScript software, the coding agent just does a better job.
• 25:22
  Brandon Kase: There's less bugs, it can do more complex things more easily.
• 25:25
  Brandon Kase: This is the power of algebra, I guess, applied to, in this case, like this abstract idea of error handling, dependency management, and concurrency.
• 25:34
  Christine Yip: Awesome.
• 25:35
  Christine Yip: Very cool.
• 25:35
  Christine Yip: So it's this powerful TypeScript library that also handles errors and dependencies and all those stuff in algebraic way.
• 25:43
  Christine Yip: But this really sets people thinking like, what can we do?
• 25:46
  Christine Yip: What else can we introduce to explore the latent space from the LLM?
• 25:50
  Christine Yip: Yeah, cool.
• 25:51
  Christine Yip: Thanks for sharing.
• 25:52
  Christine Yip: Should we go dive into the next one?
• 25:54
  Christine Yip: All right.
• 25:54
  Christine Yip: Um, emerging practice number three, match the tool interface to the task.
• 25:59
  Christine Yip: I think when we talked about this in our previous conversation, you you said be thoughtful with your agent tools.
• 26:05
  Christine Yip: And we mentioned different tools in our conversation.
• 26:08
  Christine Yip: There was this like hugely popular MCP that became super popular last year.
• 26:13
  Christine Yip: And then the second half of last year, people started to shift more to CLI.
• 26:18
  Christine Yip: And now it sounds like uh what's becoming more popular is code mode.
• 26:22
  Christine Yip: Could you tell us a bit more about these?
• 26:24
  Brandon Kase: Yeah.
• 26:24
  Brandon Kase: Um, well, I don't think code mode is becoming popular yet, but uh I think it should become popular because it's a good idea.
• 26:31
  Brandon Kase: Let's put it that way.
• 26:32
  Christine Yip: Before we start, maybe just to refresh everyone's um memory about it, why was everyone so excited about MCP?
• 26:38
  Christine Yip: And what why was CLI perceived as better by some afterwards compared to MCP?
• 26:44
  Christine Yip: And then why do you think Coke mode should be more popular now?
• 26:47
  Brandon Kase: Okay.
• 26:48
  Brandon Kase: Um so I I think in order to have this discussion, it's important to understand the agent loop carefully.
• 26:55
  Brandon Kase: So the the agent loop, which is what our coding agents do or any any agent, uh, is something like, you know, send prompt, get result, decide, call a tool or done.
• 27:06
  Brandon Kase: Um if it's call a tool, call the tool, get a result, put the result back into context, and then call inference again, get another result, and maybe you'll call a tool, and then you just keep looping until you're done.
• 27:17
  Brandon Kase: Um the early, uh in the early days, uh, you had to hard code your tools.
• 27:22
  Brandon Kase: So you in the in the system prompt for the agent, you'd say, okay, you have the ability to get the weather or use a calculator, and then you specified how the agent in its response can say, Hey, I want to call the weather and get the tool.
• 27:36
  Christine Yip: Oh yeah, I still remember those days.
• 27:38
  Brandon Kase: Yeah.
• 27:39
  Brandon Kase: Um, and this was this was a little strict, I guess.
• 27:43
  Brandon Kase: And so, so enter MCP model context protocol.
• 27:47
  Brandon Kase: What MCP is, is it's literally a layer of indirection so that your tool call can be defined outside of the agent loop.
• 27:55
  Brandon Kase: So, so the tool that we put in the agent loop is call mcp, basically.
• 28:00
  Brandon Kase: And one of them is, you know, get get information from MCP server to tell me what the tool does.
• 28:06
  Brandon Kase: And the other one is call mcp tool with the information I got from the server.
• 28:10
  Brandon Kase: And and what this what this lets you do is it lets you get, it sort of introduces a plug-in ecosystem where lots of services can provide MCP interfaces.
• 28:19
  Brandon Kase: And you can, as a user of an agent, connect to any of these providers without the agent builder directly connecting them ahead of time.
• 28:28
  Brandon Kase: So that's like a cool thing.
• 28:30
  Brandon Kase: And in the context of coding agents, this is a way for you know cloud code to talk to GitHub.
• 28:36
  Brandon Kase: It's one way for Cloud Code to talk to GitHub, for example.
• 28:38
  Brandon Kase: Okay.
• 28:39
  Brandon Kase: Okay.
• 28:40
  Brandon Kase: So now there's a CLI, a CLI interface.
• 28:44
  Brandon Kase: So let's let's let's use this example of getting information from GitHub.
• 28:48
  Brandon Kase: You could use a GitHub MCP server, or you can use a GitHub CLI tool.
• 28:52
  Brandon Kase: And so rather than giving in the agent loop an MCP tool to the coding agent, you would give a bash tool.
• 28:59
  Brandon Kase: So the bash tool lets the coding agent run any bash command or a small bash script.
• 29:04
  Brandon Kase: So in this bash script, they can call one or more CLI tools and combine and the results together in interesting ways.
• 29:12
  Brandon Kase: Um, so for example, if we wanted to get the issues associated with a certain GitHub repository, I could make an MCP call to GitHub to get the issues, I get the result back, and then I give it to the user.
• 29:25
  Brandon Kase: Or with the CLI tool, I can make a bash call.
• 29:29
  Brandon Kase: The bash script can call, use the GitHub tool to get the information about the issues, and then I give it back to the user.
• 29:36
  Brandon Kase: Okay, it's about the same.
• 29:37
  Brandon Kase: Now, the interesting thing comes when you want to do something more complex.
• 29:41
  Brandon Kase: I'm gonna use another simple example, which is count the number of issues on a GitHub project.
• 29:47
  Brandon Kase: And let's just say there's no MCP call for that directly, or there's no CLI call for that directly either.
• 29:52
  Brandon Kase: You only have the ability to get all the issues.
• 29:54
  Brandon Kase: If you think about what the agent loop is doing, the first the loop goes and the agent decides, I want to call GitHub to get the issues, gets back the issues, those issues enter context, and then inference runs again on whatever the context was before plus the results of those issues.
• 30:10
  Brandon Kase: And then maybe the agent needs to use another tool to count, or maybe it counts by looking at the context and it gives you the answer.
• 30:16
  Brandon Kase: Now, with the CLI tool approach, the agent decides, okay, I need to use the GitHub tool.
• 30:21
  Brandon Kase: But rather than just getting the results of the issues back into context, the agent decides, okay, get the issues with the GH tool and pass the results of those issues to another CLI tool called WC, for example, which which can count the number of issues.
• 30:38
  Brandon Kase: And then what it gets back is just the number.
• 30:40
  Brandon Kase: And so the the context that's used is much smaller.
• 30:44
  Brandon Kase: There's only one tool call and and then and then you get the result.
• 30:47
  Brandon Kase: So so using the CLI, using bash, lets the agent put more information or more logic, can do more and compose together different tools in one logical agent tool call, which is just more fit.
• 31:00
  Christine Yip: So because it was just one agent tool call, when you do it with the CLI, and it in your example, it did getting the issues and counting the issues right after one another.
• 31:12
  Christine Yip: Um so basically the LLM doesn't feed all the intermediate results back into the loop and redo the loop basically before it does the counting.
• 31:21
  Christine Yip: It just gets past the intermediate results from the previous step.
• 31:24
  Christine Yip: Uh basically your context window stays cleaner.
• 31:26
  Christine Yip: Yes.
• 31:26
  Christine Yip: Well, interesting.
• 31:27
  Christine Yip: That's that makes sense.
• 31:28
  Brandon Kase: And this is why uh the world has shifted a lot of momentum towards CLI interfaces rather than MCP.
• 31:37
  Brandon Kase: Um, because it's just more efficient.
• 31:41
  Brandon Kase: Um now, now, the interesting thing is you can take another step, and this is this is code mode.
• 31:47
  Brandon Kase: So this is something that people have been discussing, I think since this year.
• 31:51
  Brandon Kase: So it's been a few months.
• 31:53
  Brandon Kase: I'm gonna motivate this with another example.
• 31:55
  Brandon Kase: So I'm gonna use real example.
• 31:57
  Brandon Kase: So so uh Anthropic just this week released a let me share my screen.
• 32:02
  Brandon Kase: Let's see.
• 32:03
  SPEAKER_00: Here it is.
• 32:04
  Brandon Kase: Anthropic released dynamic workflows this week or last week.
• 32:08
  Brandon Kase: Anyway, one of these weeks recently.
• 32:09
  Brandon Kase: What is a dynamic workflow?
• 32:11
  Brandon Kase: It's code.
• 32:13
  Brandon Kase: Okay.
• 32:14
  Brandon Kase: So um so let's let's use this as our example to explain what code mode is and why it's valuable when when you when you can use it.
• 32:24
  Christine Yip: Is this uh their form of code mode or is this something different?
• 32:28
  Christine Yip: So dynamic workflow.
• 32:30
  Brandon Kase: Code mode is a um you can think of code mode as the interface to um a set of tools or like a tool that you'd plug into your agent.
• 32:40
  Brandon Kase: So so in this case, dynamic workflows is uh a code mode style way of allowing cloud code to interact with subagents, to orchestrate subagents, basically.
• 32:52
  Christine Yip: Cool.
• 32:52
  Christine Yip: So maybe the analogy is like dynamic workflows and allows cloud code to interact with subagents using code.
• 32:59
  Christine Yip: And in code mode, it allows the agent to interact with tools using code.
• 33:03
  Christine Yip: That's the analogy.
• 33:04
  Christine Yip: Is that right?
• 33:05
  Brandon Kase: Yeah, dynamic workflows allows cloud code to interact with subagents using code, but I haven't motivated why that's good yet.
• 33:12
  Brandon Kase: Okay, tell us.
• 33:12
  Brandon Kase: So, okay, so so so the task is to work with subagents.
• 33:16
  Brandon Kase: So that's spawn subagents, wait for subagents, and sort of organize subagents in different ways.
• 33:21
  Brandon Kase: So you could start with just a simple tool call or an MCP.
• 33:26
  Brandon Kase: And let's say I want to create five agents, and each of those creates two agents, and then I wait for all the results of those agents to come back.
• 33:34
  Brandon Kase: So that's that's the orchestration I want to do.
• 33:36
  Brandon Kase: So if I was using tool calls, I would make a tool call to spawn each agent.
• 33:41
  Brandon Kase: Maybe I could spawn five at a time.
• 33:42
  Brandon Kase: And then for each one, I would have to spawn more agents.
• 33:45
  Brandon Kase: So we're at, I don't know, 10 tool calls, maybe at the minimum, maybe like 20, and then wait for the results of all of these, which is at least another one.
• 33:56
  Brandon Kase: We're at like 25 tool calls, lots of intermediate context.
• 33:59
  Brandon Kase: It's kind of the same as the last example, a lot of context wasted, bad.
• 34:03
  Brandon Kase: Okay.
• 34:03
  Brandon Kase: So then you could think, okay, well, what if it was a CLI tool?
• 34:05
  Brandon Kase: That's better.
• 34:06
  Brandon Kase: So I could use bash and I could uh if there was a CLI tool that could spawn a subagent and wait for the results, well, I could spawn all the subagents.
• 34:15
  Brandon Kase: I could represent that structure in a bash script.
• 34:20
  Brandon Kase: Um, but it turns out bash is a bad programming language.
• 34:24
  Brandon Kase: And so when when when bash when a bash script gets really complicated, errors start happening.
• 34:30
  Brandon Kase: The like both humans and LLMs just can't like kind of struggle to make it work.
• 34:35
  Brandon Kase: Um and uh and you know, JavaScript is a better programming language than Bash.
• 34:40
  Brandon Kase: And so what code mode does is instead of saying bash is my tool and I can call some CLIs and connect them together, code mode says, here's a JavaScript interface and just write JavaScript.
• 34:51
  Brandon Kase: And and and that's exactly what dynamic workflows is.
• 34:54
  Brandon Kase: It's a it's a mechanism for for Claude when it's spawning subagents or describing a subagent orchestration workflow to use JavaScript or TypeScript for coding.
• 35:05
  Brandon Kase: Uh and so in the agent loop, the the agent, instead of making a tool call to MCP or instead of making a tool call to bash with a bash script, it makes a tool call to do dynamic workflows and gives it a bunch of JavaScript.
• 35:17
  Brandon Kase: And then that JavaScript can efficiently and effectively, and in a way that the LMs don't make many mistakes, describe very complex structures of workflow orchestration.
• 35:28
  Christine Yip: Um interesting.
• 35:29
  Christine Yip: And how's that where where how does it relate to using code mode for using tools?
• 35:36
  Brandon Kase: This is just an example of a tool that works really well with code mode.
• 35:40
  Brandon Kase: I can give more examples.
• 35:42
  Brandon Kase: So there's a Cloudflare in uh, I guess in February, presented a mechanism to do search with code mode.
• 35:51
  Brandon Kase: Um, I won't go through that whole blog post, but basically the the search API for Cloudflare is very powerful.
• 35:58
  Brandon Kase: And so rather than exposing all those belt, like little details through a CLI tool, maybe they are also exposed through the Cloudflare CLI tool.
• 36:06
  Brandon Kase: But in order to take advantage of all those features and being able to be flexible, the Cloudflare team found it helpful to provide a code mode interface.
• 36:14
  Brandon Kase: So the tool gives JavaScript to Cloudflare.
• 36:18
  Brandon Kase: Um, and that JavaScript describes how you want to do your search when the when the tool call comes back.
• 36:23
  Brandon Kase: So that's another example of code mode.
• 36:25
  Christine Yip: Cool.
• 36:25
  Christine Yip: So it sounds like code mode is really good if you want your agents to write arbitrary programs that are doing like more complex, or if you want it to be more flexible and do more complex logic.
• 36:38
  SPEAKER_00: Yes.
• 36:39
  Christine Yip: That's when code mode will shine.
• 36:40
  Christine Yip: And bash is basically really good at combining like maybe CLI calls from different tools, but not really good for writing, again, writing arbitrary programs that can do very flexible things and puzzling them together, then then uh code mode is better in that situation.
• 36:56
  Christine Yip: Is that right?
• 36:57
  Brandon Kase: Yes, exactly.
• 36:58
  Brandon Kase: Yeah.
• 36:58
  Brandon Kase: So so code mode is not something that can just replace using the bash tool.
• 37:03
  Brandon Kase: The bash tool is amazing.
• 37:04
  Brandon Kase: It's important for combining CLI tools together, like like you said, efficiently.
• 37:08
  Brandon Kase: But uh code mode is useful when um when their tool that you're using has enough complex behavior that you could design some kind of interface, some kind of API to control it with code.
• 37:21
  Brandon Kase: Let me give one more example.
• 37:22
  Brandon Kase: This is something that I used in in one of my projects a month or two ago.
• 37:27
  Brandon Kase: So let's see if I can I have it here.
• 37:31
  Brandon Kase: Um so so I one of my projects is uh building a Game Boy game that runs LLM inference.
• 37:37
  Brandon Kase: Um so it's just like a fun little process.
• 37:40
  Brandon Kase: When you're when you're building a Game Boy game, it's important to have ways to debug issues.
• 37:47
  Brandon Kase: And there's a tool called a debugger, which lets you step through instructions carefully uh and inspect hardware state of the Game Boy um when it's running in an emulator.
• 37:58
  Brandon Kase: So um you can build a debugger that has a bunch of buttons on it, and that's hard for an LLM to use.
• 38:06
  Brandon Kase: So then you can say, okay, well, I can build a CLI debugger, and then that's easier for the LLM to use, and it is.
• 38:12
  Brandon Kase: But I found debuggers are complex enough that actually it makes sense to provide a code mode interface.
• 38:18
  Brandon Kase: So so my debugger CLI tool has a code mode in it where you feed it JavaScript.
• 38:26
  Brandon Kase: Um so uh and and that lets you that sort of gives a lot more power to the to the system.
• 38:32
  Brandon Kase: So for example, one thing that's like very doesn't really make sense to do from a CLI tool context is like automatically running logic every time a certain part of the the Game Boy game executes.
• 38:44
  Brandon Kase: So for example, every time I hit this instruction, you know, look at the the Game Boy state, and if it's in this current state, set new breakpoints that do this over here.
• 38:53
  Brandon Kase: So these kinds of instructions, it's very hard to come up with a way for a CLI tool to express that logic, but it's very simple to express that in JavaScript.
• 39:01
  Brandon Kase: And when I tell my coding agent to use this debugger in code mode, it can come up with those complex scripts by itself.
• 39:08
  Brandon Kase: And I found it useful for debugging.
• 39:10
  Christine Yip: Interesting.
• 39:11
  Christine Yip: So basically you're telling like the LLM this is how you code the tool in order to use it.
• 39:17
  Brandon Kase: Yeah, so I have a skill for the Game Boy debugger that has a code mode interface.
• 39:22
  Brandon Kase: Yeah.
• 39:23
  Brandon Kase: And that skill teaches the LLM how to work with the complex interface and write software to sort of get the most out of it when it's debugging.
• 39:31
  Christine Yip: Very cool.
• 39:32
  Christine Yip: So basically, I guess uh the lesson here is understand why or the cost of using these different ways of giving your LLM tools so you can make good decisions, tasteful decisions to choose one over the other.
• 39:48
  Brandon Kase: Yeah, yeah.
• 39:48
  Brandon Kase: And by the way, sometimes you need to use MCP.
• 39:51
  Brandon Kase: If you're it's easier to deal with stateful services using MCP, it's still possible with CLI tools, but it's a little bit annoying.
• 39:59
  Brandon Kase: So if you have some kind of stateful service that you're interacting with, it can be nice to use MCP over CLI in that case.
• 40:09
  Brandon Kase: So yeah, it's be be thoughtful.
• 40:11
  Brandon Kase: And I think when you can use code mode, uh, you'll unlock a new superpower with your coding agent.
• 40:17
  Christine Yip: That's very cool.
• 40:18
  Christine Yip: It sounds like if you want to have good taste, you definitely also want to have deeper understanding so you can make those tasteful decisions.
• 40:28
  Christine Yip: All right.
• 40:28
  Christine Yip: Yeah, thanks for walking us through this.
• 40:31
  Christine Yip: Anything you want to share before we go to our last emerging practice?
• 40:34
  SPEAKER_00: No, let's let's do it.
• 40:36
  Christine Yip: All right.
• 40:36
  Christine Yip: I also really like this last one.
• 40:38
  Christine Yip: Emerging practice number four.
• 40:39
  Christine Yip: Use oracles when you can.
• 40:41
  Christine Yip: Can you tell us more about it?
• 40:42
  Brandon Kase: Yeah.
• 40:43
  Brandon Kase: Um, so this idea was one of the points that a speaker at AI Native DevCon made.
• 40:50
  Brandon Kase: It was Justin Kormack.
• 40:51
  Brandon Kase: He's the ex-CTO of Docker.
• 40:53
  Brandon Kase: And Justin is rebuilding his own version of AWS S3 from scratch as a as a fun project.
• 41:00
  Brandon Kase: And I guess so Oracles.
• 41:03
  Brandon Kase: So what is an Oracle?
• 41:04
  Brandon Kase: Let's start with that.
• 41:06
  Brandon Kase: An Oracle is uh some system that you just say is correct.
• 41:12
  SPEAKER_00: So in in Justin's example, since he's rebuilding AWS S3, the real using an Oracle, just quick question, so so you can all follow along.
• 41:23
  Brandon Kase: Well, so he's building a couple of AWS S3.
• 41:27
  Brandon Kase: Um there's uh the real Amazon S3 is uh an Oracle or can be an Oracle for his version.
• 41:36
  Christine Yip: So is he making a case that using Oracle is so good and he it's a lesson from his own experience?
• 41:43
  Brandon Kase: Yes, it's one of one of the lessons from his presentation was using an Oracle uh during during exploration of behavior and testing was very useful.
• 41:54
  Brandon Kase: So he found that um the documentation for AWS S3 is actually not accurate when compared to the behavior.
• 42:01
  Brandon Kase: And so it's it's better, it was better for him to have his agent just explore what the real S3 does and learn that behavior so that he is able to implement something that works in the same way.
• 42:13
  Christine Yip: And that's so that's why you meant he used AWS as the Oracle.
• 42:18
  Brandon Kase: Yeah, yeah.
• 42:19
  Brandon Kase: Um so there's a lot of ways to use Oracles in your development.
• 42:25
  Brandon Kase: Um and uh and you know, as Justin's saying, it's really useful.
• 42:31
  Brandon Kase: So I have did a project and and take took advantage of this.
• 42:35
  Brandon Kase: I guess just really quickly, I worked on another Game Boy emulator project where I was building a GPU emulator for Game Boy, and I used a known good CPU Game Boy emulator as an Oracle.
• 42:47
  Brandon Kase: And once I had infrastructure set up so that in my harness, so that the agent could just automatically check against the real working emulator and mine, the the agent was able to run autonomously for a few days and just get everything working perfectly, basically.
• 43:06
  Christine Yip: So it sounds like it's good to use an Oracle when you want to convey the intent of the thing that you want to build.
• 43:13
  Christine Yip: Is that right?
• 43:13
  Christine Yip: So basically it's a shortcut.
• 43:15
  Christine Yip: You're conveying it more specifically.
• 43:18
  Brandon Kase: So it's not, it's it's very useful in testing, but as you're saying, it's also useful as a communication mechanism.
• 43:24
  Brandon Kase: It's kind of like, you know, the saying like a picture is worth a thousand words, a working version of the thing is worth more words than you trying to describe how it works, just showing it to the agent.
• 43:35
  Brandon Kase: Something that Justin mentioned in his presentation.
• 43:37
  Brandon Kase: He just said a sentence on it.
• 43:38
  Brandon Kase: Again, something that was just one sentence that really stuck with me, which was he said, it's probably worth implementing a basic version of the service or application that you're trying to build first and get the behavior just perfect.
• 43:52
  Brandon Kase: But take shortcuts.
• 43:54
  Brandon Kase: It doesn't actually work fully.
• 43:55
  Brandon Kase: No, it's not ready to be shipped to people, but it but it works enough so that the behavior is exactly.
• 44:00
  Brandon Kase: And then use that as an Oracle to then go and implement the real thing that might be a lot more complicated.
• 44:05
  Brandon Kase: And I think that is a really cool idea.
• 44:06
  Christine Yip: So, as an example, let's say I build a journaling app.
• 44:16
  Christine Yip: And it only works locally on my machine.
• 44:19
  Christine Yip: But if I want to scale it up and I want it to be able to serve thousands of users, it's okay if my own version on my machine works only well with me as the only user, because then we can use it as an Oracle and AI will know how it should behave even when there are thousands of people using it.
• 44:35
  Brandon Kase: Exactly.
• 44:36
  Brandon Kase: Shortcut to building the version that supports thousands of users, to have the version, to build the version that works with one user first and get that behavior exactly how you want it.
• 44:46
  Brandon Kase: And then you get a shortcut because you can say, okay, this is the Oracle, make Oracle tests, make it behave exactly like this, and then go implement this thing that's much more complicated that supports all these users.
• 44:57
  Christine Yip: Right.
• 44:57
  Christine Yip: Because if you even if you write a spec, sometimes there are still like great areas or maybe there are blind spots.
• 45:03
  Christine Yip: But if you give your AI access to an Oracle whenever it has any doubts or doesn't know what decisions it should make, basically it will just query the Oracle.
• 45:12
  Christine Yip: Yes.
• 45:13
  Christine Yip: That's very smart.
• 45:13
  Christine Yip: Actually, it makes me also wonder whenever I see things, uh, people building things in a space and they're like, oh, I just built this in two days using AI.
• 45:21
  Christine Yip: I built this browser, you know, in in a day with an agent swarm.
• 45:26
  Christine Yip: Did they build it from scratch, like in the traditional way, like how you would like start with a spec and just build it from the ground up?
• 45:33
  Christine Yip: Or did they use an Oracle for it?
• 45:35
  Christine Yip: Because it sounds like using an Oracle could be a shortcut.
• 45:37
  Brandon Kase: It's a real, it's it's a shortcut and it lets you do things that are otherwise impossible with this version of agents.
• 45:44
  Brandon Kase: Certainly impossible to do autonomously without human input.
• 45:47
  Brandon Kase: So all the examples that we've seen that are really impressive, building a browser, building an operating system, building a C compiler, these are things that have oracles that can be used to ensure that the thing is correct as it's being built.
• 46:00
  Brandon Kase: Yeah, and it's not, I don't think that's bad.
• 46:03
  Brandon Kase: I think it's good.
• 46:03
  Brandon Kase: I mean, it's something it's something to learn from.
• 46:05
  Brandon Kase: I think, like in the context of thinking about taste, um uh the best way to express little nuances in your taste is by having a prototype of the thing that behaves exactly how you want it to behave, and then telling the agent it needs to be exactly like this.
• 46:23
  Christine Yip: Yeah.
• 46:24
  Christine Yip: No, this makes me really excited to try it out.
• 46:27
  Christine Yip: Not not only this emerging practice, but also the others that you just mentioned.
• 46:31
  Christine Yip: This is so this is so good.
• 46:33
  Christine Yip: So if we zoom out, uh, all four of these practices that we talked about pointed kind of in the same direction.
• 46:40
  Christine Yip: Like good taste when building with AI is not just using better models or just writing better prompts, but well, among many things, you have to think about how to shape the environment around a model.
• 46:52
  Christine Yip: You have to learn how to convey the intent more precisely, either through an Oracle or more precise expert language.
• 46:59
  Christine Yip: And also you'd have to think about how to present agent tools to the agents in every situation.
• 47:05
  Christine Yip: So, yeah, I guess these were the four emergent practices that we wanted to share.
• 47:09
  Christine Yip: And this is also what we're going to keep exploring.
• 47:12
  Christine Yip: So, how builders can develop better taste and how they can channel that taste into tools, agents, and workflows.
• 47:19
  Christine Yip: Yeah, I got a lot out of it, and I'm very excited to try these things out.
• 47:24
  Brandon Kase: Yeah, and and these were just four things that happened to be mentioned in the AI space last week.
• 47:29
  Brandon Kase: And you know, a few of them were just one sentence that we turned into an hour-long discussion.
• 47:34
  Brandon Kase: So there's so many more of these practices.
• 47:37
  Brandon Kase: I mean, I have a hundred in my head right now.
• 47:39
  Brandon Kase: So Christine and I are gonna have a lot of conversations about this just for fun.
• 47:43
  Brandon Kase: And uh yeah, I guess let us know if you want us to keep sharing these, and we will.
• 47:48
  Christine Yip: Thank you so much for listening, and thank you as well, Brendan.
• 47:51
  Christine Yip: This was a lot of fun.