Latest / Superlinear / Point, Don't Describe: A Better Way to Work With AI Agents

Transcript

Point, Don't Describe: A Better Way to Work With AI Agents · Superlinear

• 0:00
  Brandon Kase: These tools that convert that higher dimensional information down to a lower dimension make it easier for models to understand.
• 0:08
  Christine Yip: The right workflow is to build that spatial feedback layer into the game.
• 0:22
  Christine Yip: Emerging practices for building with agents.
• 0:25
  Christine Yip: See new angles.
• 0:26
  Christine Yip: Try different approaches.
• 0:27
  Christine Yip: Build more powerful things.
• 0:29
  Christine Yip: I'm Christine Yip.
• 0:30
  Christine Yip: And I'm Brandon Case.
• 0:32
  Christine Yip: And this is Super Linear.
• 0:34
  Christine Yip: Today we're going to talk about a pattern that we've been noticing, which is the higher dimensional interactions between humans and agents.
• 0:43
  Christine Yip: Well, what does this actually mean?
• 0:44
  Christine Yip: Maybe, Brandon, you can start us off and explain what the different dimensional interactions are.
• 0:52
  Brandon Kase: When when we think of one dimension, I mean like a line.
• 0:55
  Brandon Kase: Yeah.
• 0:56
  Brandon Kase: And then let's say two dimensions, you can think of as a plane.
• 1:00
  Brandon Kase: And so when we think about dimensionality in the way in which we communicate, communicating with text, with information that you type or you speak, or on the other side that you read and you listen, this is one-dimensional because the order of the words that are expressed is important in the information that that's being shared.
• 1:23
  Brandon Kase: Does that make sense?
• 1:24
  Christine Yip: Right.
• 1:24
  Christine Yip: So it's one-dimensional because it's all about the order to sequence, like word, which word come after that word, that word, sentence after sentence.
• 1:32
  Christine Yip: It sounds like that's also the way that Agent processes things, right?
• 1:35
  Christine Yip: Like a stream of tokens in in a certain order.
• 1:38
  Brandon Kase: Exactly.
• 1:38
  Brandon Kase: Exactly.
• 1:39
  Brandon Kase: And yeah, I think there's now multimodal models that break apart images and other audio maybe into tokens.
• 1:46
  Brandon Kase: At the end of the day, these tokens are being fed in in a certain order, one at a time.
• 1:50
  Brandon Kase: And so because of that, yeah, agents are really good at processing linear information.
• 1:55
  Brandon Kase: So uh, and and as people, we're good at speaking linear information as well.
• 2:01
  Brandon Kase: Now, but maybe before we go too deep into that, uh, let's just quickly talk about two dimensions.
• 2:06
  Brandon Kase: So, well, uh, what we see with our eyes, images, uh, this is this is a two-dimensional input to our bodies.
• 2:15
  Christine Yip: They have positions in in the images, like every pixel has a different position.
• 2:19
  Brandon Kase: Exactly.
• 2:20
  Brandon Kase: Like where things are in space, uh, the color, the lights, everything.
• 2:23
  Brandon Kase: There's a tweet from Andre Karpathi from a month or so ago, and he talks about this.
• 2:30
  Brandon Kase: So it's very long, so I'm not gonna read the whole thing, but audio is the human preferred input to AIs, so it's the output from our bodies into the agents, and vision is the preferred output.
• 2:41
  Brandon Kase: And why is that?
• 2:42
  Brandon Kase: It's because we're social creatures, and so we're really good at sharing information with our words from our mouth, more than typing, actually.
• 2:50
  Brandon Kase: Uh, we've relied on being able to see and process things visually to survive.
• 2:55
  Brandon Kase: So our a lot of our brain is dedicated to processing these visual information.
• 2:58
  Brandon Kase: This was also mentioned just last week in the AI engineering world's fair.
• 3:05
  Brandon Kase: And so Alan Pike did a talk called Voices in, Visuals Out.
• 3:09
  Brandon Kase: Anyway, he he mentioned this concept.
• 3:11
  Brandon Kase: The talk is about more about like uh UIUX and latency budgets when working with these modalities with with a custom agent.
• 3:20
  Christine Yip: So I guess we are seeing that people are thinking and moving beyond just text or voice, like the one-dimensional forms that you mentioned, and and they're moving more towards the multidimensional form.
• 3:34
  Brandon Kase: Exactly.
• 3:36
  Brandon Kase: And the codex and Claude and all all the all the popular models have supported sharing images for for a while.
• 3:44
  Brandon Kase: And that's something that lots of people are using in their workflows.
• 3:46
  Brandon Kase: But as we mentioned before, agents are not very good at understanding what you mean directly from an image, even though as humans, we are really good at understanding information that's coming in as an image.
• 3:58
  Brandon Kase: So um, so there's there's this impedance mismatch.
• 4:02
  Brandon Kase: And so this is where this practice is emerging that we're gonna focus on here.
• 4:06
  Brandon Kase: Agents are different.
• 4:07
  Brandon Kase: Like we said, they they process tokens one at a time.
• 4:10
  Brandon Kase: And so it's most effective if you want to be precise to um communicate more linearly with the agents, and it's most effective as a human to receive information visually.
• 4:20
  Brandon Kase: So there's this three-step process that we can talk through.
• 4:23
  Brandon Kase: It's a cycle, right?
• 4:24
  Brandon Kase: So we have to pick somewhere to start from.
• 4:26
  Brandon Kase: But let's say, let's start from rendering.
• 4:28
  Brandon Kase: So so we see information in 2D, and then if there's some way that we can annotate that two-dimensional space, that's good for our brains.
• 4:39
  Brandon Kase: And then this two-dimensional information needs to get translated into one-dimensional information for the agents, and then the agents are gonna do something that's also one-dimensional back, typically for most things that you're building.
• 4:52
  Brandon Kase: And then you might want to render that as two dimensions to consume it as a human better.
• 4:57
  Brandon Kase: Does that make sense?
• 4:58
  Christine Yip: Yeah, so basically we're converting the shape the whole time to make it easier for the processor to process it.
• 5:07
  Christine Yip: And I guess when the agent outputs 1D, this concept is about changing it to a multidimensional, in this case, two-dimensional output.
• 5:16
  Christine Yip: So we can easily evaluate and review it.
• 5:19
  Christine Yip: And then when we annotate, it gets translated again into 1D, a form that agents can easily process.
• 5:26
  Christine Yip: Yes.
• 5:26
  Christine Yip: That makes sense.
• 5:27
  Brandon Kase: Yeah.
• 5:28
  Brandon Kase: So so maybe we should show some examples.
• 5:30
  Brandon Kase: So there's a there's a really interesting tool that I think more and more people are starting to use.
• 5:37
  Brandon Kase: It's called Planotator.
• 5:39
  Brandon Kase: And I'm gonna just show an example of using it.
• 5:44
  Brandon Kase: So this is about turning the iteration process of nailing down a plan or a spec with the agent into something uh rather than just a one-dimensional conversation with the agent.
• 5:56
  Brandon Kase: As a human, you can communicate in two-dimensional space.
• 6:00
  Brandon Kase: Um, and that gets translated down to one-dimensional space for the agent.
• 6:03
  Brandon Kase: Uh, this is just for an example, we thought, okay, let's just talk about something with the podcast stack.
• 6:09
  Brandon Kase: So I said, hey, Fable, you know, let's make a plan to add Substack to our podcast publishing window.
• 6:15
  Brandon Kase: And because it's Fable, it's a lot of this is probably good.
• 6:19
  Brandon Kase: But just for the purposes of this demo, Plan Notator has these nice tools here.
• 6:23
  Brandon Kase: So we can remove things that we don't care about.
• 6:26
  Brandon Kase: We can comment on things like there is a public API.
• 6:29
  Brandon Kase: Anyway, and you can mark up the plan, right?
• 6:33
  Brandon Kase: And then depending on how you install Plan Notator, you can send the feedback directly back to the agent or you can copy that feedback.
• 6:40
  Brandon Kase: And and what gets copied is a one-dimensional translation of this information.
• 6:44
  Christine Yip: When you have a lot of commons on a huge document or a huge block of text, it's much easier to have a translator like this.
• 6:52
  Christine Yip: So the agent can like receive very precise feedback.
• 6:56
  Christine Yip: This is like one of the basic examples of how this translation between different shapes work.
• 7:01
  Brandon Kase: Here's another example of this.
• 7:03
  Brandon Kase: Now, there's there was this tool called Agentation, which was popular early on in the code and agent revolution that did the same thing, but for websites.
• 7:12
  Brandon Kase: So you could mark in two-dimension space some part of a website to give feedback on.
• 7:16
  Brandon Kase: And at this point, Vex and Claude, both in their first-party apps, have this kind of functionality built in.
• 7:23
  Brandon Kase: And I know a lot of people are still using the CLI for everything.
• 7:28
  Brandon Kase: And I just wanted to show that uh it's something that I think more people should play with if if they haven't, if they're working with something on the front end.
• 7:38
  Brandon Kase: So this is the this is the new superlinear website, and this is Claude.
• 7:42
  Brandon Kase: This is Claude's desktop app.
• 7:44
  Brandon Kase: And so, so this is like at the actual rendering of the website here, and I can hover over different components.
• 7:51
  Brandon Kase: For example, this arrow, click on it, and then make it red and give feedback on that particular component very, very precisely.
• 7:59
  Brandon Kase: And um, you know, and again, it's it's this feedback loop where as a human I see two dimensions, I'm able to mark up and annotate that two-dimensional thing, and then that gets translated into some one-dimensional representation when it goes back.
• 8:15
  Christine Yip: Because it's it's much easier than than to be like, okay, Claude or Codex, this arrow on the top middle side of the page that should be made red, or this card on the right side of the page should have more spacing.
• 8:30
  Christine Yip: Because of that tool that you just showed us, you can do it like much more concisely and also precisely.
• 8:35
  Brandon Kase: Yes, yes.
• 8:36
  Brandon Kase: It's precision, concision.
• 8:37
  Brandon Kase: Also, um, you can focus more on your feedback and less on the instructions about where your feedback applies when you're thinking about it.
• 8:47
  Christine Yip: I feel like the the part where we get the agents to translate their 1D output to multidimensional output, that's something that most of us have already mastered.
• 8:56
  Christine Yip: We just get the agent to generate an image, a schema, or or a mock-up of a website.
• 9:01
  Christine Yip: But what I had not mastered before personally was the annotation step.
• 9:05
  Christine Yip: So I was building this game.
• 9:07
  Christine Yip: It is a tactical game, so there are a lot of rules and you have to like specify everything, make sure everything works correctly for the system to behave well.
• 9:14
  Christine Yip: So during the playtesting, there's there's there's often things that are not exactly correct.
• 9:19
  Christine Yip: So when you give feedback, you have to give your coding agents feedback in text.
• 9:24
  Christine Yip: So unless the workflow helps you capture that spatial context, you have to manually describe the location, the object, the state, the behavior.
• 9:33
  Christine Yip: And that was a really annoying part.
• 9:35
  Christine Yip: I was spending so much time on my feedback step.
• 9:38
  Christine Yip: So basically, when the game was being rendered on a canvas, I told the agents to also build a lightweight feedback or annotation mode from the start.
• 9:47
  Christine Yip: So when I click on, for example, when I click on an enemy, it would already capture like the location, like was it in a forest, the position, the coordinates, what state it is, for example, chasing player, what scripts it touched.
• 10:00
  Christine Yip: And then I would only need to like give the feedback while related to all these things, I actually need the enemy to respond in this way.
• 10:08
  Christine Yip: So basically, that tool translated my spatial click into something that was agent readable context.
• 10:15
  Christine Yip: So yeah, I think my takeaway for me was when when playtesting a game or building a more complex project, and when the feedback is naturally very spatial, then uh the coding agent needs that feedback as a text.
• 10:28
  Christine Yip: So the right workflow is to build that spatial feedback layer into the game.
• 10:32
  Christine Yip: So you can click the thing, capture the context, and then the agent can turn that annotation into code.
• 10:38
  Brandon Kase: Yeah, so cool.
• 10:39
  Brandon Kase: This is the first time I I've heard of someone building a custom tool that that layers on this extra contextual feedback, similar to Plan Notator and the things that are built into Cloud and Codex for static websites.
• 10:53
  Brandon Kase: But it makes me also think there's lots of opportunities for this when people are building whatever they're building, not just games, but but any anything with a higher dimension.
• 11:04
  Brandon Kase: So so I guess we we've been talking about images or like the visual, the visual dimension for most of this episode.
• 11:10
  Brandon Kase: But something that I think is really interesting is there's also other kind of higher dimensional communication forms, right?
• 11:18
  Brandon Kase: So we could extend images into videos, and that's sort of adding a time dimension, right?
• 11:24
  Brandon Kase: And so maybe there's some interesting way that we can translate giving feedback on specific parts of a video into linear text.
• 11:33
  Brandon Kase: And I I don't I don't know of a tool that does this, but it's interesting.
• 11:35
  Brandon Kase: Then there's, you know, maybe uh instead of a static two-dimensional image, we have a static three-dimensional model that we're trying to give precise feedback on.
• 11:46
  Brandon Kase: And and there's another opportunity to build a custom tool in that workflow.
• 11:50
  Christine Yip: Well, I feel I wonder whether this is related to something else that you mentioned the other day, how Gemini understands video now.
• 11:57
  Brandon Kase: Yeah.
• 11:57
  Brandon Kase: Yeah.
• 11:58
  Brandon Kase: So the the Gemini models are the the only kind of like frontier-ish uh model that is multimodal also in in video.
• 12:08
  Brandon Kase: So so text gets tokenized and sent to the model as tokens.
• 12:12
  Brandon Kase: Then modern models can tokenize images.
• 12:14
  Brandon Kase: So they take an image and then that also gets converted into these token streams that get sent to the model.
• 12:19
  Brandon Kase: But and Gemini can tokenize videos.
• 12:21
  Brandon Kase: And that's the that's kind of the other way around.
• 12:24
  Brandon Kase: That's like giving the model this higher dimensional information directly, which is useful for some things, but it's hard to be precise.
• 12:31
  Brandon Kase: And so these tools that convert that higher dimensional information down to a lower dimension make it easier for for models to understand things.
• 12:41
  Brandon Kase: But I I think we will definitely be talking about Gemini's uh video video support in a future episode because it's so so cool and there's so much cool stuff you can do with it.
• 12:51
  Brandon Kase: Um, but you know, we won't talk about it now.
• 12:53
  Christine Yip: I'm excited about that, that future episode.
• 12:56
  Christine Yip: Yeah.
• 12:56
  Christine Yip: So I guess some maybe to like summarize all of this, it's it sounds like a lot of the AI products and workflows, they're moving beyond like plain back and forth chat towards like creating more, like you said, higher dimensional interfaces where maybe humans speak and type linearly, but then when they get um the output from the LLM, they review and interact with it more spatially.
• 13:20
  Brandon Kase: Exactly.
• 13:21
  Brandon Kase: While we're seeing this two dimensional and one dimensional translation right now, I'm really excited to see different communication forms translate between the dimensions, like we were kind of gesturing at earlier.