Latest / Superlinear / Give Coding Agents Eyes Beyond the Chat: Screen Recordings, Session History, and Devices

Transcript

Give Coding Agents Eyes Beyond the Chat: Screen Recordings, Session History, and Devices · Superlinear

• 0:54
  Christine: Hey everyone. Gemini can take video as a native input and reason over what happens across the recording.
• 1:01
  Christine: So this means that we do not have to only use transcripts and screenshots as input for agents.
• 1:07
  Christine: We
• 1:07
  Brandon: So
• 1:07
  Christine: can
• 1:08
  Brandon: I'm
• 1:08
  Christine: now give it a recording or video file and the recording itself becomes model context.
• 1:13
  Christine: So we
• 1:13
  Brandon: just gonna
• 1:14
  Christine: will extend
• 1:14
  Brandon: say that
• 1:14
  Christine: this idea across anything that
• 1:16
  Brandon: I'm
• 1:16
  Christine: would help an agent see.
• 1:18
  Christine: You can think of screen of the phone if you're building an app, a video feed of your desk when you're tinkering with something physical
• 1:26
  Brandon: not
• 1:26
  Christine: like a robot.
• 1:27
  Brandon: sure
• 1:28
  Christine: And in this episode, we will help the coding agent break out of the chat by progressively expanding its
• 1:34
  Brandon: I'm
• 1:35
  Christine: field of view.
• 1:36
  Christine: So from the current session to its history to your screen and finally to environments even outside of your computer.
• 1:45
  Christine: Then give it hands to reflect or act on what it sees.
• 1:50
  Christine: So before we get into the examples, Brandon, what is the simplest useful way to reflect on how we work around an agent
• 1:58
  Brandon: tell it to.
• 2:00
  Brandon: So, what do by that whenever you're working with an agent, at any point in time you can just ask, hey can you analyze this session and tell me something that I can do better?
• 2:12
  Brandon: And it'll tell you. And it's really helpful.
• 2:15
  Brandon: What can you or what can you do better? Or are is there some way that we can set up tools or scripts or our environment or adjust our agents at MD to improve your performance.
• 2:29
  Brandon: This is this is the kind of questions that you should be asking your agent from time to time.
• 2:32
  Christine: Yeah. And a coding agent is the best the the best tool to ask since, everything is in its context.
• 2:40
  Brandon: Exactly.
• 2:41
  Brandon: Exactly. So.
• 2:43
  Brandon: So, whenever you're trying to get feedback and improve within a session, the best way to do it is to just ask the agent in that session.
• 2:51
  Brandon: But that's not the only situation or the only way in which it's useful to reflect with your agents. Cuz it's also useful to ask about maybe history across lots of sessions.
• 3:03
  Christine: Yeah.
• 3:05
  Christine: I This reminds me of the Cloud Code reports that Cloud Code generates automatically. Maybe Code X does this too But those kind of reports, they analyze the session summary across multiple sessions you had, and they're really good at surfacing recurring patterns or repeating behaviors.
• 3:32
  Christine: Maybe I can show one So here is the Cloud Code Insights report as you can see it's in this report forty forty-five sessions out of ninety-four were included.
• 3:49
  Christine: Probably the very short sessions were skipped and probably sub-agent work is not analyzed.
• 3:55
  Christine: So this is a view of the main session rather than the entire the entire workflow think it this is one of the interesting things it gave me a prompt to build a skill
• 4:09
  Brandon: sorry.
• 4:09
  Christine: and it's touching on some steps in my workflow that keeps repeating all the time. So I often clear
• 4:17
  Brandon: I'm
• 4:18
  Christine: context during a session and then I start a new session.
• 4:22
  Brandon: not sure what
• 4:22
  Christine: So it
• 4:23
  Brandon: you're saying.
• 4:23
  Christine: noticed that I
• 4:23
  Brandon: I'm sorry
• 4:24
  Christine: keep doing
• 4:24
  Brandon: I'm
• 4:24
  Christine: that across
• 4:25
  Brandon: sorry
• 4:26
  Christine: more than twenty sessions.
• 4:28
  Christine: And I often it also noticed that I do it manually.
• 4:30
  Christine: So here it recommends me to do to create a skill called session start and session close. So it doesn't happen it doesn't need to happen manually
• 4:40
  Brandon: I'm sorry
• 4:40
  Christine: anymore
• 4:40
  Brandon: I'm Yeah, I think I think these kinds of insights are really helpful but the the Cloud Code insight report is fixed and it's just for Cloud, right? So
• 4:52
  Christine: Mm-hmm.
• 4:53
  Brandon: we might also want to ask more specific questions and analyze session history across all of our coding agents if you're like me and you use Codex and Cloud and Pi and play with other tools.
• 5:06
  Brandon: And so you can just ask your agents to look through the histories of all these different things but they're all in different formats and so it'll be kind of clunky.
• 5:16
  Brandon: So there's actually a nice tool by Doodle Steen.
• 5:21
  Brandon: So there's this tool called Coding Agent Session Search or CASS that you can install.
• 5:28
  Brandon: And I've just pulled up the read me and I'll just read the section of why this exists.
• 5:33
  Brandon: I'll read snippets of it anyway so basically the problem is that all of these tools, all these different harnesses, create conversation trails, but the knowledge is scattered and unsearchable. And so it's fragmented, there's little cross-agent visibility, et cetera.
• 5:55
  Brandon: And so what CASS does is it indexes all of your session history from all your different tools and creates a unified knowledge base that's normalized, it's indexed, and it's super-fast at doing specific searches. So I'm going to just show as an example of me having used CASS.
• 6:21
  Brandon: So I asked I asked CASS to analyze my work in the latest podcast video editor that I'm building to edit this podcast in a way that's less annoying And it noticed that it noticed that applying patches fails five percent of the time, which was over fifty the time.
• 6:51
  Brandon: So so it's saying and then it suggests something that I can add to agents dot MD to to make this better. So that's nice
• 7:01
  Christine: Yeah, I like that it's interactive
• 7:02
  Brandon: and say that again
• 7:05
  Christine: I like that it's interactive, that you can ask follow-up questions, like maybe you could like dive deeper or try to understand like what improvement exactly could be.
• 7:17
  Brandon: Exactly. So
• 7:17
  Christine: That
• 7:17
  Brandon: this is just
• 7:18
  Christine: that's
• 7:18
  Brandon: a conversation
• 7:18
  Christine: like a good improvement
• 7:18
  Brandon: I'm having.
• 7:19
  Christine: compared to the Cloud Code report.
• 7:22
  Brandon: this CASS tool, you can use it manually, I'm not even gonna show that, but it just lets you search quickly through your history.
• 7:27
  Brandon: Or you can use it through Cloud Code and you can get insights over your session history across all different kinds of coding agents, which is really cool.
• 7:38
  Brandon: But let's break out of transcripts.
• 7:44
  Brandon: Let's break out of transcripts.
• 7:46
  Christine: Yeah. I actually tried that out, breaking out of the transcripts So because even if I use transcripts it still cannot really see what I the the the transcript captures things that I did in the chat with the coding agent. But my workflow consists of more things than just working in the chat with the coding agent. Like I might open different documents, I might browse, I might do some research in other tools. So I tested another tool called ScreenPipe.
• 8:25
  Christine: I think that tool as well, right, Brendan?
• 8:28
  Brandon: I tried out an early version of ScreenPipe over a year ago, I think it's pretty cool, like ScreenPipe records what you do on your computer and indexes it so that you can search and ask questions about what happened on the screen later. And and that's cool, cuz it can it can capture this kind of information that you're saying that we can later surface to agents. So I haven't actually used it since it briefly, since it came out a year ago. I know the team's been working on it a lot. So what did you find what what did you do with ScreenPipe
• 9:05
  Christine: Yeah. So I was building this workflow to prepare podcast episodes for our podcast. And I had ScreenPipe on the whole time.
• 9:23
  Christine: And so this is here is like here's a good example of what ScreenPipe noticed and it's something that Codex did not notice because Codex cannot
• 9:37
  Brandon: Can you
• 9:37
  Christine: look
• 9:37
  Brandon: read it out?
• 9:38
  Christine: beyond the chat. Yes.
• 9:40
  Brandon: Can you read
• 9:40
  Christine: So ScreenPipe
• 9:40
  Brandon: it out for the listeners?
• 9:41
  Christine: here observes a sequence where I was in a finder to find a Markdown file. And then it saw that I moved to Chatty PT in my browser to ask questions about it.
• 10:01
  Christine: And then I went back to local files in finder to inspect another file or folder that I downloaded after my conversation with Chet TBT.
• 10:13
  Christine: And then I returned to the coding agent and gave it the files that I just downloaded.
• 10:19
  Christine: So basically what here's ScreenPipe is saying and is observing is while you were comparing and transferring material between a planning conversation a Google Docs outline, local Markdown files, and then the workflow package that you used for the coding agent.
• 10:37
  Christine: And it did say like, well probably this cross-checking and moving across different tools was necessary but maybe the repeated navigation was not. And it flagged that the missing capability is that the coding agent, and I was using Codex at the time, Codex lacks direct, convenient access to Google Docs and Jet-CBT planning context.
• 11:02
  Christine: And it suggested that a read-only integration or exported local snapshot would have eliminated the need to visually shuttle between the tools, what I did.
• 11:11
  Christine: And if I scroll down more to like, OK, what should you move into the hardness of the coding agent?
• 11:17
  Christine: What should you automate in a workflow?
• 11:19
  Christine: ScreenPipe told me, well, give it read-only imports for tools like Whisperflow, Granola, Google Docs, Jet-CBT context, all those different tools that I used and that I switched between manually So yeah, I felt that was a good use of ScreenPipe so I would not use ScreenPipe for things that coding agents are already good at answering I did also try asking ScreenPipe like, hey what should I improve around my workflow and I was more thinking, well, I was more thinking about I would get back like really good advice about, well, this is here's where you can implement tests in a better way, this is how you can avoid these kind of errors.
• 12:05
  Christine: But actually that was, Codex was much stronger at it because Codex actually has all your context in its coding session and I guess coding agents are also trained and built in a way that knows the best way of doing harness engineering.
• 12:25
  Christine: But what was but ScreenPipe was good at
• 12:27
  Brandon: Well
• 12:28
  Christine: questions like, hey, what happened every time I left CodeX and then later returned? Or what was I looking for, checking or creating or transferring outside of CodeX and it treated the whole
• 12:40
  Brandon: I'd put that I'm
• 12:41
  Christine: computer
• 12:42
  Brandon: I'm justiminating
• 12:43
  Christine: as the outermost harness around CodeX
• 12:45
  Brandon: You're
• 12:48
  Christine: So yeah yeah. So when I looked at the work around CodeX and at these observations from ScreenPipe, I realized I was acting as a retrieval layer or like the context bridge where I was moving context from one tool to another tool.
• 13:06
  Christine: And when these kind of activities recur ScreenPipe can signal what capabilities is helpful to implement that the harness does not have yet.
• 13:20
  Brandon: Yeah this is this is like a good way to surface like the drudgery of white-collar work.
• 13:28
  Brandon: That's like a phrase that I'm hearing nowadays. Like anything that's repetitive and boring that we're doing in a in a way that we don't even realize like in this case switching between a couple different programs and copy pasting stuff between each other these kinds of insights that we get from a tool like ScreenPipe helps us notice them and so that we can improve our workflow and get rid of these repetitive tasks and just focus on the stuff that matters, which is really nice
• 13:59
  Christine: Yeah. And in this example the first the first thing that I would solve is the retrieval. So give Codex reliable read-only access to other tools where I was manually gathering context but I did notice that in ScreenPipe the recording was still very sparse and event driven.
• 14:18
  Christine: So it was useful to that ScreenPipe flagged where I was gluing different contexts from different windows together but it was not measuring like every action. So it's still a little bit felt like there's something missing.
• 14:39
  Christine: And I hypothesize that it's probably more useful to reason use ScreenPipe to reason over days of usage and finding patterns there and searching for it because it's because the nature of it's more sparse and event-driven recording.
• 14:56
  Brandon: what I want is insights in the moment. So when I'm focused on some work, when I sit down and I say, OK, I wanna I wanna work for ninety minutes and I wanna do something. I wanna achieve a goal. I call this like a focused work session I wanna record what I do during that session and get detailed insights When I when I have these focused work sessions, I want to be able to record exactly what I'm doing and understand and I want an agent to be able to understand what's happening with like a play-by-play.
• 15:33
  Brandon: I want it to know what I'm doing at every moment and from that information give me give me feedback about what I can improve or what I can do better or when I was distracted these kinds of things and and so I think I don't know when this was like late last year around winter time Gemini added support for natively ingesting video tokens.
• 16:01
  Brandon: OK? Which is the coolest thing ever. So
• 16:05
  Christine: what does that mean, natively ingesting
• 16:06
  Brandon: Yeah.
• 16:07
  Christine: video tokens? Is that more efficient?
• 16:09
  Brandon: Yeah.
• 16:09
  Christine: Like what what's the benefit?
• 16:12
  Brandon: because we should talk about how LLMs understand things.
• 16:17
  Brandon: Right? So they they take as input and output tokens, not bytes and tokens are, well, they're usually like a few bytes long.
• 16:33
  Brandon: they cannot natively tokenize video so in order to actually understand what's happening in a video, they have to take that video and break it apart into a series of text and image tokens,
• 16:48
  Christine: And that are you saying that step could be lossy because you you rely on the agents to capture all the relevant context? But maybe, in a corner of the screen recording there's something else happening that the agent doesn't capture if it converts the format from video file to a few sentences in text format.
• 17:14
  Brandon: If you have a specific frame of a video that's important, then it is more precise and it's better for your agent to just grab that frame and put that frame into into its context and process it but if you want the agent to reason about the full video, and yeah it might miss like little details here and there, but but it'll have much more rich information per token than than breaking down the video into frames and looking at specific frames one at a time if it could if it if it can natively consume video, right
• 17:52
  Christine: That's
• 17:52
  Brandon: and
• 17:52
  Christine: so cool.
• 17:53
  Christine: Mm-hmm.
• 17:54
  Brandon: in that context what kind of video do we want to send Gemini? It's a screen recording.
• 17:59
  Brandon: Because I'm using my computer to do work and so I want to record what I'm doing, everything that I'm doing, not just in the agent, not just the chat with the agent but everything, and all that rich information of a focus session we can give to Gemini.
• 18:15
  Brandon: OK but if you if you just naively like go into Quicktime and do a screen recording and try and send the to Gemini. It's not gonna work because you can't you can't send huge videos like too much data the the LLMs like like large video files will take up more tokens in the context window and we only have we have a million tokens with Gemini. So but that
• 18:45
  Christine: Mm-hmm.
• 18:45
  Brandon: that gets filled up fast with video data.
• 18:48
  Brandon: So so let me tell you the precise workflow that I use to screen record and get information about my workflows and I'll give an example of the information from a workflow from today.
• 19:02
  Brandon: How does that sound?
• 19:03
  Christine: Yeah.
• 19:04
  Brandon: OK.
• 19:04
  Christine: Yeah, I wanna see it.
• 19:06
  Brandon: OK. So so the first thing that you do, number one you open Quicktime and you do screen record. That's not really an easy way for me to show this because it's like in the menu of your of your Mac but just open open Quicktime, new screen recording, start, record your whole screen.
• 19:27
  Brandon: OK?
• 19:28
  Brandon: So you do that you do that at the beginning and then you do what you wanna do until you're done.
• 19:33
  Brandon: Maybe you've worked for ninety minutes.
• 19:36
  Brandon: OK and then you stop
• 19:38
  Christine: Is
• 19:38
  Brandon: the
• 19:38
  Christine: that
• 19:38
  Brandon: screen
• 19:38
  Christine: what you
• 19:38
  Brandon: recording.
• 19:38
  Christine: did as well?
• 19:40
  Brandon: That's what I do.
• 19:40
  Brandon: That's what I yeah. this is
• 19:42
  Christine: OK.
• 19:42
  Brandon: this is my workflow that I'm describing.
• 19:44
  Christine: OK.
• 19:44
  Brandon: OK?
• 19:44
  Christine: All right.
• 19:44
  Brandon: So I start a screen recording when I'm done with my with my focus session I'll stop the screen recording.
• 19:52
  Brandon: There's a button on the top in the menu bar that's like a stop icon. You just click on that.
• 19:57
  Brandon: And that creates a large video file on your computer and you can save it so then what I do is I process that file to make it lower resolution and lower the frame rate.
• 20:12
  Brandon: So the idea is we wanna get as much information as we can we wanna get as much information as we can into Gemini and it doesn't need to see a hundred twenty frames per second of me moving my mouse around. That's not really high high quality information.
• 20:29
  Brandon: I find that one frame per second's enough.
• 20:33
  Brandon: So that's just once a second a frame. Now over ninety minutes that's still a lot of information.
• 20:38
  Brandon: Right
• 20:40
  Christine: Mm-hmm.
• 20:41
  Brandon: and then I resize down to seven twenty P. I found that's a good that's a good size where enough details of the screen are visible so like the text is still visible it's also smaller so
• 20:58
  Christine: Yeah,
• 20:58
  Brandon: it
• 20:58
  Christine: and
• 20:58
  Brandon: saves
• 20:58
  Christine: you're
• 20:58
  Brandon: saves
• 20:59
  Christine: working on
• 20:59
  Brandon: tokens.
• 20:59
  Christine: a on a Macbook, right? A Macbook Pro?
• 21:02
  Brandon: I'm on a
• 21:02
  Christine: Not
• 21:02
  Brandon: MacBook
• 21:02
  Christine: a huge
• 21:02
  Brandon: Pro, yeah. So
• 21:03
  Christine: screen
• 21:03
  Brandon: I natively,
• 21:03
  Christine: monitor screen.
• 21:05
  Brandon: yeah,
• 21:05
  Christine: Mm-hmm.
• 21:06
  Brandon: I natively record at whatever, like four K basically. So
• 21:12
  Christine: Mm-hmm.
• 21:13
  Brandon: so I downsize that to seven twenty P and you can do that with FFM PEG. You just ask your agent to use FFM PEG to resize to seven twenty P and and and re-encode at one frame per second.
• 21:27
  Brandon: So that then you have a small video file, a smaller video file it's still too big. If it's ninety minutes long it's still too big. So the next thing that I do is I chop it up into fifteen minute slices.
• 21:38
  Brandon: A fifteen minute slice
• 21:39
  Christine: Mm-hmm.
• 21:40
  Brandon: at seven twenty P one frame per second is small enough that that you can just send it to Gemini.
• 21:47
  Brandon: And so
• 21:47
  Christine: Mm-hmm.
• 21:48
  Brandon: I'm gonna show an example of a session I did today.
• 21:52
  Brandon: This is something that I that I use a lot I actually should use it more, but I've done it maybe ten or twenty times and this is just one from today.
• 22:04
  Brandon: So I well, you'll see what I did because the LLM will tell you. So here's so what I'm sharing right now is the Gemini web UI. So I attached my video file.
• 22:21
  Brandon: In this case it was a ten minute ten minute recording.
• 22:24
  Brandon: And I said, this is a screen recording, help me reflect on my workflow, and then I sent a bunch of other questions to ask.
• 22:31
  Brandon: was I how was I distracted, what was I doing, what learnings could you pull out? Then I said, give me a play-by-play of the beats of what I did during this session and answer my questions.
• 22:42
  Brandon: And you get it.
• 22:44
  Brandon: Check it out so first I reviewed the content outline for this episode I looked at some other coding sessions I did in ChatGPT and in Cloud I fixed a bug in the podcast video editor I asked Cloud Code to research mining session history so that we could so that I could prepare for this episode and I did a bunch of other stuff and from that Gemini was able to determine that my objective was to dogfood this particular content piece on giving agents eyes that we're talking about right now.
• 23:26
  Brandon: And during this session I was actually setting up CASS and starting to play with it.
• 23:35
  Brandon: And Gemini noticed that I was pretty focused and that I was not setting up CAS efficiently because I was struggling with the CLI and what I should have done is sent there there's a command built into the tool you can just feed into CLOJ to understand it. So it gave me a constructive suggestion for how to improve my workflow.
• 24:06
  Brandon: I I try with my focused work sessions to be focused, to have a goal, and I
• 24:13
  Christine: Mm-hmm.
• 24:13
  Brandon: wanna know the the times that I got distracted, what I was distracted by, and how I could improve my workflow going forward if I when I get distracted in that way. And and that that's the kind of thing of like switching off of my terminal, going into the browser. And if I'm researching something related to the work I'm doing, that's not a distraction. But if I'm going on Hacker News that's probably a distraction. And the LLM is able to tell me these things. So
• 24:45
  Christine: Did you see that in the past? Like did
• 24:47
  Brandon: Yes
• 24:47
  Christine: did the did Gemini tell you like hey you're you're actually distracted when you're working.
• 24:54
  Brandon: Yep, exactly. And and it surfaced the kind of the kind of workflow improvements that that you saw with ScreenPipe where it's saying, oh, instead of jumping between all these different products, maybe you could find a way to, get all the information in one place. That's like a common mistake that people make
• 25:15
  Christine: Like
• 25:15
  Brandon: but I
• 25:16
  Christine: right.
• 25:16
  Brandon: think Yeah.
• 25:18
  Christine: And did you change anything after, that observation from Gemini?
• 25:23
  Brandon: Yeah, I've been steadily improving I think like my focused work sessions are more focused and more productive because I'm doing these feedback loops.
• 25:37
  Brandon: I was more self-aware about, one I was self-aware to not be distracted, cuz also I you can feel the kind of like big brother watching you and you'll get in trouble if you if you do it wrong.
• 25:50
  Brandon: That's sort of one thing. But yeah just doing more let me think I think there like a lot of basic stuff just it a lot of things boil down to stop doing things manually and ask your agent to do it for you.
• 26:10
  Christine: Mm-hmm.
• 26:11
  Brandon: Right?
• 26:12
  Christine: What are the
• 26:12
  Brandon: So
• 26:12
  Christine: top things
• 26:12
  Brandon: don't
• 26:13
  Christine: that you've changed in the way you work by using the insights that you got from Gemini
• 26:19
  Brandon: I'm distracted less often.
• 26:21
  Brandon: I instead of copy pasting stuff between lots of different tools, I'm using Markdown files within with my coding agent.
• 26:33
  Brandon: Instead of googling for stuff, I'm asking my coding agent to search. I'm using skills like last thirty days which is a nice skill for
• 26:44
  Christine: Seven six eight
• 26:45
  Brandon: searching
• 26:45
  Christine: nine
• 26:46
  Brandon: for interesting things
• 26:47
  Christine: zero
• 26:47
  Brandon: across Reddit and X and YouTube
• 26:49
  Christine: two one eight
• 26:49
  Brandon: and things like that
• 26:50
  Christine: one
• 26:51
  Brandon: basically I'm getting more
• 26:53
  Christine: seven
• 26:53
  Brandon: and
• 26:53
  Christine: zero
• 26:53
  Brandon: more of the stuff that's happening outside of my agent into my agent. I think that's like a big category of stuff.
• 27:00
  Christine: And this is based on
• 27:01
  Brandon: And
• 27:01
  Christine: the insights and feedback you got from Gemini after it analyzed
• 27:05
  Brandon: Yeah.
• 27:05
  Christine: how you work.
• 27:06
  Brandon: Yeah, it's it's something that it's obvious in hindsight. All these things are obvious in hindsight, even though any of these things, like once we know about them they're
• 27:15
  Christine: Yeah.
• 27:15
  Brandon: obvious, but I think them being reflected back to you by someone who's by this like you know magical entity that's observing that helps you that helps you be aware of these things and then encourages you to take action to improve them
• 27:32
  Christine: I guess the magic is that it can because it observes more things, it now has the ability to like take action or help you take action.
• 27:43
  Christine: And yeah, I can totally imagine that even I'm like sometimes when I'm building I'm just an automatic pilot and I don't really stop
• 27:50
  Brandon: Mm-hmm.
• 27:50
  Christine: to think and improve the workflow and it's helpful to give the AI eyes to help observe it
• 27:58
  Brandon: Mm-hmm.
• 27:58
  Christine: for me as well.
• 28:00
  Brandon: And you
• 28:00
  Christine: No,
• 28:00
  Brandon: know,
• 28:00
  Christine: that
• 28:01
  Brandon: I've
• 28:01
  Christine: totally
• 28:01
  Brandon: I've
• 28:01
  Christine: makes
• 28:01
  Brandon: been
• 28:01
  Christine: sense. Mm-hmm.
• 28:03
  Brandon: I've been I've been capturing the outputs from Gemini and like putting them in a folder and then I can also ask questions across different focus sessions it's a little lossy cuz it's not reasoning over the exact video at that point but I'm like distilling the video down to the information I get from Gemini
• 28:22
  Christine: Mm-hmm.
• 28:22
  Brandon: and then reasoning over that and I've even I started building a tool to do this automatically for me but I got distracted by my wedding or something so maybe I should pick that up again.
• 28:36
  Christine: Yeah, that would be interesting I would be curious and if someone wants to try this workflow or this the way how you use Gemini, would you recommend that they do it in the same way as you, like record the whole focused work session?
• 28:52
  Christine: Or should they just record their most repeated part of the workflow what do you think
• 28:59
  Brandon: Yeah, I think a way to think about this is like you get you can get information in fifteen minute chunks. So I think it's still useful to to send like an hour or two hours worth of information in fifteen minute chunks But yeah, if you if you have a particular workflow that you do that you're looking for feedback on or that you wanna try and automate but you don't know how then you can, just record that particular part.
• 29:32
  Christine: Yeah. Because
• 29:32
  Brandon: And.
• 29:33
  Christine: I could Yeah. No? Sorry. Go for it.
• 29:37
  Brandon: Yeah, Go ahead
• 29:38
  Christine: Oh, OK.
• 29:39
  Christine: No, I was I just wanted to say that I could imagine if you have a repeated representative workflow, part of the workflow that you do all the time it could you could be curious like how you can improve it because it could be quite impactful to the to to how you work and it also sounds like you cannot it's probably too much context if you want to feed Gemini like a week worth of video recordings and tell it to surface the recurring patterns. So basically to shrink that context you give to Codex you can do it intentionally by just identifying, OK, this is what I do all the time, how do I prove this?
• 30:24
  Brandon: Mm-hmm.
• 30:25
  Brandon: Yeah. Or what I do, which is like distill videos down using Gemini and then take those distillations of all those moments and put them in a folder and then asking Codex or Claude to analyze it.
• 30:39
  Brandon: Does that make sense?
• 30:40
  Christine: Wait, how do you distill it? Like I you mentioned how you cut it into like fifteen minute parts.
• 30:46
  Brandon: Mm-hmm
• 30:47
  Christine: Do you distill it further?
• 30:50
  Brandon: By distill I just mean copy and pasting the output from Gemini into a folder.
• 30:58
  Christine: Right. And then it's a log of your patterns and recurring behavior.
• 31:03
  Brandon: Yeah well specifically I will ask Gemini for very detailed play-by-play of every moment of this video and I store that. So that's a cuz you can imagine like a fifteen minute video is you know ten megabytes or so, fifteen megabytes of information that is illegible to cloud and codex
• 31:26
  Christine: Mm-hmm.
• 31:27
  Brandon: and if I give that to Gemini and I ask for that like play-by-play of what I did in every moment, as I showed before, you get a nice a few pages of text that describes what was happening in detail and then I can store that and that's whatever a thousand tokens, two thousand tokens worth of information so
• 31:47
  Christine: Gotcha.
• 31:47
  Brandon: I can store weeks worth of those in one agent's context to you
• 31:54
  Christine: Right. And you say
• 31:55
  Brandon: know
• 31:55
  Christine: like
• 31:55
  Brandon: understand
• 31:55
  Christine: illegible
• 31:55
  Brandon: patterns and things
• 31:57
  Christine: And we when you say illegible to Codex and Claude, you're talking about things that happen outside of the chat, right? So maybe working in a browser, going to forums other patterns that turn that that show that you're making a detour.
• 32:17
  Brandon: Yeah. specifically this screen recording video is unreadable directly by these tools. Yes, they can if you if you tell them to understand it, what it what Cloud Encode X will do is they'll they'll break down the videos into frames and then try and read the specific frames to get a sense of what's going on in the video, but it's more lossy that one way to put it is like the distillation if you if you ask Claude, given this fifteen minute video, tell me what's happening beat by beat and you ask Gemini the same thing, the answer from Gemini is much more detailed because Gemini can natively understand the video.
• 32:57
  Christine: Yeah.
• 32:59
  Christine: So, it's really interesting because in this all the example that you mentioned, you changed the object of reflection to something different. Like, previously the object was the transcript or the session summary. And now the object of reflection became the whole recording and we are looking at how people behaved during the entire act of building overall and not just in that chat with the coding agents.
• 33:24
  Christine: That's pretty cool
• 33:26
  Brandon: Yeah, it's so cool. And we can use that information to automate the meaningless, repetitive drudgery away and just leave people with the most interesting parts of their work.
• 33:39
  Brandon: And I think that is so important because why would you wanna be doing the thing that is repetitive and basic? Then you're not growing, you're not learning you're not you're not achieving everything that you can achieve, So
• 33:56
  Christine: Mm-hmm.
• 33:56
  Brandon: so we need to automate more. So let's talk more about
• 33:59
  Christine: Yeah.
• 34:00
  Brandon: that
• 34:01
  Christine: I do I do want to offload more of the less meaningful task to the agents. So I do a version of this too So I'm building this app that will run on my phone.
• 34:16
  Christine: And in that workflow I connect phones, my two old phones, as test rigs.
• 34:22
  Christine: And then my coding agents can control those phones, it can run the features of the app that it just built, and then observe what happens on the real device. And both phones they have different like specs so you can like see how it performs and whether there are any errors.
• 34:36
  Christine: And that way you give your agents not only eyes to look outside of their chat but also you can give them hands in that in that environment where that app will run and act on reflect and act on what they see
• 34:53
  Brandon: Yeah, I think that's I think that's so that's so that's so powerful.
• 34:57
  Brandon: Like I think a lot of people when they're working on apps they have a good feedback loop set up with an emulator or simulator like an Android emulator or an iOS simulator.
• 35:08
  Brandon: But but they but maybe they don't realize you can just plug in a phone and like you said then that allows the computer to control it and if the computer can control it then your agent can control it.
• 35:21
  Christine: And actually that reminds me of this example from Peter Steinberger, the creator of OpenClaw.
• 35:28
  Christine: Here he gave his agent a webcam access so he could end-to-end test some voice wake command on an ESP thirty-two open-claw node. So the agent would repeatedly test the thing by saying, hi ESP, and then it would watch what happens on a physical device and continue debugging it
• 35:50
  Brandon: So cool.
• 35:51
  Christine: so it, yeah. In the same way, it was giving the agent eyes outside of the chat the chat and it gave the agent hands to act on it.
• 36:05
  Brandon: anytime you're building something in the real world you can connect a camera, have the camera point at that thing and whatever you're building, if you can control it by, connecting to something then
• 36:21
  Christine: Exactly.
• 36:21
  Brandon: Oh, look, you have another example.
• 36:22
  Christine: And this is, yeah, I have another example, like here you here
• 36:26
  Brandon: Well
• 36:26
  Christine: you see another
• 36:27
  Brandon: I'd put
• 36:28
  Christine: someone
• 36:28
  Brandon: that
• 36:29
  Christine: else who
• 36:29
  Brandon: I'm
• 36:29
  Christine: have mounted
• 36:30
  Brandon: I'm just I'm
• 36:30
  Christine: a
• 36:30
  Brandon: visiting
• 36:30
  Christine: desk camera to an agent so it could inspect these external devices like Remarkable Tablet and also ESP thirty-two without human taking photos all the time.
• 36:40
  Christine: And actually I do this often
• 36:43
  Brandon: your
• 36:43
  Christine: too sometimes when I see things that happen
• 36:45
  Brandon: site
• 36:45
  Christine: outside of the chat window or even outside of my computer. I just keep taking pictures. But even that you could automate away
• 36:52
  Brandon: to
• 36:54
  Christine: like
• 36:54
  Brandon: put
• 36:54
  Christine: you said previously don't spend your human time doing these meaningful not meaningful steps but give your agent the eyes to do it itself
• 37:06
  Brandon: And I bet, a lot of these workflows you can get something that works well enough by piping the webcam footage into something that like Clutter Codex controls and it can take screenshots and or look at different frames.
• 37:20
  Brandon: But I think what's really interesting that not enough people are doing is having an intermediate step of sending that video data to Gemini and getting the insights out of Gemini and then feeding that back into your coding agent of choice. I think that's
• 37:35
  Christine: What is the difference
• 37:36
  Brandon: the.
• 37:37
  Christine: of skipping that step and including that step?
• 37:40
  Brandon: If you skip that step then you're you're losing more information from the video data before it goes into cloud or codecs, because cloud and codecs can't natively understand the video data. They have to be, they have to break it apart into frames and they have to sample those frames in some way.
• 37:59
  Brandon: But Gemini can take a continuous video stream, if it's low enough quality, low enough frame rate,
• 38:06
  Christine: Mm-hmm.
• 38:07
  Brandon: even long, streams of video and natively
• 38:10
  Christine: Mm-hmm.
• 38:10
  Brandon: process them, natively break them into tokens and understand them. And so
• 38:14
  Christine: Like in these examples
• 38:15
  Brandon: and so you get.
• 38:16
  Christine: where you have a webcam
• 38:18
  Brandon: Mm-hmm.
• 38:18
  Christine: recording something outside of what's happening on your computer, what do you have an example what Gemini would catch and other tools would miss
• 38:30
  Brandon: I'd have to try it to like know for sure but but I have I have I have tried this before
• 38:36
  Christine: I don't
• 38:36
  Brandon: on
• 38:37
  Christine: know.
• 38:39
  Brandon: on screen recordings.
• 38:39
  Brandon: And the the information that Cloud and Codex get out of the video is just less it's less detailed. And so the the same thing would happen for any video file. So any video of the real world as well. You'd get you'd get less precise information or the information that you get would use more of the context of Cloud and Codex and there would be less context available to solve whatever problem came up in that video, if that makes sense.
• 39:09
  Christine: Mm-hmm.
• 39:10
  Christine: OK. Yeah. That helps giving an idea of the difference the what you would get out of it of adding in, injecting that step of feeding it to Gemini.
• 39:21
  Christine: Yeah. So it
• 39:22
  Brandon: Cool
• 39:22
  Christine: looks like, screen recordings, cameras and other device outputs like my like the the phones, the old phones, the test rig, they can all, there're all ways to give your agent eyes outside of the chat.
• 39:37
  Christine: And I guess in a in a case of the test rig, if you make it controllable, you can even give your agent hands to act on it. And I guess having more eyes, like you said and even like the way of translating what it sees to what it interprets can reveal more patterns and problems that maybe never occur to you when working on it or never appear in an agent's chat so
• 40:07
  Brandon: Yeah, and
• 40:09
  Christine: Mm-hmm.
• 40:10
  Brandon: this I guess like throughout this conversation we've mostly been talking about having moments of time where you as the operator, the human, is going and instructing your agents or doing some action to get this feedback I guess the first example of the agent acting on its own with its hands is the one that Christine was just saying where the agent could control the the phone manually or some of those examples that we saw on Twitter.
• 40:41
  Brandon: But you can also have your agent proactively surface feedback as it's doing its work.
• 40:47
  Christine: Mm-hmm.
• 40:49
  Brandon: Actually, Cloud just introduced a new feature a few days ago
• 40:55
  Christine: Mm-hmm.
• 40:56
  Brandon: which I'll talk through. So Cloud Code can proactively draft feedback when it finds bugs within Cloud on its own.
• 41:06
  Brandon: OK?
• 41:06
  Christine: Mm-hmm.
• 41:07
  Brandon: So
• 41:07
  Christine: Mm.
• 41:08
  Brandon: this is cool, and and actually this is something that Lovable's been doing for six months, I guess. So Lovable also does this where the Lovable agent
• 41:22
  Christine: Mm-hmm
• 41:23
  Brandon: when a user is making a web app, it has a way to give feedback to the developers to Lovable, to the company that surfaces feedback when there's issues.
• 41:32
  Brandon: Which is good because then
• 41:34
  Christine: Mm-hmm. Mm.
• 41:35
  Brandon: proactively the team can triage them and fix those errors.
• 41:40
  Brandon: There's nothing about this workflow that's specific to an application getting the feedback versus you as the user of the agent. And there's been there's been a lot of examples of these kinds of
• 41:53
  Christine: Mm-hmm
• 41:53
  Brandon: of projects, but I'll just share one that came up earlier this month.
• 41:58
  Brandon: Frog.
• 42:00
  Brandon: Automated friction logging for agents your agents hit paper cuts all day and then they work around them and they forget. And this frog tool turns them into tract issues.
• 42:12
  Brandon: But you don't you don't actually need a tool you can just tell your agent you can have this in your agents dot MD for example to record anytime it runs into friction anytime it wants to complain write that down to a file call it vent dot MD or complain dot MD or friction dot MD and and have your agent just write down things that it encounters and then you can have I guess at a later can go and process that or you can build automation to automatically triage that information as it comes in.
• 42:49
  Christine: So the lesson is like don't stop at just giving your agent more eyes, more ways to observe, have more observations, but also help it to act on it
• 43:00
  Brandon: Give it the hands
• 43:00
  Christine: I and
• 43:01
  Brandon: to observe by itself
• 43:04
  Christine: Yeah.
• 43:04
  Brandon: in the moment.
• 43:06
  Christine: To observe and act on it, right? Act on what it observed. Like either it's like building these bug reports, logging them turn them into skills like what we saw earlier but there's probably, you could also give your agent computer use
• 43:25
  Brandon: Mm-hmm.
• 43:26
  Christine: access to computer use because then there's also if you observe if you give the agent eyes to observe things outside of the chat history if you give it I could imagine if you give it computer use also do things outside of the chat history it will make it much more efficient and impactful
• 43:44
  Brandon: Yeah, computer use is expensive and slow, but it's useful if it automates something that you struggle in some other way to automate so it's definitely it's a good tool and it's better than doing it by yourself manually, but it's not as good as being able to automate that task with a CLI tool, for example.
• 44:07
  Christine: Mm-hmm.
• 44:08
  Christine: Only when there's no CLI tool then computer use is the last resort.
• 44:14
  Brandon: Yeah
• 44:16
  Christine: Yeah. So many ways to give your agent more eyes and also more hands.
• 44:22
  Christine: Alright So the other day when I opened Clod, this is what I saw it. this is what I saw it said, skip the repeat work, record your screen while you do a task once, and Clod will turn the recording into a skill it can run again.
• 44:37
  Christine: So again here we
• 44:39
  Brandon: Yeah.
• 44:39
  Christine: we see
• 44:39
  Brandon: Yeah,
• 44:39
  Christine: this trend where we are giving the agents more eyes
• 44:42
  Brandon: Yeah,
• 44:43
  Christine: outside of just the transcripts or a
• 44:46
  Brandon: Yeah,
• 44:46
  Christine: summary.
• 44:47
  Brandon: yeah.
• 44:48
  Christine: And the agent
• 44:49
  Brandon: Yeah.
• 44:49
  Christine: is is acting on it after getting these observations. So here it turns whatever it observes in the in the screen recording.
• 45:01
  Christine: So maybe it's like a repeating behavior or
• 45:05
  Brandon: That's the
• 45:05
  Christine: recurring
• 45:05
  Brandon: same
• 45:05
  Christine: errors
• 45:06
  Brandon: thing.
• 45:06
  Christine: and
• 45:06
  Brandon: That's
• 45:06
  Christine: then it turns
• 45:06
  Brandon: the
• 45:06
  Christine: that into scale to improve the workflow.
• 45:09
  Christine: So they this is basically a progression of field of view and an ability to act.
• 45:16
  Christine: Where would you stop adding visibility or control though
• 45:21
  Brandon: Like everything you can take it too far if you add too much recording, too many controls, too many tools, then the agents get confused there's too much context to wade through and there's too many choices to make about what action to take and then our agents become useless. So it's this is one of those things that you wanna add incrementally and you wanna be precise when you're doing a particular task like we've talked about before.
• 45:47
  Brandon: Only expose the tools that you need and only record the things that you need to complete the task that you have at hand.
• 45:55
  Christine: yeah. I guess the takeaway is the coding agent conversation is only one view of the work.
• 46:01
  Christine: Give your agent more eyes beyond the chat through session history, screen recordings, cameras or other device output and this broader view that the agent gets can reveal patterns and problems that you did not know to look for. And then, we're safe, give them hands so they can actually act and reflect on the new findings, like through controllable test rigs or computer use or just places to complain so they can act on what they see, test results, and close more of the feedback loop themselves.
• 46:34
  Brandon: And if you have video data, give that video data to a model that can actually understand the video data, like Gemini.
• 46:41
  Christine: Yes.
• 46:43
  Christine: Alright, that's it for this episode of Superlinear.
• 46:46
  Christine: If you try this, we are curious what did you discover that you can automate and what would you want your agents to see and act on. Tell us in the comments and if you want more real examples of how we are improving our agents' workflows, subscribe so you can catch the next episode.
• 47:06
  Christine: See you next time.
• 47:07
  Brandon: Bye everyone.