Latest / AI Fire Daily / #72 Robin: "Watch Me Do It" - How Claude’s New Screen Recorder is Killing Manual Prompt Engineering
Transcript
- 0:00Pause. Imagine this scenario for a second. Instead
- 0:04of typing a massive manual to teach an AI a task,
- 0:07you just showed it once, like training a new
- 0:10coworker by having them watch your screen. Yeah,
- 0:13it completely shifts our baseline. I mean, we're
- 0:15moving away from writing rigid instructions.
- 0:17Welcome to the deep dive. We're thrilled you
- 0:19could join us today. We're exploring a fascinating
- 0:22new tool from Anthropic. Right. It's called the
- 0:24Claude Skill Builder. And it focuses heavily
- 0:27on browser automation. We're going to explore
- 0:29how to safely record a skill. Then we'll walk
- 0:32through a real world YouTube workflow demo. Right.
- 0:34And we'll analyze the surprising output it generates.
- 0:38But we also need to establish a strategic hierarchy.
- 0:41You know, you definitely shouldn't use this feature
- 0:42for everything. Exactly. So let's start at the
- 0:44very beginning. The shift from typing out prompts
- 0:47to just showing the work. We all know the core
- 0:50problem here. Writing out every single click
- 0:52for manual browser automation. tedious. Oh, it's
- 0:56incredibly frustrating. Think about the old way
- 0:57of automating things. You had to describe every
- 1:00tiny step in text. You had to map out every menu
- 1:04item. You had to anticipate like every single
- 1:06edge case. And even then, things usually break.
- 1:09I still wrestle with prompt drift myself. Yeah.
- 1:11especially when trying to write out these massive
- 1:14manual workflows. We all do, because text is
- 1:16just a poor medium for visual tasks. You're trying
- 1:19to translate a two -dimensional visual interface
- 1:22into linear words. It's incredibly inefficient.
- 1:26So this new skill builder aims to solve that
- 1:30problem. But you need to know how to actually
- 1:33access it first. All right. This feature is only
- 1:35available in the Claw desktop app. You won't
- 1:37find it in your normal web browser chat. Yeah,
- 1:40that's a really crucial detail to remember. You
- 1:42have to download and start from the dedicated
- 1:45desktop app. Once you're in, you click the little
- 1:48plus button, then you toggle on the feature called
- 1:50co -work. Finally, you choose the option to record
- 1:53a skill. Pause it. But right before you actually
- 1:55hit that record button, you're met with a warning.
- 1:58A very strict privacy warning. Yes. And listeners
- 2:01need to take that warning very seriously. The
- 2:04recording captures absolutely everything happening
- 2:07on your screen. It captures your precise mouse
- 2:10clicks. It captures your active typing. It even
- 2:13captures your ambient voice. So it's essentially
- 2:16a full environmental capture of your workflow.
- 2:19This means you must actively protect your sensitive
- 2:22data, right? Don't open your password manager
- 2:25during this process. Don't show your personal
- 2:27bank details on screen You really need to keep
- 2:30your client data Totally hidden if it appears
- 2:33on the screen the AI models it it Processes the
- 2:37visual data to understand the context so you
- 2:39have to curate your digital environment beforehand
- 2:41exactly It's like having someone shadow you at
- 2:44your desk watching every single move you have
- 2:46to make sure you're messy confidential papers
- 2:48or put away first. That's the perfect analogy.
- 2:51You're basically pulling back the curtain on
- 2:53your digital life. You're inviting an observer
- 2:54into your workspace. Yeah. But there's another
- 2:57toggle you'll see there, the microphone toggle.
- 2:59You really want to keep that microphone turned
- 3:01on. Which brings up an interesting mechanical
- 3:03question for me. Why is voice narration so crucial
- 3:07compared to just tracking mouse clicks? Because
- 3:09mouse clicks completely lack underlying context.
- 3:13Your narration provides the invisible rules governing
- 3:17those clicks. You know, when you talk through
- 3:18the task out loud, you explain your reasoning.
- 3:21Right. The screen recording only shows the physical
- 3:23steps, but your voice gives Claude the human
- 3:26judgment behind those steps. So the screen shows
- 3:29the what? but your voice provides the why. Exactly.
- 3:32It completely changes the fidelity of the instructions.
- 3:34You're giving the AI the actual motivation behind
- 3:37the action. Bede, let's move to a practical application
- 3:40to see this. We have a fascinating YouTube workflow
- 3:43demo to discuss. This really shows how the tool
- 3:45handles a messy, subjective task. Yeah, this
- 3:48is where the technology gets really fun. The
- 3:51demo task focuses on extracting YouTube comments
- 3:53from a creator's channel, but the user specifically
- 3:56only wants comments from long -form videos. Right.
- 3:59The task is to paste them into a spreadsheet.
- 4:02You wrap the text so it's readable. Then you
- 4:04judge if they're worthy of future video ideas.
- 4:07That sounds like a complete nightmare to write
- 4:09a text prompt for. You'd have to describe the
- 4:11entire YouTube Studio interface. It would take
- 4:13hours to script that out manually. Claude would
- 4:16need to know exactly where to click. It would
- 4:18need to know what specific visual tabs to filter.
- 4:21But instead of typing all that out, you just
- 4:23record it once? Pause. You mentioned filtering
- 4:28out certain types of videos. How does the system
- 4:31actually know what to filter? Well, it knows
- 4:33entirely through your voice narration. While
- 4:36the screen is recording, you literally just tell
- 4:38Claude what to do. You say out loud to skip the
- 4:41shorts tab. Okay. You verbally instruct it to
- 4:43only look at long -form videos. So it maps your
- 4:46verbal command to the visual interface. Right.
- 4:48It sees you click the long -form tab while you
- 4:50explain why. Then you tell it to paste the comment
- 4:53into the right spreadsheet column. It associates
- 4:56the spoken intent with the visual coordinate.
- 4:58Exactly. That makes a lot of sense for basic
- 5:00navigation. But the judgment part is what really
- 5:03fascinates me here. How does it actually judge
- 5:06the quality of a comment? Well, you basically
- 5:08explain your qualitative thinking out loud. Let's
- 5:10look at the specific examples from the demo.
- 5:13Someone leaves a comment asking for a side -by
- 5:15-side comparison. Right. They want to compare
- 5:18Claude Cowork and ChatGPT directly. You say out
- 5:22loud that this is a strong video idea. Because
- 5:24it's a highly relevant searchable topic for that
- 5:27creator. Exactly. You explain that reasoning
- 5:29to the recording. Now, contrast that with a weak
- 5:32video idea. Someone else asks a very specific,
- 5:35small question about project cost. And how do
- 5:38we handle that weak idea? You explain out loud
- 5:41that this is probably less useful. you narrate
- 5:43that a hyper -specific question won't appeal
- 5:45to a broad audience. You're teaching the AI exactly
- 5:48like you would teach a human team member. But
- 5:51let me ask you this. How does the AI manage to
- 5:54replicate subjective human judgment on what makes
- 5:57a good video idea? It anchors directly to the
- 6:00spoken reasoning provided during the recording
- 6:02phase. It listens to your verbal rationale and
- 6:05applies that underlying logic. Okay, so it isn't
- 6:08just guessing. Right. It extracts the heuristic
- 6:10rule from your conversational explanation. You
- 6:13speak your thought process aloud. And the AI
- 6:16mimics your actual reasoning. That's the real
- 6:18magic of this tool. You aren't starting from
- 6:20a blank, intimidating prompt. You just move through
- 6:23the browser naturally. Yeah. You simply explain
- 6:26your thinking as you work. Two -sec silence.
- 6:29So we've successfully recorded the physical task.
- 6:32We've also recorded the underlying logic. Let's
- 6:34talk about processing that recording. Right.
- 6:37What actually happens behind the scenes when
- 6:39you click Done? Claude immediately goes to work
- 6:42processing all that multimodal data. It starts
- 6:44turning that raw recording into a reusable automated
- 6:49skill. And the stats on this processing phase
- 6:52are pretty wild. Oh, they really are. Let me
- 6:54give you the breakdown. In this specific test,
- 6:56it processed roughly 73 recorded steps. Wow.
- 7:00It analyzed about 76 high resolution screenshots
- 7:03of the browser state. It also read and parsed
- 7:07the full voice transcript. That's a massive amount
- 7:10of visual and textual data. It is. And the whole
- 7:12processing phase took just about two minutes.
- 7:14Two minutes. That's incredibly fast for that
- 7:17volume of information. It has to map audio timestamps
- 7:20to visual screen states. What does the actual
- 7:23output look like after those two minutes? It
- 7:25gives you several distinct artifacts. First,
- 7:28it generates a plain language summary of the
- 7:30entire workflow. It essentially tells you what
- 7:32it thinks you just taught it. Second, it creates
- 7:35a highly detailed skill .md file. And what exactly
- 7:39is the function of that markdown file? This file
- 7:42is the actual reusable brain of the automation.
- 7:45It breaks down the step -by -step process into
- 7:48structured rules. It documents exactly how to
- 7:51open the YouTube Studio interface. It notes exactly
- 7:55how to apply the right visual filters. Does it
- 7:57also capture the subjective elements we discussed
- 7:59earlier? Yes. It includes the exact judgment
- 8:02rules for evaluating the comments. It translates
- 8:04your spoken heuristics into logic gates within
- 8:07the file. That's amazing. It also clearly outlines
- 8:10any edge cases. you told it to handle carefully.
- 8:14Which leads us to a really crucial insight from
- 8:16the demo. Something Claude apparently figured
- 8:19out entirely on its own. Right. This is the part
- 8:22that surprised the testers the most. Claude automatically
- 8:25inferred several invisible boundaries for the
- 8:28task. Yeah. understood that it should never actually
- 8:31reply to any comments. It knew it should never
- 8:34click the like button. It also understood it
- 8:37should never delete anything. Whoa. Imagine scaling
- 8:39to a billion queries with that level of contextual
- 8:42understanding. It's genuinely profound when you
- 8:45think about it. Nobody explicitly told the AI
- 8:48to avoid those specific actions. The user never
- 8:51said, don't reply to this comment. It just understood
- 8:54the safe boundaries of the task purely from context.
- 8:57Because the stated goal only to read, copy, and
- 9:00evaluate. Exactly. It recognized that engaging
- 9:03with the audience was outside its mandate. It
- 9:06inferred the negative space of the assignment.
- 9:08That raises a critical question for me about
- 9:10the final product. Is the output just a rigid
- 9:13script of clicks or something more dynamic? It's
- 9:16highly dynamic by design. It builds a robust
- 9:19conceptual model of the task itself. OK. It deeply
- 9:22understands what not to do just as much as what
- 9:25to do. It isn't just memorizing coordinates on
- 9:27a screen. It creates a mental model of the task,
- 9:30not just a blind click track. Precisely. It's
- 9:33intelligent, adaptive navigation. It's not just
- 9:36rote mechanical repetition. Two -sec silence.
- 9:39This all sounds incredibly powerful and intuitive.
- 9:43The natural temptation is to use this tool for
- 9:46absolutely everything. Oh, for sure. But we really
- 9:48need to look at the automation hierarchy. When
- 9:50is this actually the right tool for the job?
- 9:53That is a sobering but deeply necessary reality
- 9:57check. You absolutely must establish a strategic
- 10:00hierarchy for automation. Claude Skill Builder
- 10:03is amazing, but it isn't for everything. So what's
- 10:05the first step in that decision hierarchy? Step
- 10:08one is always looking for official connectors.
- 10:10If you're using major platforms like Gmail, Slack,
- 10:13or Google Drive, you should always use their
- 10:15official built -in connectors first. Why are
- 10:18official connectors always the mandated step
- 10:21one? Because they're highly stable. and reliable.
- 10:23They interact directly with the applications
- 10:25underlying data. They don't rely on visual screen
- 10:27layouts at all. Visual layouts are constantly
- 10:30changing and will break your automation. A simple
- 10:32button color change could ruin a visual workflow.
- 10:36So what happens if a direct connector doesn't
- 10:38exist? What's step two? Step two is turning to
- 10:42third -party automation tools. If no direct integration
- 10:45exists natively within the app itself, you check
- 10:48if the platform works with those established
- 10:51automation services. Things like Zapier MCP.
- 10:54A tool connecting AI to thousands of apps using
- 10:57structured actions. Exactly. Zapier MCP is often
- 11:00the much cleaner, safer choice here. It uses
- 11:03those structured API actions instead of relying
- 11:06on visual clicks. It moves the data securely
- 11:08behind the scenes. Which finally brings us to
- 11:10step three in the hierarchy. Step three is deploying
- 11:13the quad skill builder. This should generally
- 11:15be viewed as your last resort, but I want to
- 11:17emphasize that it's a very powerful last resort.
- 11:20So when do we actually pull the trigger on step
- 11:21three? Where does it shine? You use it for messy,
- 11:24purely visual browser navigation tasks. You use
- 11:27it on apps that have no clean API available.
- 11:30Think about community platforms like school or
- 11:33newsletter builders like Beehive. Places where
- 11:35you normally have to click through five different
- 11:37menus. Right. Think about bespoke internal company
- 11:40dashboards or those clunky outdated admin panels
- 11:43we all absolutely hate using. Yeah. Those are
- 11:46the perfect targets for this kind of visual training.
- 11:49But if this recording feature is so incredibly
- 11:51easy to use, why shouldn't we just use the skill
- 11:54builder for everything? Because visual browser
- 11:56actions are fundamentally fragile by their very
- 11:59nature. They're easily broken by minor user interface
- 12:02updates. They simply cannot compete with the
- 12:05long -term stability of direct data connections.
- 12:08Official connectors are just more stable because
- 12:10they bypass the visual browser interface entirely.
- 12:12Exactly. You always want to use the right tool
- 12:14for the right job. You don't use a hammer to
- 12:16turn a screw. Two secs silence. This deep dive
- 12:20is brought to you by our partners who make complex
- 12:22technology easy to understand. Try their new
- 12:24workflow tools today. Two secs silence. Okay,
- 12:29we're back. We've covered a lot of technical
- 12:31ground today. Let's synthesize the ultimate takeaway
- 12:33from all of this information. What is the big
- 12:35picture? The overarching big idea here is a massive
- 12:39paradigm shift. The era of staring at a blank
- 12:43text prompt is shifting. We're rapidly moving
- 12:46towards something called workflow capture. Workflow
- 12:48capture. I really like that term. It sounds much
- 12:51more active. It perfectly describes the new reality
- 12:54of AI interaction. It means recording the way
- 12:57you actually do your daily work. You just explain
- 13:00your nuanced decisions out loud as you go. You're
- 13:02capturing the human element of the process. Yes.
- 13:05Then you effortlessly turn those repeated manual
- 13:08actions into scalable automation. You're giving
- 13:11the AI a real tangible example of success. Right.
- 13:15You're no longer trying to describe success abstractly.
- 13:18You're giving it your voice, your precise clicks,
- 13:20and your judgment. Exactly. It saves an immense
- 13:22amount of tedious setup time. It makes automating
- 13:25those messy repeated workflows so much easier.
- 13:28Yeah. It lowers the barrier to entry for complex
- 13:31automation significantly. Boss. This leaves me
- 13:35with a final provocative thought for you to ponder
- 13:37today. We've seen how AI can now learn simply
- 13:41by watching us work and by listening to our reasoning
- 13:44out loud. How soon until the main bottleneck
- 13:48in our work is no longer coding? or even our
- 13:50ability to write a clever prompt. Right. How
- 13:53soon until the primary bottleneck is simply our
- 13:55own ability to clearly articulate our decision
- 13:58-making process out loud? Oh, absolutely. We're
- 14:00going to have to get very good at explaining
- 14:02ourselves. That's a fascinating angle. We'll
- 14:04have to truly understand our own tacit knowledge.
- 14:06Exactly. Just like training that brand new coworker,
- 14:09pulling up a chair, showing them the ropes. But
- 14:12this time, the coworker scales infinitely. Thank
- 14:15you so much for joining us on this Deep Dive.