Latest / AI Fire Daily / #87 Robin: How AI Browser Agents are Doing the Clicking, Scrolling, and Downloading For You 🖱️

Transcript

#87 Robin: How AI Browser Agents are Doing the Clicking, Scrolling, and Downloading For You 🖱️ · AI Fire Daily

• 0:00
  Imagine an AI running 85 unique quality assurance
• 0:04
  checks on a web app. Beat. Then downloading monthly
• 0:09
  bank statements from a secure portal that has
• 0:11
  absolutely no API. Right, just visually navigating
• 0:15
  it. Exactly. And finally turning a raw YouTube
• 0:18
  video into a perfectly formatted X article draft
• 0:21
  with screenshots placed exactly where they belong.
• 0:24
  All of this happening from one single setup.
• 0:27
  It really completely shifts our mental model
• 0:29
  of how we interact with software. It's a genuine
• 0:32
  leap forward. Welcome to the Deep Dive. Today
• 0:35
  we're figuring out how this is actually happening.
• 0:38
  We're looking at AI browser agents, specifically
• 0:41
  the Codex desktop setup. Yeah, the one's been
• 0:44
  making waves recently. Right. And we're exploring
• 0:46
  how they finally learn to see and click the web
• 0:48
  exactly the way you and I do. For years, the
• 0:51
  boundary of automation just stopped at the API.
• 0:53
  If an app didn't have a back door for code to
• 0:55
  talk to code, you were kind of out of luck. You
• 0:57
  really were. But now that boundary is entirely
• 0:59
  gone. Yeah, and it's a fundamental shift in capability.
• 1:03
  I mean, we're moving from systems that read underlying
• 1:06
  code to systems that visually reason with rendered
• 1:09
  pixels. We have a really clear roadmap for this
• 1:12
  deep dive. We're going to unpack three very different
• 1:16
  real -world workflows to understand the mechanics
• 1:19
  here. Got to keep it practical. Always. We'll
• 1:22
  start with automated QA testing, move into the
• 1:24
  strict security of banking, and then tackle the
• 1:27
  inherently messy world of content publishing.
• 1:29
  And then... will break down a core decision framework
• 1:32
  so you know exactly when to unleash an agent
• 1:35
  versus relying on a standard script. because
• 1:38
  it's all about practical application. We want
• 1:40
  to keep your AI automation pipelines actually
• 1:43
  useful instead of letting them turn into a chaotic
• 1:46
  map. Exactly. Nobody wants the AI just wildly
• 1:49
  clicking around their computer. No, definitely
• 1:51
  not. So let's start with the mechanics of the
• 1:53
  setup itself. We're talking about the Codex desktop
• 1:55
  application. When you open it, you get this split
• 1:59
  screen. You have your chat interface on one side
• 2:01
  and an in -app browser on the other. And that
• 2:03
  side -by -side view is critical for building
• 2:06
  trust. Both you and the agent are looking at
• 2:08
  the exact same rendered page at the exact same
• 2:11
  time. Right, you type your plain English instruction
• 2:13
  on the left and just watch Codex physically control
• 2:16
  the browser on the right. Yeah, you're actually
• 2:18
  watching every single click, scroll, and form
• 2:21
  fill happen in real time. It's wild to see. It
• 2:24
  really is. But before we get into unleashing
• 2:26
  this thing on actual workflows, we need to establish
• 2:29
  a mental model for automation. Yes, because having
• 2:33
  a powerful tool is useless if you deploy it wrong,
• 2:36
  you know? The golden rule here is incredibly
• 2:38
  simple. API first, browser use second. Let's
• 2:42
  unpack that a bit. An API is a direct code connection
• 2:45
  for apps to talk. It's like a secret backstage
• 2:48
  door. Perfect analogy, yeah. It's incredibly
• 2:51
  fast and it's perfectly stable. But browser use,
• 2:54
  that's like standing in line at the front door
• 2:56
  using human eyes to navigate the lobby. That's
• 2:58
  exactly it. I mean, most established tools already
• 3:01
  have APIs. Codex actually has hundreds of built
• 3:04
  -in plugins for them. So you always try the API
• 3:06
  first. Because it's cheaper and faster. Right.
• 3:09
  Browser use is your fallback power move. You
• 3:11
  deploy it when a platform has no API or when
• 3:14
  the API is heavily restricted. Or when an API
• 3:17
  makes simple user interface work incredibly painful.
• 3:21
  which happens a lot with formatting text or images.
• 3:24
  Oh, constant. So there are two modes of operation
• 3:26
  we need to clearly define here. Headed mode and
• 3:29
  headless mode. Right. Headed mode means you watch
• 3:32
  codecs work visually in real time. This is your
• 3:35
  training phase. And headless mode means running
• 3:37
  silently in the background without a visual interface.
• 3:40
  Exactly. That's the handoff phase, where the
• 3:42
  computer just does the work silently. I have
• 3:45
  to admit, I still wrestle with prompt drift myself
• 3:47
  when giving these vision agents instructions.
• 3:49
  Oh, we all do. You think you're being crystal
• 3:51
  clear. You say, ah, summarize this page. But
• 3:55
  the AI interprets it entirely differently and
• 3:57
  starts clicking into menus you never intended
• 3:59
  or writing a 10 page essay. Yeah. Which is why
• 4:02
  you really need that visual confirmation at first.
• 4:05
  To ensure it graphs the context. You absolutely
• 4:09
  do. You watch every move in headed mode. You
• 4:11
  fix the instructions when it inevitably misunderstands
• 4:14
  a layout. You're essentially making sure the
• 4:16
  agent truly understands the visual hierarchy
• 4:18
  of the page. Only when the workflow is perfectly
• 4:21
  stable do you tell it to run headless. And what's
• 4:24
  brilliant is that Codex even shows its own little
• 4:26
  ghost cursor when it takes control of the screen.
• 4:29
  That's a highly reassuring touch. You always
• 4:31
  know exactly who is driving the machine. Yeah.
• 4:33
  beat. So how do you ensure the agent actually
• 4:36
  understands the page before you trust it to do
• 4:39
  real work? Well, you basically observe its cursor
• 4:42
  movements during those initial headed runs to
• 4:45
  verify its visual logic and decision making.
• 4:48
  So headed mode first to watch it learn, then
• 4:50
  let it fly headless? Precisely. It builds the
• 4:53
  necessary trust before you step away. Okay, if
• 4:55
  we understand how the agent sees a page, let's
• 4:58
  look at what happens when we unleash it. Let's
• 5:00
  do it. We want to see how it finds both visual
• 5:02
  and functional flaws in a piece of software.
• 5:05
  This brings us to our first major workflow. Automated
• 5:08
  QA testing. Essentially letting Codex try to
• 5:11
  completely break your app. Right. Think about
• 5:14
  a typical development environment. You spin up
• 5:17
  a form submission app on your local network.
• 5:19
  On your local host, yeah. In this scenario, Codex
• 5:22
  ran 85 highly -focused QHX, mimicking a real
• 5:26
  user flow. Now, a standard automation tool looks
• 5:30
  at the DOM, which is the underlying code structure
• 5:33
  of a web page. But Codex uses Vision. Which is
• 5:36
  completely different. Entirely. At first, this
• 5:39
  test page had some very obvious UI problems,
• 5:42
  like the spacing looked entirely wrong. A phone
• 5:45
  number field was visually out of bounds, physically
• 5:48
  covering the company text field. These are exactly
• 5:51
  those annoying layout bugs you spot by eye. The
• 5:53
  ones you usually try to explain terribly in a
• 5:55
  Slack message to a frustrated developer. Yeah,
• 5:58
  the box is weirdly over the other box. But Codex
• 6:01
  has a specific feature called annotate to solve
• 6:03
  this. How does that work? You open the app in
• 6:06
  the in -app browser, you click the annotate button,
• 6:09
  you visually click the broken element on the
• 6:11
  screen. Like pointing a finger. Exactly. Then
• 6:13
  you just describe the issue in plain language.
• 6:15
  You type, this phone number box is out of bounds
• 6:17
  and covered in the company box. Fix the spacing.
• 6:20
  And you can add multiple annotations across the
• 6:22
  screen, right? One for bad spacing, one for poor
• 6:24
  alignment, one for broken input field. Right.
• 6:27
  And then you send all of those notes into the
• 6:29
  chat. Codex processes the visual context, dives
• 6:32
  into your files, and turns those visual notes
• 6:35
  directly into coding tasks. It literally writes
• 6:39
  the fix. It does. But that's just the superficial
• 6:42
  visual pass. The real test starts when you ask
• 6:44
  it to behave like a user. Oh yeah. You give it
• 6:46
  a beautifully chaotic prompt. You say, use your
• 6:49
  browser and test the heck out of this app. Try
• 6:51
  your best to break it. Report every bug you find
• 6:54
  and what needs fixing. Let me push back slightly
• 6:56
  on this though, because we already have entire
• 6:59
  teams dedicated to human QA testing. True. And
• 7:03
  humans inherently know what a functional app
• 7:05
  is supposed to feel like. Isn't AI simulating
• 7:08
  chaos actually better than a trained human eye?
• 7:10
  It often is, primarily because of fatigue. Yeah.
• 7:13
  I mean, human QA is incredibly important, but
• 7:16
  humans usually test the happy path. Right, the
• 7:18
  way it's supposed to work. Exactly. We maybe
• 7:20
  check five or six edge cases before we get tired
• 7:23
  or bored. Real users do incredibly strange things.
• 7:27
  They skip fields, they navigate backward, they
• 7:30
  hit submit way too early. They mash the keyboard
• 7:33
  instead of clicking buttons. All of that. And
• 7:35
  an AI told to break the app doesn't get tired.
• 7:37
  It simulates that human chaos perfectly and relentlessly.
• 7:41
  The results from this specific chaos testing
• 7:43
  are actually fascinating. Yeah. It found that
• 7:46
  invalid contact data successfully passed validation.
• 7:50
  Bad user data could corrupt the backend system.
• 7:53
  Wow. It also found that the UK country code silently
• 7:57
  reset back to the US after a user edited their
• 8:00
  profile. It also dynamically found mobile responsiveness
• 8:03
  issues, which is wild. Yeah, it resized the browser
• 8:05
  and switched to mobile view entirely on its own.
• 8:08
  Nobody specifically asked it to check that layout.
• 8:11
  The app looked completely fine on a desktop monitor,
• 8:13
  but the navigation broke entirely on phones and
• 8:16
  the vision model recognized the overlap. It's
• 8:19
  just seeing the screen. And there's a scaling
• 8:21
  prompt mentioned in the setup that feels particularly
• 8:23
  powerful. It's called the goal prompt. Right.
• 8:25
  You literally type slash goal into the chat.
• 8:28
  Goal. Don't stop until you have tested 100 unique
• 8:32
  edge case scenarios. You tell the agent to try
• 8:34
  invalid data. empty fields, strange keyboard
• 8:37
  submission methods, restart behavior. And the
• 8:40
  LLM reasons through every permutation. It returns
• 8:43
  every single bug with precise steps to reproduce
• 8:46
  it and suggested code fixes. And you just send
• 8:49
  the aggregated list back into the chat and tell
• 8:51
  it to fix the code base. Pete, what makes this
• 8:54
  better than a standard automated testing script?
• 8:57
  Well, standard scripts break the moment a developer
• 8:59
  moves a button. Agents adapt visually to page
• 9:03
  changes dynamically. Got it. Real users do weird
• 9:06
  stuff, and agents simulate that human chaos perfectly.
• 9:10
  Exactly. It tests the application visually, exactly
• 9:13
  as a human would. So breaking a local development
• 9:15
  app is relatively low stakes, but how does this
• 9:17
  agent handle high security environments? That's
• 9:20
  the real test. We need to talk about downloading
• 9:22
  monthly statements from a bank. This is a massive
• 9:24
  recurring pain point for operators. We're looking
• 9:27
  at the Relay Bank dashboard use case. Logging
• 9:30
  in securely, navigating to accounts, clicking
• 9:33
  into statements, and downloading specific CSV
• 9:36
  files. And all of this happening without an API?
• 9:39
  There's no custom code integration. It's just
• 9:41
  a plain English instruction. Right. The prompt
• 9:44
  is simply, open Relay. Go to the dashboard, Accounts,
• 9:47
  Statements. Download the latest statements for
• 9:50
  these two specific accounts as CSV files, then
• 9:53
  save them securely into this local folder. A
• 9:55
  normal macro, which is a script automating a
• 9:58
  fixed sequence of clicks, needs the buttons in
• 10:00
  the exact same pixel location every single time.
• 10:03
  Exactly. If the bank shows a pop -up ad... the
• 10:05
  macro clicks the ad and crashes. Right. But Codex
• 10:08
  handles the real dynamic interface. It reads
• 10:11
  the rendered page as it moves through the workflow.
• 10:13
  However, banking platforms introduce a very specific
• 10:16
  intentional hurdle, the login problem. Banks
• 10:19
  forcefully log you out. It's a strict security
• 10:21
  measure. So if Relay opens up the login screen
• 10:24
  again, Codex needs a safe way to sign back in.
• 10:26
  And you absolutely do not want to paste your
• 10:28
  raw banking password into an AI chat history.
• 10:31
  No, never. Codex solves this with a built -in
• 10:34
  password manager. You import a secure CSV file.
• 10:37
  It has clean columns for name, URL, username,
• 10:41
  and password. So it pulls the credentials securely
• 10:44
  and needily. But what about two -factor authentication,
• 10:48
  the dreaded SMS text code? Yeah, the 2FA limit
• 10:51
  is incredibly real. And frankly, it still needs
• 10:54
  you in the loop. If the bank sends a code to
• 10:56
  your phone, you must enter it manually while
• 10:58
  the agent waits. Wait, you could technically
• 11:00
  write an automation to read text messages off
• 11:02
  a Mac and paste them in, couldn't you? People
• 11:04
  certainly try that. But you shouldn't. Banking
• 11:08
  workflow should always favor absolute control
• 11:10
  over pure convenience. You need strict stop rules
• 11:13
  in your prompts to prevent catastrophic errors.
• 11:15
  Something like only download statements. Do not
• 11:18
  click transfer. Do not move money. Stop immediately
• 11:21
  if the page layout looks different. Stop if a
• 11:23
  new 2FA prompt appears or if the account balance
• 11:26
  is entirely unclear. That is the only safe way
• 11:29
  to use an AI browser agent in finance. You do
• 11:32
  not want it hallucinating a wire transfer. You
• 11:34
  really need to watch it run 10 times in headed
• 11:36
  mode before trusting it with sensitive financial
• 11:39
  data. So why can't we just automate the 2FA text
• 11:43
  messages via a Mac to save time? Financial environments
• 11:46
  demand hard security breaks, really, to prevent
• 11:49
  catastrophic automated errors from occurring
• 11:52
  silently in the background. Right. Financial
• 11:54
  tasks need guardrails. Safety always beats total
• 11:57
  convenience. Completely. You cannot risk automated,
• 12:00
  unverified funds transfers. Speaking of seamless
• 12:04
  workflows and eliminating friction, we want to
• 12:06
  take a quick second to thank the sponsors of
• 12:08
  today's deep dive. Yes, absolutely. To sex silence.
• 12:13
  Our friends providing the workspace infrastructure
• 12:15
  that makes organizing all this data possible
• 12:17
  are truly essential. We appreciate their support
• 12:20
  in helping us navigate the future of work. OK,
• 12:24
  let's get back to these workflows. We've thoroughly
• 12:26
  covered extracting data from secure systems.
• 12:28
  Now we need to look at content creation. We've
• 12:31
  seen how strict the agent has to be in rigid,
• 12:33
  secure environments like a bank. But what happens
• 12:36
  when you point it at an inherently messy creative
• 12:38
  interface? We're talking about formatting a draft
• 12:41
  for social media. Specifically. the YouTube to
• 12:44
  X article workflow, turning a long -form video
• 12:48
  into a highly polished Twitter draft. Why does
• 12:50
  browser use WinHere over an API? I mean, X definitely
• 12:54
  has a developer API available. It does have one.
• 12:57
  But formatting rich text through the X API is
• 13:00
  notoriously messy. It strips out paragraph breaks.
• 13:04
  Media uploads can fail silently. Plus, API usage
• 13:08
  costs actual money. Right. Browser use runs right
• 13:11
  inside your local codec setup. It formats the
• 13:14
  text inside the ReelX Editor interface. It clicks
• 13:17
  and uploads to the actual browser window. The
• 13:19
  prompt you use here is very straightforward,
• 13:21
  but it's powerful. Turn this YouTube video into
• 13:24
  an X article. Add screenshots where they fit
• 13:26
  contextually, do not publish. That last part
• 13:28
  is absolutely crucial. You paste the YouTube
• 13:30
  link, Codex natively opens the video. It visually
• 13:33
  scans the frames. It captures highly useful screenshots.
• 13:36
  It generates a compelling thumbnail image. Then
• 13:38
  it physically opens X using your saved session
• 13:40
  state. It navigates to drafts, it clicks compose,
• 13:44
  paste the generated title, paste the full article
• 13:46
  body, and uploads the thumbnail. And it carefully
• 13:49
  moves images into the correct places within the
• 13:52
  text flow. A normal AI writing tool just spits
• 13:55
  out a massive wall of text that you have to copy.
• 13:58
  An AI browser agent puts that text into the actual
• 14:02
  publishing interface for you. It handles all
• 14:04
  the tedious visual steps around it. And the actual
• 14:07
  writing is only half the job anyway. The tedious
• 14:09
  formatting, the image resizing, the copy pasting,
• 14:12
  that's where your creative productivity actually
• 14:14
  dies. That's precisely why the key rule here
• 14:17
  is so important. Always force the AI to stop.
• 14:20
  before publishing. Yeah, let the agent handle
• 14:22
  the boring browser labor. You step in at the
• 14:24
  very end for human review. You check the images.
• 14:26
  You fix anything that sounds weird or robotic.
• 14:29
  Then you hit publish yourself. This brings up
• 14:31
  another incredibly powerful concept. Session
• 14:34
  persistence, what are commonly known as cookies.
• 14:37
  All right. In a fresh Codex chat window, X can
• 14:40
  still open already logged in because the agent
• 14:42
  remembers your session state. So you bypass the
• 14:45
  login wall entirely and you can just ask it to
• 14:48
  scan your feed. Yeah. You say, Open X, scroll
• 14:52
  my main feed and tell me what is new in the world
• 14:54
  of AI. Codex literally browses as you. It visually
• 14:57
  scans the feed. It summarizes useful trending
• 15:00
  updates. In one example, it noticed granular
• 15:03
  updates around Grokbot and Grok 4 .6. Exactly.
• 15:06
  And that easily becomes a scheduled daily routine.
• 15:09
  Every morning at 8 a .m., check my X feed for
• 15:12
  AI updates, summarize the best posts, and suggest
• 15:15
  three new article ideas. The pipeline gets incredibly
• 15:18
  powerful. It finds the core idea, turns a reference
• 15:21
  video into a draft, places the media perfectly,
• 15:24
  and prepares the post for your final review.
• 15:27
  Beat. How much human editing is typically needed
• 15:30
  after the agent formats a draft? Usually it's
• 15:33
  just a quick review, really, to ensure the tone
• 15:35
  and visual alignment are perfectly suited to
• 15:37
  your brand. Makes sense. Let the AI do the heavy
• 15:39
  lifting. You just add taste. Exactly. Curating
• 15:41
  taste remains a strictly human element. We've
• 15:44
  been entirely inside the web browser so far.
• 15:46
  But what happens when the workflow needs to step
• 15:48
  outside the browser and interact with your actual
• 15:50
  machine's operating system? This is the critical
• 15:53
  distinction between browser use and computer
• 15:55
  use. They work the exact same way at their core.
• 15:58
  It utilizes vision, AI reasoning, and direct
• 16:01
  control of the screen coordinates. The difference
• 16:04
  is simply where Codex is allowed to work. Right.
• 16:07
  Browser use is sandboxed for websites and web
• 16:09
  apps. Dashboards, portals, forms. Computer use
• 16:12
  is for your actual local machine. Desktop applications,
• 16:16
  local system files, in -app settings. You just
• 16:18
  go to the plugins menu, search for computer use,
• 16:21
  and install it. We have the Glido desktop application
• 16:23
  example to illustrate this. U .S. Codex. Open
• 16:26
  the Glido desktop app, go into settings, and
• 16:28
  turn on Bluetooth device support. It opens the
• 16:31
  local app. It visually scans the UI to find the
• 16:34
  settings gear. It switches the Bluetooth toggle
• 16:37
  on. It's interacting with your operating system.
• 16:39
  Whoa. Imagine scaling to a hundred unique edge
• 16:42
  case scenarios or having it actually tweak your
• 16:45
  computer's hidden network settings while you
• 16:46
  watch. That feels like a massive leap in how
• 16:48
  we command computers. It is a massive leap. Okay.
• 16:52
  And because it operates at the OS level, it requires
• 16:54
  explicit safety signals. When Codex has control
• 16:57
  of your actual computer, a thick blue border
• 17:00
  appears around your entire screen. So you always
• 17:03
  know exactly when the agent is driving the mouse.
• 17:06
  Right. It also halts automatically for admin
• 17:08
  passwords or sensitive system permissions. That
• 17:11
  visual safety signal is deeply reassuring when
• 17:13
  an AI has access to your files. We really need
• 17:16
  to recap the big idea here. the overarching decision
• 17:19
  framework for any AI agent automation workflow.
• 17:23
  Priority 1 is always the API or a direct plugin.
• 17:26
  It's structured data. It's the fastest, cheapest,
• 17:28
  and most stable path imaginable. Priority 2 is
• 17:31
  a macro script. This is best when every click
• 17:34
  is always the exact same. Same screen size, same
• 17:36
  buttons, same chronological order. Priority 3
• 17:39
  is browser or computer use. You use this when
• 17:41
  the task actually requires human -like vision
• 17:44
  and reasoning. When the interface is messy, updates
• 17:47
  frequently, or is highly unpredictable, do not
• 17:49
  deploy an AI vision agent just because it looks
• 17:51
  cool clicking around. Use it when the next logical
• 17:54
  step depends on what currently appears rendered
• 17:56
  on the screen. API for clean systems, macro for
• 18:00
  fixed, rigid steps, browser use for messy human
• 18:04
  interfaces. What's the ultimate payoff of this
• 18:08
  framework, actually, if an agent only saves you
• 18:11
  maybe 5 or 10 minutes a task? It eliminates the
• 18:13
  constant draining mental friction of switching
• 18:16
  your focus between entirely different operational
• 18:19
  contexts. Exactly. It is not about the minutes
• 18:21
  saved. It is about eliminating context switching.
• 18:24
  Context switching completely destroys deep creative
• 18:27
  focus. It takes 20 minutes to regain your flow
• 18:29
  state after doing a trivial admin task. An AI
• 18:32
  browser agent can open the banking portal, click
• 18:34
  through the dashboard, test the QA form, download
• 18:37
  the CSV file, and format the social media draft.
• 18:40
  It moves the tedious visual friction forward.
• 18:43
  And you stay entirely focused on solving the
• 18:45
  hard creative problems. One single codex task
• 18:48
  might only save a few physical minutes, but a
• 18:50
  scheduled library of these skills changes everything.
• 18:53
  You start with one annoying repetitive task you
• 18:55
  hate doing every Friday. Ideally one with no
• 18:58
  clean API. Yes. You describe it clearly in plain
• 19:01
  English. You watch codecs run it a few times
• 19:03
  in headed mode. You tighten the text instructions.
• 19:07
  You save it as a repeatable skill. Then you schedule
• 19:09
  it to run headless. That becomes your very first
• 19:12
  autonomous AI agent automation pipeline. It fundamentally
• 19:16
  changes how a modern workday runs. You move from
• 19:19
  physically doing the tedious work to managing
• 19:21
  the intelligent systems that do the work for
• 19:23
  you. Look at your daily routine tomorrow morning.
• 19:26
  How many clicks are you doing that rely purely
• 19:28
  on visual reasoning rather than actual human
• 19:30
  ingenuity? Which repetitive, messy UI task is
• 19:34
  going to be your first AI agent pipeline? Thank
• 19:37
  you for joining us on this deep dive. We'll catch
• 19:39
  you next time.