Latest / AI Fire Daily / #574 Neil: AI Agent Workflow Runs Parallel Research in Claude Code

Transcript

#574 Neil: AI Agent Workflow Runs Parallel Research in Claude Code · AI Fire Daily

• 0:00
  Most of us are stuck in a basic AI loop. Yeah,
• 0:03
  that endless hamster wheel. Exactly. You write
• 0:06
  a prompt, you get a bad result, and you just
• 0:08
  keep tweaking that same prompt. That is so frustrating.
• 0:12
  And notoriously slow. But there is a much more
• 0:14
  powerful way. Beat. Welcome to our deep dive
• 0:18
  for today. We are unpacking a workflow from Maestro's
• 0:22
  DayIA. They use Claude code and a tool called
• 0:25
  Ultracode. The mission here is running multiple
• 0:28
  AI agents all at once. Yeah, this workflow completely
• 0:32
  flips the standard script. We are moving way
• 0:35
  beyond basic prompt engineering today. Right.
• 0:37
  We're going to cover the foundational harness
• 0:39
  structure first. We'll look at these intense
• 0:41
  verification loops, and then we'll see true parallel
• 0:45
  processing in action. To break out of that simple
• 0:48
  prompt cycle, things must change. We have to
• 0:50
  change the environment the AI operates in. You
• 0:52
  can't just keep throwing tasks into a blank chat
• 0:54
  box. Right. Which brings us to a concept called
• 0:56
  the harness. Let's clearly define that technical
• 0:59
  term right now. A harness is a setup of rules
• 1:02
  and tools guiding an AI's specific job. That
• 1:05
  is a highly accurate way to frame it. You are
• 1:07
  giving the agent a very restricted workspace.
• 1:11
  You remove the chaos of an open -ended conversation
• 1:14
  completely. It is like hiring a brand new employee.
• 1:16
  You don't just yell a task at them in the hallway.
• 1:19
  Right, you'd never do that. You give them a full
• 1:21
  onboarding packet. You give them a dedicated
• 1:23
  desk and resources. Aaron Levy actually highlights
• 1:26
  this exact concept. Oh, yeah. He says an agent
• 1:29
  harness is crucial for complex work. Right. And
• 1:31
  without that structure, agents just get completely
• 1:34
  lost. They desperately need those rigid boundaries.
• 1:37
  Otherwise, the AI tries to be the researcher,
• 1:41
  writer, and editor simultaneously. I still wrestle
• 1:43
  with prompt drift myself. Yeah. Two -sec silence.
• 1:47
  I often start with a very clear goal. The AI
• 1:49
  gives me a decent draft, but I want one small
• 1:52
  change. I know where this is going. I ask it
• 1:55
  to adjust one specific paragraph. It fixes it,
• 1:58
  but somehow ruins all the formatting. Exactly.
• 2:00
  After three prompts, it has forgotten my very
• 2:02
  first instruction entirely. Breaking instructions
• 2:05
  into smaller, harnessed skills fixes this. And
• 2:09
  that focused environment leads right into the
• 2:11
  verification loop. The workflow does not just
• 2:13
  blindly accept the first answer. Right. Standard
• 2:16
  AI outputs text and just assumes it's finished.
• 2:19
  But this system adds a mandatory quality control
• 2:21
  checkpoint. Addy Osmani calls this loop engineering.
• 2:26
  OK, loop engineering. Yeah. Outputs are scored
• 2:28
  against very strict criteria. If the score is
• 2:31
  too low, the task simply runs again. So it actively
• 2:34
  refuses to move forward with bad data. Precisely.
• 2:37
  It keeps looping back until it passes the test.
• 2:40
  But wait, how does the workflow prevent an endless
• 2:42
  loop if the output keeps failing? It relies on
• 2:45
  highly specific checking criteria to eventually
• 2:48
  force a pass. The rubric asks binary yes or no
• 2:52
  questions to break the cycle. So strict grading
• 2:55
  rubrics force the agent to eventually hit the
• 2:57
  required target. Exactly. It creates a mathematically
• 3:00
  guaranteed baseline of quality. But this verification
• 3:02
  loop is only effective under certain conditions.
• 3:05
  The automated grader cannot be biased by the
• 3:08
  worker's messy process. This is exactly why fresh
• 3:10
  context is so incredibly important. The verifier
• 3:13
  agent is kept entirely separate from the worker
• 3:15
  agent. Right. It doesn't carry over any context
• 3:19
  from the initial work. It gets a completely wiped
• 3:22
  blank slate memory to work with. Two sec silence.
• 3:25
  That architectural separation makes a massive
• 3:28
  amount of sense. It is like having an independent
• 3:31
  auditor check a company's financials. Yeah. Perfect
• 3:34
  analogy. You don't ask the accountant who made
• 3:36
  the errors to double check their own math. You'd
• 3:38
  never trust that. Right. You bring in an external
• 3:41
  reviewer with a clean slate. Think about a quality
• 3:44
  control laser on an assembly line. The laser
• 3:47
  only scans the final car door for dents. It doesn't
• 3:51
  ask the welding robot how it feels about its
• 3:53
  job. Yeah, that makes sense. It objectively measures
• 3:55
  the physical product against the blueprints.
• 3:57
  This matches finder and verifier setups across
• 4:00
  all of AI. The verifier just objectively looks
• 4:03
  at the final output. The context window isn't
• 4:06
  cluttered with previous failed attempts. But
• 4:08
  this still raises an interesting architectural
• 4:10
  question. Why can't the original worker agent
• 4:13
  just be told to check its own work? Because the
• 4:16
  worker carries the heavy baggage of its past
• 4:18
  decisions, it naturally assumes its original
• 4:22
  internal logic is entirely correct. Because the
• 4:24
  worker's hidden assumptions would blind it to
• 4:26
  its own mistakes. Precisely. you desperately
• 4:29
  need a fresh pair of digital eyes. So we have
• 4:32
  this high quality verification system in place.
• 4:35
  Right. We have a worker and a fresh verifier
• 4:37
  checking it, but doing this sequentially is just
• 4:40
  far too slow. Oh, incredibly slow. Waiting for
• 4:42
  each step creates a massive operational bottleneck.
• 4:45
  We need to speed this complex process up significantly.
• 4:49
  This brings us to a concept called graph engineering.
• 4:52
  Yeah, this is the really exciting part. Let's
• 4:55
  define graph engineering clearly. It is splitting
• 4:58
  independent tasks into separate paths that run
• 5:00
  at the same time. This is where the Maestro's
• 5:02
  DIA workflow gets seriously impressive. Each
• 5:05
  independent task gets its very own branch. They
• 5:08
  don't have to wait in a slow line anymore. Let's
• 5:11
  look closely at the specific demo they ran. They
• 5:14
  used Claude code and Ultra code to build this
• 5:17
  structure. Just typing slash Ultra code and slash
• 5:21
  workflows into the command line. The primary
• 5:23
  mission was to research three different YouTube
• 5:26
  channels, OpenAI, Anthropic, and Google DeepMind.
• 5:30
  And they specifically wanted videos published
• 5:33
  in the last seven days. Think about doing this
• 5:35
  normally. You would wait for the OpenAI research
• 5:38
  to finish completely, then Anthropic would slowly
• 5:40
  take its turn. But here, three separate AI agents
• 5:44
  spin up and run simultaneously. Instead of waiting
• 5:47
  patiently, the system branches out immediately.
• 5:49
  It is like cooking a massive dinner for a large
• 5:51
  party. You don't wait for the chicken to finish
• 5:53
  before chopping carrots. Exactly. Akshay Pachar
• 5:56
  has actually demonstrated very similar subagent
• 5:59
  splits recently. Paul Huron also shows how context
• 6:02
  is isolated to its specific task. That isolation
• 6:05
  is the absolute magic of true parallel execution.
• 6:09
  And the strict verification loop still happens
• 6:11
  inside each branch. OK, so the verifier agents
• 6:14
  review outputs without talking to the other branches.
• 6:17
  Yeah, entirely separately. Once all branches
• 6:20
  are finished and verified, things finally merge.
• 6:23
  Claude Code brings everything into one final
• 6:25
  comparative PDF report, beat. But managing all
• 6:29
  these streams seems precarious. Does running
• 6:32
  these branches at the same time mix up their
• 6:34
  data? Not at all. The underlying system walls
• 6:38
  off the context for each branch entirely. They
• 6:40
  don't share any memory until the final merge.
• 6:43
  Got it. Each branch keeps its context strictly
• 6:45
  isolated until the final combination. That strict
• 6:48
  isolation makes the parallel processing so incredibly
• 6:50
  reliable. You don't get agents hallucinating
• 6:53
  details across different companies. We understand
• 6:55
  the theory behind this complex structure now.
• 6:57
  Yeah. But what exactly happened when they hit
• 7:00
  enter on this workflow? What did the real world
• 7:03
  execution actually look like? The initial demo
• 7:05
  results are honestly pretty mind blowing. It
• 7:08
  was incredibly fast. How fast? Each research
• 7:10
  session took only about 40 seconds to finish
• 7:12
  and used roughly 54 ,000 tokens to process the
• 7:16
  information. That is a truly massive amount of
• 7:18
  raw data. Processing 54 ,000 tokens is roughly
• 7:22
  equivalent to reading a short novel. And the
• 7:25
  system processed that in under a minute. Whoa,
• 7:28
  imagine scaling to a billion queries. The sheer
• 7:31
  volume of automated processing is completely
• 7:34
  staggering. Two -sec silence. Let's talk about
• 7:37
  the verification mechanics in action. What exactly
• 7:40
  did these independent verifier agents catch?
• 7:43
  They caught some very critical timeline errors
• 7:45
  immediately. None of the five initial videos
• 7:48
  were actually published recently. We know them.
• 7:50
  Nope. They fell way outside the strict seven
• 7:52
  -day window. So the initial worker agents just
• 7:55
  completely hallucinated the publication dates.
• 7:58
  Exactly. Language models are notoriously bad
• 8:00
  at strictly tracking chronological time, but
• 8:03
  the fresh verifiers caught that instantly. They
• 8:05
  also flagged a highly specific Google DeepMind
• 8:08
  video. It could not be confirmed on the official
• 8:10
  channel at all. So the verification loop actively
• 8:13
  stopped bad data, but they also confirmed plenty
• 8:15
  of good data during this. Oh, absolutely. The
• 8:18
  verifications confirmed six videos with zero
• 8:21
  discrepancies whatsoever. It really showcases
• 8:23
  the rigorous nature of this setup. They validate
• 8:26
  the good data while catching the subtle errors.
• 8:29
  all clearly displayed in the final PDF. There
• 8:32
  is almost always a hidden catch with these powerful
• 8:35
  systems. I have to note the very real operational
• 8:38
  trade -off here. Yeah, the costs. Adding separate
• 8:40
  workers and verifiers burns through tokens incredibly
• 8:43
  fast. You are essentially paying for three or
• 8:46
  four API calls instead of one. It is definitively
• 8:49
  a much more expensive way to operate. For an
• 8:51
  enterprise company, that token cost is a rounding
• 8:54
  error. But for an individual developer, it really
• 8:57
  adds up. Which brings up a very practical concern.
• 9:00
  With such high token usage, is this setup actually
• 9:03
  practical for everyday tasks? It is strictly
• 9:06
  for large tasks where accuracy is completely
• 9:09
  non -negotiable. You wouldn't use this heavy
• 9:11
  framework to draft a casual email. More tokens
• 9:14
  mean higher costs, but you get guaranteed quality
• 9:17
  in return. That is the ultimate trade -off you
• 9:20
  have to weigh. You are buying structural peace
• 9:22
  of mind. Let's step back for a moment and summarize
• 9:24
  the core lesson. Let's recap. An AI workflow
• 9:27
  is vastly stronger when each agent has a clear
• 9:30
  role. Tasks run in parallel branches, and outputs
• 9:34
  are checked by an independent verifier with fresh
• 9:36
  context. It totally redefines how we should interact
• 9:39
  with language models. We're moving from being
• 9:42
  prompt writers to actual system architects. But
• 9:46
  understanding this does leave me with a slightly
• 9:48
  provocative thought. If we are building systems
• 9:51
  where AI independently verifies other AI, what
• 9:56
  happens when these agents begin writing their
• 9:58
  own strict grading criteria? Who verifies the
• 10:01
  verifier when the entire system becomes fully
• 10:03
  autonomous? That is a deeply fascinating problem
• 10:06
  for the very near future. Thank you for taking
• 10:09
  this deep dive with us today. We highly encourage
• 10:11
  you to experiment with workflow branching yourself.
• 10:13
  Stop tweaking the exact same basic prompt over
• 10:16
  and over, build a strict harness, add a clean
• 10:18
  verifier, and run it all in parallel. Out to
• 10:20
  your own music.