Latest / AI Fire Daily / #638 Neil: Claude Opus 5.5 Extra Wins The Full Same App Effort Test

Transcript

#638 Neil: Claude Opus 5.5 Extra Wins The Full Same App Effort Test · AI Fire Daily

• 0:00
  If you slide the AI effort file all the way to
• 0:02
  maximum, does it actually build better software?
• 0:06
  Or, you know, just completely waste your time?
• 0:08
  Yeah, I mean, that is the ultimate productivity
• 0:10
  question right now. We kind of just assume more
• 0:14
  computing power automatically equals a better
• 0:16
  product. But that equation is definitely starting
• 0:19
  to break down. It really is. Welcome to a new
• 0:22
  Deep Dive. Today we're unpacking a truly fascinating
• 0:24
  developer experiment. We're going to test that
• 0:26
  exact question. Exactly. We're looking at a highly
• 0:29
  structured test of Claude Opus 5 .5. Specifically,
• 0:33
  we're testing it inside Claude Code. Right. And
• 0:36
  just to be clear on terms for you listening,
• 0:38
  Claude Code is an AI tool that writes and tests
• 0:41
  software directly on your machine. Yep. It operates
• 0:43
  right there in your local environment. So we're
• 0:46
  looking at an experiment with this brilliant
• 0:48
  setup. The exact same prompt was run across six
• 0:51
  different effort levels. We're talking about
• 0:53
  low, medium, high, extra, max, and ultra code
• 0:57
  settings. We basically want to see what actually
• 1:00
  happens to app quality when you tweak those settings.
• 1:03
  We're tracking runtime, logic, and overall design.
• 1:06
  But before we even look at the results, we absolutely
• 1:09
  have to understand the boundaries here. We need
• 1:12
  to set the stage for this experiment. Well, yeah,
• 1:14
  because if you don't lock down the environment,
• 1:16
  the results mean absolutely nothing. You have
• 1:18
  to understand the specific mission first. Right.
• 1:21
  So the goal was incredibly specific. The AI had
• 1:24
  to build an AI mission control dashboard. It
• 1:27
  was designed to manage multiple AI agents, right?
• 1:30
  It needed to track their status, live activity,
• 1:33
  and complex analytics. Which tests much more
• 1:35
  than just a pretty visual design. The AI had
• 1:38
  to handle real application structure. It had
• 1:41
  to manage state across the app, and state management
• 1:44
  is just keeping track of app data as it changes
• 1:46
  over time. Exactly. And it had to deliver a really
• 1:50
  high level of product polish. The constants in
• 1:52
  this experiment were completely rigid. Every
• 1:54
  single run started from scratch with a clean
• 1:57
  project folder. It was forced to use the exact
• 1:59
  same brand colors. It used identical sample data
• 2:02
  for every single run. And it used a painfully
• 2:06
  specific prompt. I mean, it's basically like
• 2:08
  giving six different master chefs the exact same
• 2:11
  ingredients and recipe. And just changing how
• 2:13
  many hours they're allowed to obsess over the
• 2:15
  plating. That's a great way to think about it.
• 2:17
  The prompt requirements were really fascinating.
• 2:21
  The AI was told to build a living, breathing
• 2:23
  product. It couldn't just build a static, lifeless
• 2:26
  prototype. It was instructed to actively avoid
• 2:29
  obvious dead ends in the code. It had to include
• 2:31
  distinct system states. It needed to show healthy,
• 2:34
  busy, or warning states for the agents. Yeah,
• 2:37
  and it had to verify its own work. It had to
• 2:40
  actually check the console for errors and fix
• 2:42
  them before stopping. The constraints basically
• 2:45
  force the AI to rely entirely on its reasoning
• 2:48
  capabilities. It cannot cheat the test. So let's
• 2:51
  pause and unpack that. Why is setting such a
• 2:53
  strict, rigid boundary so absolutely critical
• 2:56
  for evaluating this AI? Well, without rigid constraints,
• 3:00
  an LLM will naturally shift the goalposts. It
• 3:03
  looks for the easiest path. You wouldn't know
• 3:06
  if a better result came from higher effort, or
• 3:08
  if the AI just hallucinated a simpler, lucky
• 3:11
  path. Strict rules isolate the effort variable
• 3:14
  completely for accurate testing. Spot on. So
• 3:18
  now that the kitchen is prepped, let's look at
• 3:20
  the baseline. Let's see what happens when the
• 3:22
  AI is told to work fast. We begin with the low
• 3:26
  effort setting. This initial run took about 12
• 3:29
  minutes to complete. Which is remarkably fast
• 3:32
  for a full interface. It was much faster than
• 3:34
  anyone expected for an app with multiple screens.
• 3:36
  It built a surprisingly solid foundation in that
• 3:39
  time, actually. It included a functional workspace
• 3:42
  overview. Right. It had detailed agent views,
• 3:46
  working search functions, and active filtering.
• 3:48
  The design was what you might call safe. It looked
• 3:52
  like a standard web template. It didn't have
• 3:54
  many distinctive stylistic touches, but it was
• 3:57
  highly functional, and it worked immediately.
• 3:59
  The overview tracked active agents and tasks
• 4:02
  in progress. It successfully showed block tasks.
• 4:05
  It even generated a seven -day throughput chart.
• 4:08
  The underlying data felt truly connected across
• 4:11
  the whole interface. Then we step up to the medium
• 4:14
  effort setting. This run took a little over 30
• 4:17
  minutes. So 12 minutes gets us a functional but
• 4:20
  boring baseline. The 30 -minute run is almost
• 4:23
  three times slower. Yeah, and you really start
• 4:26
  to need the extra time to show up in the structure.
• 4:28
  The overview felt much more deliberately organized.
• 4:31
  It felt significantly more operational. Urgent
• 4:34
  items were grouped much more clearly for the
• 4:37
  user. System health visibility was much stronger.
• 4:40
  You could scan agent workloads easily at a single
• 4:43
  glance. The task priority stood out beautifully.
• 4:46
  And the filters didn't crowd the page at all.
• 4:48
  It basically looked like a product that went
• 4:50
  through a careful second round of refinement.
• 4:52
  It was no longer just a solid first pass. I have
• 4:55
  to offer a vulnerable admission here. I still
• 4:58
  wrestle with prompt FOMO myself. Oh yeah, that's
• 5:01
  a very real thing for developers. I constantly
• 5:04
  fear I'm missing out on a better answer. I always
• 5:06
  assume I need maximum settings to get anything
• 5:08
  good. It feels risky to turn the dial down. It's
• 5:11
  a classic psychological trap. We just assume
• 5:14
  more inference time inherently guarantees a better
• 5:16
  foundation. And inference time is just the time
• 5:19
  an AI takes to process and generate answers.
• 5:22
  Right. But the jump from 12 to 30 minutes is
• 5:25
  a massive investment of time. Which brings up
• 5:27
  a crucial question. Why does a nearly three times
• 5:30
  time increase yield such diminishing returns
• 5:32
  at this stage? Well, the AI solves the core logical
• 5:36
  puzzle very quickly. The vast majority of that
• 5:38
  extra time is spent iterating on minor visual
• 5:41
  tweaks. It obsesses over padding and colors that
• 5:44
  don't fundamentally change the core functionality.
• 5:47
  It wastes compute time polishing pixels instead
• 5:49
  of building better logic. Exactly. It's painting
• 5:52
  a house that doesn't have a roof yet. So if 30
• 5:55
  minutes gets us a refined product, doubling the
• 5:57
  time should create a masterpiece, right? You
• 6:00
  would certainly think so. But the physics of
• 6:02
  AI efforts start to get really weird right here.
• 6:05
  Let's look at the high effort run. This attempt
• 6:07
  took about 55 minutes. The resulting dashboard
• 6:11
  felt much tighter. There was a massive improvement
• 6:14
  in the product's actual depth. We specifically
• 6:17
  saw this in the agent detail panel. Let's take
• 6:19
  a hypothetical agent named Kite, for example.
• 6:22
  In the high run, the AI pulled Kite's success
• 6:25
  rate into one single view. It pulled in the current
• 6:28
  workload and the historical error count. It centralized
• 6:31
  all that disparate data perfectly. It requires
• 6:34
  a lot of context gathering to do that. It felt
• 6:37
  deeply thought through. But 55 minutes is a massive
• 6:41
  workflow breaking time jump. Then we move to
• 6:44
  the extra effort setting. Yeah, and this is where
• 6:47
  we hit a massive paradox in the data. The extra
• 6:50
  effort run took about 45 minutes. Wait, it took
• 6:53
  10 minutes less time than the high effort run.
• 6:56
  That is completely counterintuitive. It's totally
• 6:59
  backward. But the result was arguably much better.
• 7:02
  It focused heavily on deep user interactions.
• 7:06
  You could seamlessly edit task status, priority,
• 7:08
  and due dates from one single place. It also
• 7:11
  added brilliant dynamic loading states. It built
• 7:14
  skeleton placeholders instead of just leaving
• 7:16
  blank screens while data loaded. Right. And skeleton
• 7:19
  placeholders are just those gray boxes that blink
• 7:21
  before real data actually loads. It shows the
• 7:24
  AI understands user experience over time, not
• 7:27
  just static pixels. Once the board loaded, it
• 7:30
  stayed beautifully clean. It was incredibly easy
• 7:32
  to scan. It felt much stronger in the small details
• 7:36
  that actually mattered during daily use. Well,
• 7:38
  I mean, imagine the AI dynamically deciding it
• 7:41
  can optimize its own workflow. It maps the architecture
• 7:44
  so well it finishes a superior build 10 minutes
• 7:46
  faster. It's a profound shift in how we think
• 7:49
  about compute allocation. It prioritized fluid
• 7:53
  user interaction over generating massive visual
• 7:55
  density. I need to understand the mechanics here.
• 7:58
  How can asking an AI for higher effort result
• 8:01
  in a faster execution time? At higher effort
• 8:05
  levels, the LLM maps out a highly efficient initial
• 8:08
  architecture before writing. By thinking deeper
• 8:10
  at the absolute start, it completely avoids coding
• 8:13
  dead ends that require time -consuming rewrites
• 8:16
  later. Better upfront planning prevents costly
• 8:19
  time -consuming code rewrites later on. Precisely.
• 8:22
  It measures twice and cuts once. We're going
• 8:25
  to take a quick break, but do not go anywhere.
• 8:28
  We've got a lot more to cover on the other side.
• 8:32
  And we are back. OK, so if the extra setting
• 8:35
  broke the time curve entirely, what happens next?
• 8:39
  We push it to the absolute limit. Do the absolute
• 8:42
  maximum settings perfect the app? Or do they
• 8:46
  just wildly overcomplicate it? Let's examine
• 8:48
  the max effort run. This was the longest run
• 8:51
  of the entire test. It took about one hour and
• 8:54
  five minutes. It delivered a deeply intensely
• 8:57
  refined app. It added complex trend lines on
• 9:00
  success rates. It featured incredibly rich, detailed
• 9:04
  activity tracking for every single agent. The
• 9:07
  workspace overview was definitely the clearest
• 9:09
  version yet. You could instantly see system health
• 9:12
  and blocked work without digging through pages.
• 9:14
  You could filter the full event history by progress
• 9:16
  or by specific retries. The sheer context it
• 9:21
  provided was incredible, but it took over an
• 9:25
  hour to generate. That is a significant, painful
• 9:28
  chunk of time for a single iteration. Next, we
• 9:31
  look at the final setting, which is Ultracode.
• 9:35
  Ultracode took about 50 minutes. So it was magically
• 9:38
  faster than Max, again. beautifully balanced
• 9:42
  quality across the whole application. It didn't
• 9:45
  obsessively over -polish just one single screen.
• 9:48
  It created highly specific analytics charts.
• 9:51
  These charts actually answered clear, logical
• 9:54
  product questions. They meticulously tracked
• 9:57
  daily output and long -term error trends. It
• 10:00
  was incredibly useful. But crucially, it didn't
• 10:03
  create a massive gap in quality over extra or
• 10:06
  max. Right. The visual and functional improvements
• 10:08
  hit a very hard ceiling there. I have to push
• 10:11
  back on the usefulness of these maximum settings.
• 10:13
  Waiting over an hour for Macs completely ruins
• 10:16
  the iterative conversational flow of coding.
• 10:19
  Oh, I agree completely. The developer experience
• 10:22
  totally falls apart at that point. If you're
• 10:24
  waiting 65 minutes for a code response, you completely
• 10:27
  lose your own context. You step away from the
• 10:29
  keyboard to get coffee. You check your email.
• 10:31
  You look at your phone. You forget the entire
• 10:34
  architecture you were holding in your head. It
• 10:36
  defeats the entire purpose of having an AI coding
• 10:38
  assistant. You want rapid feedback. You need
• 10:40
  to maintain momentum. When the runtime stretches
• 10:43
  past 45 minutes, you're essentially batch processing
• 10:46
  your development. You're treating the AI like
• 10:48
  a slow compiler. That is a very different, very
• 10:51
  frustrating workflow. It really is. So let's
• 10:54
  establish a rule of thumb. When does AI compute
• 10:57
  time actually stop being worth the wait for a
• 10:59
  developer? Basically, when the wait time breaks
• 11:02
  your own flow state. If you lose your train of
• 11:05
  thought, the slight marginal code improvement
• 11:07
  isn't worth the massive cognitive disruption
• 11:09
  it causes. When the long wait breaks your own
• 11:12
  cognitive coding momentum. Exactly. Keep your
• 11:16
  momentum going. Stepping back from all these
• 11:18
  individual benchmarks, we need to declare a definitive
• 11:21
  winner here. We need to look at the grand unified
• 11:24
  theory of AI effort levels. The core lesson from
• 11:27
  this data is glaringly clear. More effort does
• 11:31
  not scale neatly with runtime. It also does not
• 11:34
  scale neatly with overall software quality. It's
• 11:37
  not a straight line up and to the right. The
• 11:40
  undisputed winner of this entire experiment is
• 11:42
  the extra effort setting. It gave the absolute
• 11:45
  best balance across the board. It perfectly balanced
• 11:48
  quality, deep usability, and reasonable runtime.
• 11:52
  It clocked in at exactly 45 minutes and it delivered
• 11:55
  incredible value. It had those strong user interactions.
• 11:59
  It had those brilliant dynamic loading states
• 12:01
  we talked about. It felt like a serious working
• 12:05
  version of a product, and it did it without wasting
• 12:07
  your entire afternoon. The core takeaway here
• 12:10
  is highly actionable for anyone building software
• 12:13
  today. Do not automatically jump to the highest
• 12:16
  setting for serious coding. Start your projects
• 12:19
  around the extra setting. Check the structural
• 12:22
  result first. Only move up to max or ultracode
• 12:24
  if the project specifically demands much more
• 12:27
  depth in one area. It requires a fundamental
• 12:30
  shifting user behavior. We're just heavily conditioned
• 12:33
  to want the maximum version of everything. It's
• 12:35
  kind of like stacking Lego blocks of data. Giving
• 12:38
  the AI more time doesn't necessarily mean it
• 12:40
  builds a cooler, sleeker spaceship. Sometimes
• 12:43
  it just dumps more blocks on the table. That
• 12:45
  is exactly it. You do not need every single block
• 12:49
  on the table to build a fast ship. You just need
• 12:51
  the right blocks placed carefully. The maximum
• 12:54
  setting just gives you a heavier ship. Based
• 12:56
  on all this data, how should this fundamentally
• 12:59
  change our everyday interactions with AI tools
• 13:01
  moving forward? It proves we must treat AI effort
• 13:04
  levels like a throttle, not an on switch. You
• 13:07
  start in the middle, verify your direction, and
• 13:10
  only accelerate the compute when absolutely necessary.
• 13:13
  Treat AI effort levels like a throttle, not an
• 13:16
  on switch. Precisely. It is entirely about maintaining
• 13:19
  control. That brings us to the end of this deep
• 13:22
  dive. It has been a truly fascinating look at
• 13:24
  how we allocate machine thinking. The data really
• 13:27
  speaks for itself here. The basic assumptions
• 13:29
  we have about effort and reward are fundamentally
• 13:32
  flawed when it comes to AI generation. Take a
• 13:35
  moment today to check your own default settings.
• 13:37
  Think about your own workflow. When you work
• 13:39
  with AI tools, are you automatically cranking
• 13:41
  the dial to max just out of habit? You might
• 13:44
  be wasting your own valuable time. You might
• 13:47
  be getting a final product that is denser. but
• 13:49
  not actually better. You might just be asking
• 13:51
  for more Lego blocks. Let me leave you with this
• 13:54
  final provocative thought. If maximum machine
• 13:58
  effort doesn't yield maximum results, our jobs
• 14:01
  are changing. How so? Well, maybe the most valuable
• 14:05
  human skill of the future isn't writing the code
• 14:07
  yourself. Maybe it's knowing exactly how much
• 14:09
  effort to ask the machine to expend in the first
• 14:11
  place. That changes the whole paradigm of software
• 14:14
  engineering. It really does. Thank you for exploring
• 14:17
  this with us. We'll catch you on the next deep
• 14:19
  dive.