Latest / AI Fire Daily / #615 Neil: GPT 6 Astra Looks Powerful. But Can It Handle Real Work?

Transcript

#615 Neil: GPT 6 Astra Looks Powerful. But Can It Handle Real Work? · AI Fire Daily

• 0:00
  So September 1st arrives, Claude Fabel 5 .1 drops,
• 0:05
  and the entire tech world is just suddenly busy
• 0:07
  testing it. Everyone wants to see what it can
• 0:09
  actually do. Right. The hype was massive. And
• 0:11
  OpenAI just stays completely quiet for exactly
• 0:15
  two days. Then September 3rd hits with a massive
• 0:18
  shock. They drop GPT -6 Astra with an absolutely
• 0:23
  audacious claim. They say it can handle almost
• 0:25
  anything you normally do on a computer and do
• 0:29
  it fast. Welcome to the deep dive. I am really
• 0:31
  glad you were here with us today. So glad to
• 0:33
  be here. Today we are taking a measured curious
• 0:35
  look at Astra. We are going to explore the real
• 0:38
  world performance benchmarks. We will look closely
• 0:41
  at that hefty API pricing and we will examine
• 0:43
  its startling computer use capabilities. Which
• 0:46
  are just wild to see in action. Truly. Our mission
• 0:49
  here is actually quite simple today. We need
• 0:50
  to figure out if this is just another model upgrade.
• 0:52
  or a fundamental shift in how we interact with
• 0:54
  computers. The sheer scale of their ambition
• 0:56
  is genuinely fascinating. We are officially moving
• 0:59
  past the simple chatbot era. We are entering
• 1:02
  what they call the agentic era. Let us unpack
• 1:04
  that specific term for a moment. Yeah, let's
• 1:06
  do that. Because OpenAI is positioning Astra
• 1:09
  differently from all previous models. They say
• 1:12
  it handles complex tasks entirely on its own.
• 1:14
  Right. It works seamlessly across different desktop
• 1:17
  web browsers. It navigates live websites and
• 1:20
  uses everyday desktop applications. It represents
• 1:23
  a massive leap in functional capability. Traditional
• 1:26
  chatbots are completely reactive by their very
• 1:28
  nature. Exactly. You type a specific prompt and
• 1:32
  they generate text. Then they simply stop and
• 1:35
  wait for your next command. But Astra operates
• 1:38
  using a continuous, proactive feedback loop.
• 1:41
  The core feature they are touting is agentic
• 1:43
  work. So let me define that AI jargon really
• 1:47
  quickly. It completes many steps on its own without
• 1:50
  needing constant instructions. That is exactly
• 1:52
  it. You give the model a high -level goal to
• 1:55
  achieve. It figures out all the necessary intermediate
• 1:57
  steps internally. It creates a plan, executes
• 2:01
  it, and checks the results. If a step fails,
• 2:03
  it adjusts the plan automatically. Using older
• 2:06
  models was like giving someone turn -by -turn
• 2:08
  directions. Astra's like slacking over and letting
• 2:11
  the AI take the steering wheel. I love that.
• 2:13
  That analogy perfectly captures the massive shift
• 2:15
  in control. But handing over the steering wheel
• 2:18
  naturally introduces new risks. Which brings
• 2:20
  us directly to their very cautious staged rollout.
• 2:24
  OpenAI is strictly limiting user access right
• 2:26
  now. Yeah, they are being super careful. They
• 2:28
  are rolling it out slowly in specific tiers.
• 2:31
  Plus... Pro and business users all get different
• 2:35
  access levels. With much more autonomy comes
• 2:38
  a significantly larger risk. Right. You cannot
• 2:40
  just unleash an autonomous agent on a billion
• 2:43
  computers, not on day one, without extensive
• 2:46
  real -world safety testing. Does this staged
• 2:48
  rollout mean they are genuinely afraid of what
• 2:52
  autonomous agents might break? Well, system unpredictability
• 2:55
  spikes drastically when an AI can click buttons.
• 2:58
  Think about what a fully autonomous agent can
• 3:00
  actually do. It can modify local files and send
• 3:02
  real corporate emails. It could accidentally
• 3:04
  delete a crucial production database. They really
• 3:06
  need to monitor how it behaves in the wild. So
• 3:09
  it is about containing the blast radius while
• 3:11
  the AI learns. They absolutely need to see how
• 3:14
  it handles strange edge cases. They have to ensure
• 3:17
  it does not execute destructive actions. If we
• 3:21
  are handing over the steering wheel to Astra,
• 3:24
  we need to look closely at its driving test scores.
• 3:27
  We have to see if it is actually safe. Right,
• 3:29
  the benchmarks. Let us dive into these new...
• 3:32
  performance benchmarks. These new benchmark scores
• 3:35
  are frankly mind -blowing to read. Let us start
• 3:37
  with the ARC -AGI -3 test. This is the specific
• 3:41
  number that really caught my attention. GPT -6
• 3:43
  Astra scores an incredible 99 .9%. Which is an
• 3:47
  absolutely absurd jump in technical capability.
• 3:50
  I mean, Claude Opus 5 reached 30 .2 % on this.
• 3:53
  And GPT -5 .6L only managed a dismal 7 .8%. Wow.
• 3:59
  Moving from 7 % to nearly 100%. that represents
• 4:02
  a complete paradigm shift in reasoning capability.
• 4:05
  But we should explain what this test actually
• 4:07
  measures. The ARC test evaluates how well a model
• 4:10
  finds patterns. It tests how well it solves entirely
• 4:13
  new logic problems. These are visual puzzles
• 4:16
  the AI has never seen before. It requires the
• 4:19
  AI to deduce the underlying rules dynamically.
• 4:22
  A score of 7 .8 % means older models were basically
• 4:25
  guessing. Hitting that near -perfect score suggests
• 4:29
  aster reasons on a different level. It figures
• 4:31
  out the actual logic instead of just memorizing
• 4:34
  trading data. We see a very similar pattern with
• 4:37
  Frontier Math Tier 4. In the stronger settings,
• 4:40
  Astra hits around 98%. While GP 5 .6 sole peaked
• 4:45
  at about 83%, Claude Fabel 5 can reach around
• 4:48
  90 % accuracy. But there is a major catch regarding
• 4:51
  the competition here. Astra achieves these incredible
• 4:54
  scores while keeping operational costs much lower.
• 4:57
  It beats Claude Fabel while requiring significantly
• 4:59
  less API costs. It is huge. But let us shift
• 5:03
  our focus over to software coding. Yeah, let
• 5:05
  us talk about the deepest WE version 1 .1. Astra
• 5:07
  reaches around 74 % at its highest setting. The
• 5:10
  performance gap is a little bit smaller right
• 5:12
  here. GPT 5 .6 Sol and Claude Fabel get fairly
• 5:15
  close to Astra. But the DeepSWE benchmark is
• 5:19
  absolutely crucial for modern developers. How
• 5:21
  is this test different from abstract math or
• 5:24
  logic puzzles? Because it is much closer to real
• 5:27
  software engineering work. It is not just answering
• 5:30
  a short, isolated coding question. Right. Real
• 5:33
  code bases are messy webs of interconnected digital
• 5:36
  files. Yeah. The AI has to thoroughly understand
• 5:39
  a complex technical task. It has to change code
• 5:42
  across a large digital architecture. Older models
• 5:45
  suffered from a sort of digital amnesia. If they
• 5:48
  fixed line 10, they forgot about line 500. Exactly.
• 5:52
  Astra has to maintain context across a very long
• 5:55
  workflow. It literally holds the entire system
• 5:57
  architecture in its digital mind. It anticipates
• 6:00
  the cascading domino effect of its own code changes.
• 6:03
  And then we have the Terminal Bench 4 .0 results.
• 6:05
  Astra hits an impressive 58 % accuracy on that
• 6:08
  test. GPT 5 .6. Sol sat way back at 37%. Yeah.
• 6:13
  Claude Fabel reaches... about 56%, but at a higher
• 6:15
  cost. The gap widens significantly on Terminal
• 6:18
  Bench Science .1. Astra reaches about a 64 %
• 6:21
  resolution rate there. Claude tops out at roughly
• 6:23
  53 % on that test. Why do these specific terminal
• 6:26
  tests actually matter for us? Well, they test
• 6:29
  system resilience in a very messy digital environment.
• 6:32
  The command line terminal is where real developer
• 6:35
  work happens. Can the AI successfully work through
• 6:38
  a complex terminal interface? Can it handle several
• 6:41
  disconnected steps without losing its focus?
• 6:44
  And crucially, can it keep trying when a command
• 6:47
  ultimately fails? Whoa. Imagine it just silently
• 6:50
  fixing its own terminal errors in the background
• 6:52
  until the code actually runs. It completely changes
• 6:55
  the traditional debugging process for human engineers.
• 6:58
  Terminal errors are often incredibly cryptic
• 6:59
  and frustrating to solve. Tell me about it. Astra
• 7:02
  just reads the error log, adjusts the syntax,
• 7:04
  and recompiles. You do not have to hold its hand
• 7:06
  every single minute. Do these terminal resilience
• 7:09
  tests actually mirror how a human developer solves
• 7:12
  a broken script? Absolutely. Humans try a command,
• 7:16
  read the error, and adjust the syntax. Then they
• 7:18
  try again until the software finally compiles
• 7:20
  successfully. Astra executes this exact iterative
• 7:24
  trial and error loop completely independently.
• 7:26
  Basically, it mimics our own human trial and
• 7:29
  error process just much faster. Yeah, exactly.
• 7:32
  And that brings us directly over to the automation
• 7:34
  bench test. Astra reaches more than 40 % accuracy
• 7:37
  on those workflows. While GPT 5 .6 Sol unfortunately
• 7:41
  stays below 20%, the Claude model score much
• 7:45
  lower on these specific tasks too. This benchmark
• 7:48
  connects closely with how Astra is actually meant
• 7:50
  to work. It tests complex workflows that require
• 7:53
  multiple, deeply connected steps. It proves Astra
• 7:56
  handles harder reasoning over sustained periods.
• 7:59
  It can maintain attention during incredibly long
• 8:02
  coding tasks. Two secs silence. That kind of
• 8:05
  relentless trial and error requires serious computing
• 8:08
  power. So what is this digital chauffeur actually
• 8:10
  costing us? Let me tell you, this new model is
• 8:12
  certainly not cheap. No. If you look purely at
• 8:15
  token price, Astra is expensive. Let us break
• 8:18
  down the exact API pricing structure here. GBT
• 8:21
  -6 Astra. is $10 per 1 million input tokens,
• 8:24
  and it costs $50 per 1 million output tokens.
• 8:27
  Compare that directly to the older GPT -5 .6
• 8:30
  SAW model. That was only $4 for the input token
• 8:33
  batch, and it was just $20 for the generated
• 8:35
  output tokens. Claude Fabel matches Astra's token
• 8:39
  price exactly across the board. $10 in and a
• 8:43
  massive $50 out. $50 for a million output tokens
• 8:47
  is objectively steep. If I am an independent
• 8:50
  developer, my API budget will evaporate. How
• 8:54
  is anyone justifying this enormous operational
• 8:57
  cost? You have to consider the entirely new concept
• 9:01
  of cost per task. You cannot just look at the
• 9:03
  raw token price anymore. Okay, explain how cost
• 9:05
  per task actually changes the math here. Think
• 9:08
  about how you normally use a standard, cheaper
• 9:10
  AI model. You write a prompt, the model fails,
• 9:13
  and you correct it repeatedly. You are paying
• 9:15
  for every single one of those failed attempts.
• 9:18
  Right. But if Astra finishes a job correctly
• 9:20
  on the very first try, you use far fewer correction
• 9:23
  prompts and waste fewer input tokens. So the
• 9:25
  $50 output cost might actually save you money
• 9:28
  because you are not paying for 10 frustrating
• 9:30
  failed attempts. It is entirely about overall
• 9:33
  workflow efficiency and saving human time. But
• 9:35
  we also need to talk about the strict usage limits.
• 9:38
  Pro users currently get the full existing API
• 9:40
  allowance. That allowance sits right between
• 9:43
  $100 and $200. Plus, users get limited Astra
• 9:46
  usage inside work and codex environments. Business
• 9:49
  standard is heavily limited, while business premium
• 9:51
  gets the full allowance. Here is where the situation
• 9:54
  gets really interesting for developers. Long
• 9:57
  agent tasks burn through those financial allowances
• 10:01
  incredibly fast. Because Astra fundamentally
• 10:04
  reasons out loud during its entire process, it
• 10:07
  calls multiple steps and produces much larger
• 10:09
  data outputs. It generates way more output tokens
• 10:12
  than a standard chat model. I still wrestle with
• 10:15
  micromanaging my prompts myself. So letting an
• 10:17
  expensive model run free makes me sweat a little.
• 10:20
  It is completely nerve wracking to watch your
• 10:22
  budget drain in real time. You see those API
• 10:25
  dollars disappearing while the AI sits and thinks.
• 10:28
  At $50 a million output tokens, where is the
• 10:30
  breaking point for a regular user? You absolutely
• 10:33
  must stick to high value complex workflows. If
• 10:36
  the task saves you three hours of tedious manual
• 10:38
  labor, then the API cost is more than worth the
• 10:41
  financial investment. Use it for the heavy lifting,
• 10:44
  not your daily email drafting. Save the cheaper
• 10:46
  models for the simple, everyday text generation
• 10:49
  tasks sponsor. Knowing the steep price tag, we
• 10:53
  have to look at what that money actually buys
• 10:55
  you in a real day -to -day workflow. Seeing this
• 10:59
  agentic model actually in action changes your
• 11:02
  entire perspective. Let us talk about Riley Brown's
• 11:05
  fascinating real -world test. Okay. Riley gave
• 11:09
  Astra access to his first -person shooter game
• 11:11
  project. The video game project is currently
• 11:14
  called AfterFlash. He simply asked the AI to
• 11:17
  find the best improvements. He told it to just
• 11:19
  implement them without asking for permission.
• 11:21
  And the final results were nothing short of stunning.
• 11:24
  Astra autonomously changed 20 different files
• 11:27
  across the game project. It successfully passed
• 11:30
  80 automated developer checks without any human
• 11:32
  intervention. It didn't just tweak random text
• 11:35
  or fix a tiny bug. It added complex terrain changes
• 11:38
  and new vehicle movement mechanics. It added
• 11:40
  surface behavior and reactive fuel barrels to
• 11:43
  the game. It did all of this entirely inside
• 11:45
  the existing code base. It did not just generate
• 11:47
  isolated snippets of useless code. Right. It
• 11:50
  worked seamlessly inside the established project
• 11:52
  architecture. It made systemic changes, tested
• 11:55
  them, and ensured nothing broke. Modifying video
• 11:58
  game physics engines is incredibly complicated
• 12:01
  work. Another developer built a 3D browser game
• 12:04
  from a single starting prompt. Astra handled
• 12:07
  absolutely all the tedious in -between work automatically.
• 12:11
  It didn't just return a massive block of code
• 12:13
  and say good luck. It handled the entire project
• 12:16
  lifecycle until it reached a working result.
• 12:19
  But here is where it gets truly wild for average
• 12:22
  users. Oh yeah. We need to discuss literal direct
• 12:26
  computer use by the AI. I am talking about the
• 12:29
  incredible Canva software demonstration. The
• 12:32
  user's initial prompt was incredibly simple and
• 12:35
  brief. They just typed the phrase, draw me inside
• 12:37
  Canva. Astra did not output basic text instructions
• 12:40
  for the user. It did not tell the user where
• 12:42
  they should physically click. No, it actually
• 12:44
  opened the Canva application on the computer
• 12:46
  desktop. It looked carefully at a provided reference
• 12:49
  image on the screen, and it clicked around the
• 12:52
  software interface itself to recreate it. It
• 12:54
  is calculating pixel coordinates directly on
• 12:57
  the screen. It is physically operating the graphical
• 13:00
  user interface like a human. It's like hiring
• 13:03
  a junior assistant who doesn't just read the
• 13:05
  manual to you, but actually grabs the mouse and
• 13:08
  clicks the buttons. It is exactly like having
• 13:10
  an invisible digital assistant. The AI is continuously
• 13:14
  taking screenshots to understand the current
• 13:16
  interface. It places bounding boxes around icons
• 13:20
  to know where to click. And the workflow gets
• 13:22
  even more complex from there. Sharif Shamim tested
• 13:25
  this visual capability with Blender software.
• 13:28
  He asked Astra to recreate the Palace of Fine
• 13:30
  Arts. He wanted a full 3D model of the historic
• 13:34
  building. Astra quickly realized it did not have
• 13:36
  enough structural information. It recognized
• 13:38
  a massive gap in its own internal training data.
• 13:41
  So it autonomously performed its own independent
• 13:43
  web research. It opened a browser and was looking
• 13:46
  for reference images online. Yep. It did this
• 13:48
  just to complete the 3D modeling task. Humans
• 13:51
  do this kind of problem solving naturally all
• 13:53
  the time. But AI models usually just hallucinate
• 13:56
  an answer when they lack data. Astra solved its
• 13:59
  own knowledge gap and returned to the main task.
• 14:02
  This workflow feels much closer to real professional
• 14:05
  work. You are not doing the tedious background
• 14:08
  research for the AI anymore. This agentic behavior
• 14:11
  translates directly to complex business data,
• 14:14
  too. Let us look at Simon Smith's recent presentation
• 14:17
  test. Simon gave Astra a raw, unstructured CSV
• 14:21
  data file. It contained complex data on various
• 14:24
  companies affected by AI. He asked it to build
• 14:27
  complete professional slide presentation. Astra
• 14:30
  analyzed the raw numbers and found the hidden
• 14:32
  narrative. It built the presentation directly
• 14:34
  inside Excel and Power BI software. The very
• 14:38
  first version was technically usable, but a bit
• 14:40
  plain. So Simon gave the AI one quick round of
• 14:43
  feedback. He asked it to improve the overall
• 14:45
  narrative story structure. You wanted varied
• 14:47
  slide layouts and much stronger visual elements.
• 14:50
  Astra delivered a second version that was 95
• 14:52
  % ready to present. It did not just write generic
• 14:55
  text for the slides. It found a clear, compelling
• 14:58
  narrative story hidden in the raw data. Right.
• 15:02
  This represents a massive leap in everyday business
• 15:05
  utility. It works right inside the environment
• 15:07
  where the data already lives. If it can operate
• 15:10
  Canva and Excel directly, does this mean knowing
• 15:13
  how to navigate software interfaces is becoming
• 15:15
  a useless skill? It fundamentally shifts our
• 15:18
  entire role in the modern workplace. We're not
• 15:21
  going to be clicking software buttons manually
• 15:23
  anymore. We are defining strategic goals and
• 15:26
  managing the final digital outcomes. We're moving
• 15:28
  from software operators to high level project
• 15:31
  directors. You essentially become the manager
• 15:33
  of a very capable digital workforce. Yeah. Beat.
• 15:36
  Pulling all these different threads together
• 15:38
  into one cohesive picture. We have the proactive
• 15:41
  agentic behavior and the high API costs. We have
• 15:43
  these incredible real world business applications
• 15:46
  changing workflows. Yeah. What is the primary
• 15:48
  big idea we should take away today? The big idea
• 15:51
  is that GPT -6 is strictly for complex work.
• 15:54
  It requires strong reasoning capabilities and
• 15:57
  deeply connected execution steps. It is not designed
• 16:00
  for drafting quick, simple emails. If you need
• 16:03
  a quick rewrite, stick to a lighter, cheaper
• 16:05
  model. But for serious multi -step projects,
• 16:08
  this is a true paradigm shift. You give Astra
• 16:11
  a serious goal and you let it work. It moves
• 16:14
  through the entire complex task completely on
• 16:16
  its own. And you only step in when human judgment
• 16:18
  is actually needed. It fundamentally shifts chat
• 16:21
  GPT from a simple tool you guide every second.
• 16:24
  It becomes a highly capable colleague carrying
• 16:26
  the actual heavy workload. It is no longer waiting
• 16:29
  for you to guide every single move. It is taking
• 16:31
  initiative and solving unexpected problems proactively.
• 16:34
  And that changes the entire nature of how we
• 16:36
  work with machines. We are entering an era of
• 16:39
  true digital delegation. I want to leave you
• 16:41
  with this final provocative thought today. If
• 16:43
  GPT -6 can already navigate Canva and Excel perfectly,
• 16:47
  if it can do this just by looking at the computer
• 16:49
  screen, and if it is physically moving the cursor
• 16:52
  to click the menus, how much longer will graphical
• 16:54
  user interfaces even need to exist? If the AI
• 16:57
  is doing all the tedious clicking for us, maybe
• 17:00
  the future of software has absolutely no buttons
• 17:02
  at all. Thank you for joining us on this deep
• 17:04
  dive. Stay curious, and we will catch you next
• 17:06
  time.