Latest / AI Fire Daily / #642 Neil: GPT-6.1 Sol In Codex Surprised Me Across Three Real Tests

Transcript

#642 Neil: GPT-6.1 Sol In Codex Surprised Me Across Three Real Tests · AI Fire Daily

• 0:00
  We hear big promises in technology constantly.
• 0:02
  Oh, absolutely. Every single week. Right? But
• 0:05
  GPT 6 .1 Sol just launched with a massive claim.
• 0:09
  It promises near GPT 6 Astra performance overall,
• 0:12
  but it costs drastically less to run daily. Words
• 0:16
  are incredibly easy until you test them. Welcome
• 0:19
  to the deep dive. We are glad you are here. Exactly.
• 0:22
  I mean, new models always sound amazing on a
• 0:25
  spec sheet. Always. But then you try to make
• 0:27
  them do actual work. That is when the wheels
• 0:30
  usually fall off. So that is exactly what we
• 0:33
  are testing today. We pushed GPT 6 .1 sole to
• 0:37
  the absolute technical limit. We ran it inside
• 0:41
  Codex for this experiment. Codex is essentially
• 0:44
  our secure internal testing sandbox environment.
• 0:47
  We're going to look at the raw numbers first,
• 0:49
  then we will analyze three intense stress tests.
• 0:53
  Okay, let's unpack this. We will cover a full
• 0:55
  browser operating system build first, then we
• 0:58
  will recreate Jerry Seinfeld's apartment in 3D,
• 1:00
  and finally we will run a dynamic robot arm simulation.
• 1:04
  And to wrap it up, we compare it directly. We
• 1:08
  stacked it up against Claude Sonnet 5 .5. We
• 1:11
  need to understand the baseline numbers first.
• 1:13
  This release is incredibly interesting on paper
• 1:15
  today. It really is. OpenAI is essentially commoditizing
• 1:19
  raw processing power now. We need to look at
• 1:22
  the standard pricing structure. Yeah. So the
• 1:24
  pricing structure is the biggest selling point
• 1:26
  here. You pay $2 per million input tokens. Output
• 1:30
  costs $10 per million generated tokens. Wow.
• 1:33
  I know. That is already roughly a fifth of the
• 1:36
  cost of its predecessor. That aggressive baseline
• 1:38
  price completely changes the math. But the caching
• 1:41
  features make it significantly better. Cached
• 1:43
  input drops down to just $0 .10. Yeah, that is
• 1:46
  a massive operational cost reduction. Cached
• 1:49
  writes cost $2 .50. Right. This makes repetitive
• 1:52
  daily tasks vastly cheaper over time. You're
• 1:55
  basically saving compute on things it already
• 1:57
  processed. We also have long context pricing
• 2:00
  to consider now. It jumps slightly for those
• 2:02
  massive prompt payloads. It costs $4 per million
• 2:05
  for standard long context. Right, but cached
• 2:08
  input sits at exactly 20 cents. Cache writes
• 2:11
  go up to $5 total. Output costs $15 per million
• 2:15
  tokens for heavy context. This pricing structure
• 2:17
  feels very intentional to me. It is like stacking
• 2:21
  Lego blocks of data. Yeah, exactly. You only
• 2:24
  pay a premium for complex builds. The long -term
• 2:26
  memory builds cost slightly more upfront. It
• 2:29
  still easily beats standard GPT -6 Astra pricing.
• 2:32
  Oh, without a doubt. Plus, you get a massive
• 2:35
  overall context window. We get 1 .05 million
• 2:38
  tokens. That is a completely staggering amount
• 2:41
  of data. I mean, you can feed it entire code
• 2:43
  bases, all simultaneously. Yeah. You also get
• 2:46
  up to 128k max output tokens. And the knowledge
• 2:50
  cutoff date is April 30, 2026. Which is an incredibly
• 2:54
  recent date for these models. Let us define the
• 2:57
  context window very clearly. It is how much text
• 3:00
  the AI can remember at one time. Perfectly summarizes
• 3:04
  the technical concept for everyone. Sol also
• 3:07
  introduces several different internal reasoning
• 3:09
  levels. You can choose settings ranging from
• 3:11
  low to max. It offers low, medium, high, x high,
• 3:16
  and max. That gives you way more granular operational
• 3:18
  control. Right. You can perfectly balance speed,
• 3:21
  cost, and task difficulty. You do not always
• 3:24
  need maximum reasoning power. Exactly. Simple
• 3:27
  data extraction runs incredibly fast and cheap.
• 3:30
  Complex architectural coding gets the heavy processing
• 3:32
  it requires. Why burn expensive compute if you
• 3:36
  are just summarizing emails? So why do adjustable
• 3:39
  reasoning levels actually matter here? Well,
• 3:41
  they let developers surgically manage their API
• 3:44
  budgets. You prevent wasting expensive compute
• 3:46
  on completely trivial tasks. Matching computational
• 3:49
  effort to the problem saves real money. We know
• 3:52
  the context window is absolutely massive now,
• 3:55
  so we wanted to max out that output capacity.
• 3:57
  We tested a continuous massive coding task next.
• 4:01
  Yep, we gave it a really heavy project prompt.
• 4:03
  We asked Sol to build a full browser OS. We forced
• 4:06
  it to run locally inside Chrome, and the entire
• 4:09
  project required one single HTML file. No external
• 4:13
  frameworks or additional build tools were allowed.
• 4:16
  No React, no Node, no helpful libraries, just
• 4:19
  vanilla web code packed into one document. That
• 4:21
  forces the model to be incredibly efficient internally.
• 4:24
  It had to build a fully functional desktop environment.
• 4:28
  We required a polished UI with a task bar. It
• 4:31
  also needed draggable and resizable app windows.
• 4:34
  We demanded an app launcher and full screen support.
• 4:37
  Right. We also asked for procedurally generated
• 4:40
  desktop wallpapers. Those wallpapers needed subtle,
• 4:44
  animated movement built in. They also had to
• 4:46
  use user -adjustable numerical seeds. Generating
• 4:50
  moving visual patterns through pure math is tough.
• 4:54
  The test required five working applications to
• 4:56
  run smoothly. We demanded an email client and
• 4:59
  a text app. It also needed a sound or music -making
• 5:02
  app. We required a dedicated system settings
• 5:05
  application too. plus one additional useful app
• 5:08
  of its own choice. It actually chose to build
• 5:10
  a working journal application. We also demanded
• 5:13
  two fully playable 3D games. The first was a
• 5:17
  GTA -style city sandbox game. It needed vehicles,
• 5:21
  police chases, and open environments. The second
• 5:23
  game was a 3D flight game. That one was called
• 5:26
  Sky Runner in our prompt. It needed responsive
• 5:28
  controls, level progression, and sound effects.
• 5:31
  Both games had to play directly inside the OS.
• 5:34
  They could not just open as separate browser
• 5:36
  pages. Here is where we really pushed the cognitive
• 5:39
  load. We added one special requirement to the
• 5:42
  prompt. The system needed a connected feature
• 5:44
  across different apps. We wanted it to feel like
• 5:48
  one unified system, so activity from SkyRunner
• 5:51
  had to show up in the journal. That kind of cross
• 5:53
  -app communication is incredibly hard. It requires
• 5:56
  managing global state within a single file. Sol
• 6:00
  finished... everything in exactly 33 minutes
• 6:03
  and 42 seconds. Whoa. Imagine generating an entire
• 6:06
  operating system in just half an hour. That is
• 6:10
  staggering. Yeah. Human developers take that
• 6:13
  long just setting up their environments. Running
• 6:16
  npm install alone takes half that time. Right.
• 6:19
  It is honestly mind blowing to watch happen live.
• 6:23
  Everything actually worked perfectly right out
• 6:24
  of the gate. It successfully avoided fake interactions
• 6:28
  and placeholder UI buttons. It kept performance
• 6:31
  smooth enough for standard Chrome playback. It
• 6:34
  even tested major interactions before finalizing
• 6:36
  the code. Though it was slightly less visually
• 6:39
  polished overall. Yeah, Sonnet 5 .5 definitely
• 6:42
  looks a bit cleaner visually. But Sol reached
• 6:45
  the second playable game way faster. Its core
• 6:48
  strength is clearly raw speed and task completion.
• 6:51
  What does this sky runner to journal connection
• 6:53
  actually prove here? It proves the model understands
• 6:56
  complex global state management. It comprehends
• 6:59
  how independent programs share underlying data
• 7:01
  architecture. The AI truly grasps how separate
• 7:04
  programs share data. Yeah. Sol handled massive
• 7:08
  logic code incredibly fast today. But raw code
• 7:11
  is just pure logical instructions. We wanted
• 7:14
  it to test its spatial awareness next. Exact
• 7:18
  visual replication is a totally different skill
• 7:20
  entirely. We gave it a very specific cultural
• 7:24
  touchstone. We asked it to recreate Jerry Seinfeld's
• 7:27
  apartment. It had to be a highly detailed 3D
• 7:30
  scene. It had to match the real TV set perfectly.
• 7:34
  This forces the model to understand physical
• 7:36
  3D space. It cannot just generate a random generic
• 7:39
  sitcom room. The layout and furniture placement
• 7:42
  had to match reality. We demanded specific details
• 7:44
  across the whole living space. We asked for the
• 7:48
  main living room area specifically. We needed
• 7:51
  the kitchen, front door, and the bedroom. We
• 7:53
  specifically requested Jerry's iconic green Klein
• 7:56
  bike too. We wanted the TV and surrounding wall
• 7:59
  decorations included. The prompt included the
• 8:01
  desk and computer area too. It even needed the
• 8:05
  window view of adjacent buildings. We also asked
• 8:08
  for highly useful viewing controls. We wanted
• 8:10
  daylight and warm interior lighting toggle options.
• 8:14
  We required multiple camera presets for the entire
• 8:16
  scene. We even asked for a full overhead dollhouse
• 8:19
  view. The build process took almost an hour entirely.
• 8:22
  This was much more ambitious than the browser
• 8:25
  OS. The initial results were honestly highly
• 8:29
  impressive to see. It successfully created multiple
• 8:32
  angles and interactive lighting controls. I could
• 8:34
  spot the green Klein bike very easily. The kitchen
• 8:38
  setup and the front door looked great. The dollhouse
• 8:40
  view made exploring this space really fun. The
• 8:43
  small props and recognizable cultural references
• 8:45
  were there. But the model did reveal a very clear
• 8:48
  flaw. The room positions did not match perfectly
• 8:51
  overall. The layout proportions felt a bit off
• 8:54
  geometrically. It looked completely convincing
• 8:56
  at a quick first glance, but checking the full
• 8:59
  apartment exposed spatial mapping errors. A true
• 9:03
  recreation requires exact geometric layout math.
• 9:07
  I still wrestle with getting AI to understand
• 9:09
  basic spatial layouts myself. What's fascinating
• 9:12
  here is the distinct difference in skills. Generating
• 9:15
  specific recognizable objects like a bike is
• 9:18
  easy. Right. It knows the semantic concept of
• 9:20
  a bicycle, but understanding true spatial geometry
• 9:23
  and room flow is much harder. It built the pieces
• 9:26
  but struggled with physical placement. A language
• 9:29
  model maps token proximity, not actual mathematical
• 9:32
  space. It knows the word couch is near the word
• 9:34
  TV. Exactly. It does not know the exact physical
• 9:37
  coordinate distance required. So why does AI
• 9:40
  struggle with exact room proportions? It relies
• 9:43
  heavily on predicting sequential text patterns.
• 9:46
  It simply lacks a true internal physics engine.
• 9:51
  Text prediction simply fails at creating precise
• 9:53
  mathematical layouts. Mm -hmm. Mid -roll sponsor
• 9:56
  read. It struggled slightly with static 3D layout
• 10:00
  roles. We decided to push it even further next.
• 10:02
  We wanted to test a dynamic 3D environment now.
• 10:06
  We needed to test dynamic planning and failure
• 10:09
  recovery. We asked it to control a virtual robot
• 10:12
  arm. And the prompt required inspecting the codex
• 10:15
  environment first. It searched for access to
• 10:18
  real hardware components initially. It checked
• 10:21
  for robot joint or standard Cartesian controls.
• 10:24
  You know, Cartesian controls just mean moving
• 10:25
  in X, Y, and Z space. It also looked for real
• 10:29
  camera feeds or poses. It checked for local control
• 10:32
  frameworks or setups. When it found none, it
• 10:35
  intelligently built a simulation. A local interactive
• 10:38
  browser simulation, to be exact. The constraints
• 10:41
  were incredibly strict for this specific test.
• 10:45
  It needed a multi -joint articulated arm with
• 10:47
  limits. The gripper had to visibly show its reach
• 10:50
  capability. We also required a virtual toy car
• 10:53
  on the table. The car needed realistic collision
• 10:56
  physics to slide correctly. It also required
• 10:59
  visual feedback from a virtual camera. It needed
• 11:01
  a main view and a secondary angle. This created
• 11:05
  a very complex internal logic loop. The AI agent
• 11:08
  had to estimate the car position. Then it had
• 11:11
  to plan a safe, physical approach. It moved the
• 11:13
  arm in small, highly controlled steps. You check
• 11:17
  the virtual camera after every important movement.
• 11:20
  If the car moved accidentally, it had to adjust.
• 11:23
  It had to align and close the gripper perfectly.
• 11:25
  It had to visually verify the grasp before lifting.
• 11:29
  We specifically forbade it from repeating movements
• 11:31
  blindly. It had to actively learn from its immediate
• 11:34
  mistakes. It had to detect exactly why grasping
• 11:38
  failed initially. Then it had to rewrite its
• 11:40
  code and change its approach. Like watching a
• 11:43
  toddler learn to pick up a block, but the brain
• 11:45
  is a massive neural net. Right, and it actually
• 11:48
  stopped when things got unsafe. It evaluated
• 11:51
  its own failures by reading the camera logs.
• 11:54
  It respected the mechanical joint limits perfectly
• 11:56
  during operation. It acted as a true agent in
• 12:00
  a continuous reasoning loop. It wrote code, observed
• 12:03
  the output, and adjusted dynamically. What does
• 12:06
  this complex failure recovery truly tell us?
• 12:09
  It proves the model can dynamically alter its
• 12:12
  own plans. It processes real -time feedback to
• 12:15
  overcome unexpected physical obstacles. The AI
• 12:18
  adapts instantly instead of blindly repeating
• 12:20
  failures. Yeah. We have seen Sol's raw speed
• 12:23
  and coding capability. We also saw its slight
• 12:26
  spatial geometry limitations. We need to compare
• 12:29
  it to the direct competition. We stacked it up
• 12:32
  against Claude Sonnet 5 .5. Both of these models
• 12:35
  sit at the exact same price. $2 for input, $10
• 12:38
  for output. Price is completely removed from
• 12:41
  the decision process here. Which makes this a
• 12:43
  purely capability -driven comparison test. GPT
• 12:46
  6 .1 Sol is definitely faster in most coding
• 12:49
  scenarios. It is highly efficient for long coding
• 12:52
  tasks. It remains incredibly stable, handling
• 12:54
  massive complex prompt instructions. You can
• 12:57
  use it reliably in longer developer workflows.
• 13:00
  It handles those massive context windows incredibly
• 13:02
  cleanly, without forgetting. Sonnet 5 .5 is slightly
• 13:06
  slower on those large builds, but it delivers
• 13:09
  far more polished visual results consistently.
• 13:12
  Yeah, Sonic creates richer visual environments
• 13:14
  and better creative detail. Rendered characters
• 13:17
  and environments just look much more natural
• 13:19
  overall. Sol sometimes feels a bit rough around
• 13:21
  the visual edges. It acts more like a pragmatic
• 13:24
  engineer than an artist. But Sol is just so practically
• 13:28
  efficient for logic. Sonnet feels more visual
• 13:31
  and heavily creative in execution. So what does
• 13:35
  this all mean? We have a clear user choice matrix
• 13:38
  developing here. You choose Sol for serious logical
• 13:42
  building in raw speed. It is perfect when you
• 13:45
  need complex systems working fast. You choose
• 13:47
  Sonnet for visual polish and artistic creativity.
• 13:51
  It wins when the task depends heavily on exact
• 13:53
  3D detail. Sol handles massive prompt workflows
• 13:57
  without structurally falling apart. It rarely
• 14:00
  needs much manual cleanup after finishing a project.
• 14:03
  Sonnet easily wins if you need exact visual replication.
• 14:07
  How should everyday users choose between these
• 14:09
  two powerful models? Evaluate whether your project
• 14:12
  requires logical speed or visual beauty. Then
• 14:16
  simply pick the tool that matches that requirement.
• 14:18
  Match the AI model strictly to your specific
• 14:21
  workflow needs. This brings us to our core takeaway
• 14:24
  today. GPT 6 .1 Sol left us with a very clear
• 14:28
  impression. It is not just a cheap budget AI
• 14:31
  model. It is a true powerhouse for long, complex
• 14:34
  workflows. It turns massive prompts into highly
• 14:38
  usable digital tools. And it does it with minimal
• 14:40
  human intervention required. It rarely falls
• 14:43
  apart under heavy cognitive logic load. The reliability
• 14:46
  factor alone makes it incredibly valuable today.
• 14:50
  If we connect this to the bigger picture. Wait.
• 14:52
  I mean, if we connect this to the bigger picture,
• 14:55
  AI efficiency is taking a truly massive leap
• 14:58
  forward. We are getting top -tier capability
• 15:00
  for absolute pennies now. We want you to really
• 15:03
  think about this idea. GPT 6 .1 Sol can reliably
• 15:07
  engineer a working operating system. It builds
• 15:10
  interconnected applications from scratch in exactly
• 15:13
  33 minutes. What happens to the definition of
• 15:16
  a software developer when iteration speed is
• 15:19
  effectively zero? Thank you for taking this deep
• 15:23
  dive with us. Outro music.