Latest / AI Fire Daily / #643 Neil: Gemini 4 Argon Gets A Real Test Beyond The Benchmarks

Transcript

#643 Neil: Gemini 4 Argon Gets A Real Test Beyond The Benchmarks · AI Fire Daily

• 0:00
  We are watching this massive AI war unfold every
• 0:03
  single day. Yeah, it moves incredibly fast. Gemini
• 0:07
  usually looks really impressive on paper, but
• 0:10
  constantly gets overshadowed by GPT or Claude.
• 0:13
  That narrative is shifting very aggressively
• 0:15
  right now. It really is. Gemini 4 Argon is officially
• 0:19
  stepping into the ring. It boasts up to 1 million
• 0:22
  output tokens, and it brings a shockingly low
• 0:25
  hallucination rate. It is a massive structural
• 0:29
  shift in the landscape. We are finally moving
• 0:31
  toward actual dependable reliability. Welcome
• 0:35
  to the deep dive. I am really thrilled you are
• 0:38
  joining us today. I've got to admit something
• 0:40
  right off the bat. I still wrestle with prompt
• 0:42
  drift myself when trying to get AI to build complex
• 0:45
  things. Oh, you are definitely not alone in that.
• 0:48
  So these new specs we are looking at are genuinely
• 0:50
  wild. Today, we have a great roadmap for you.
• 0:54
  We are exploring Gemini 4 Argon's benchmark surprises.
• 0:57
  We're going to look at a brutal real world test,
• 0:59
  too. Right. It actually built a playable 3D browser
• 1:02
  game from scratch. Then we will examine its aggressive
• 1:06
  pricing model and we'll see why its 1 .5 % hallucination
• 1:10
  rate is a game changer. We will compare it against
• 1:13
  GPT 6 .1 Sol and Claude Opus 5 .5. This is going
• 1:18
  to be a fascinating one to unpack. OK. Let's
• 1:20
  get right into this new model. What exactly are
• 1:23
  we looking at with Gemini 4 Argonne? Well, it
• 1:25
  is a new frontier model from Google. It's built
• 1:28
  specifically for deep knowledge work. It handles
• 1:31
  advanced coding and heavy multimodal tasks. And
• 1:34
  there is a massive focus on cybersecurity here.
• 1:37
  Access is still pretty restricted right now.
• 1:39
  It is currently in early access via the Fairwind
• 1:42
  program. Google is offering it to limited cloud
• 1:44
  customers. They're working closely with specific
• 1:47
  government agencies, too. select cybersecurity
• 1:50
  partners are really putting it through the wringer.
• 1:53
  What really grabs your attention is the output
• 1:55
  capacity. It can generate up to one million output
• 1:58
  tokens. Let's break that jargon down for a second.
• 2:01
  Think of tokens as the basic building blocks
• 2:03
  of words. One million output tokens is a massive
• 2:07
  canvas. Right. It equals roughly three thousand
• 2:10
  pages of text. That scale completely changes
• 2:13
  the game. That massive capacity fundamentally
• 2:19
  changes how we actually work. You can feed it
• 2:23
  incredibly long workflows. You don't have to
• 2:26
  break complex projects into tiny tasks. You don't
• 2:28
  have to constantly hold its hand. Exactly. It's
• 2:31
  a totally different approach to prompting. It
• 2:33
  is like stacking Lego blocks of data. You build
• 2:37
  the entire structure in one go. You don't hand
• 2:39
  them instructions for every single brick. You
• 2:42
  hand them the blueprint and they build it. That
• 2:44
  is a perfect way to visualize it. And the early
• 2:47
  benchmark data is showing some very surprising
• 2:50
  victories. It is absolutely dominating in pure,
• 2:53
  dense knowledge work. Yeah. Let's look at the
• 2:55
  VALS index really quickly. Gemini 4 scores near
• 2:58
  69 % on this test. It pushes slightly ahead of
• 3:02
  both GPT -6, Astra, and Claude Opus 5 .5. Those
• 3:06
  are solid wins, but they are relatively close.
• 3:08
  They are close. But the gap becomes an absolute
• 3:11
  canyon on legal tasks. Harvey's legal agent benchmark
• 3:15
  is notoriously difficult for AI. It tests how
• 3:18
  well a model handles massive documents. It is
• 3:20
  a brutal test of reading comprehension. I was
• 3:24
  looking at this chart and couldn't believe it.
• 3:26
  Gemini 4 dominates with nearly a 20 % score.
• 3:31
  GPT -6 Astra only manages about 5%. And Claude
• 3:35
  Opus 5 .5 drops down to under 4%. If you use
• 3:39
  AI for dense research, this is huge. These benchmarks
• 3:42
  matter way more than flashy general intelligence
• 3:45
  scores. I am stuck on that legal score, though.
• 3:48
  A gap like 19 % versus 5 % is massive. Why is
• 3:52
  the gap on the legal benchmark so massive compared
• 3:54
  to general tests? What is fascinating here is
• 3:57
  how it handles pure textual density. Think about
• 4:00
  a complex commercial lease agreement. You have
• 4:03
  a clause on page 2. It references a penalty on
• 4:05
  page 40. Right, and that gets confusing fast.
• 4:08
  But that penalty only applies if a condition
• 4:11
  on page 12 is met. Older models completely lose
• 4:14
  their minds trying to track that. They forget
• 4:16
  the first condition by the time they reach the
• 4:18
  penalty. Exactly. Gemini 4 seems specifically
• 4:21
  tuned for deep, nuanced parsing. It holds the
• 4:24
  logical thread across incredibly dense paragraphs.
• 4:27
  It maps the dependencies perfectly. So it understands
• 4:30
  the nuance of dense text, not just raw data.
• 4:33
  Yeah, exactly. It reads the underlying architecture
• 4:36
  of the argument itself. Since it understands
• 4:39
  dense legal texts so cleanly, I am really curious
• 4:42
  how it handles the ultimate logic test. Let's
• 4:45
  talk about its push into coding and cybersecurity.
• 4:48
  Well, coding is actually a bit of a mixed bag
• 4:50
  here. On the deep SWE version 1 test, it definitely
• 4:54
  excels. It hits a very impressive 78%. That pushes
• 4:58
  it comfortably ahead of GPT -6 Astra and Claude
• 5:00
  Opus. It does. But we really have to look at
• 5:03
  the whole picture. Yeah, it doesn't win every
• 5:05
  single coding test, so we can't just crown it
• 5:08
  the undisputed king of code yet. That is exactly
• 5:11
  why we can't rely on a single chart. Different
• 5:14
  models have very different structural strengths.
• 5:17
  On the Frontier SWE version 2 test, it falls
• 5:20
  behind. It scores around 55 % there. That is
• 5:23
  well below GPT -6 Astra's 65%. However, its cybersecurity
• 5:28
  performance is incredibly hard to ignore right
• 5:30
  now. Google is clearly pivoting this model toward
• 5:34
  aggressive digital defense. Let's look at the
• 5:37
  practical security test for a second. It hits
• 5:39
  nearly 86 % on real -world vulnerability discovery.
• 5:44
  That is a massive leap in finding actual usable
• 5:48
  exploits. It also scores very high on the WIS
• 5:50
  penetration test. The gap becomes even clearer
• 5:53
  when you look at defense. There's this benchmark
• 5:55
  called the Grey Swan IPI test. Security researchers
• 6:00
  try to break the model on purpose. They use complex
• 6:04
  prompt injections. They try to trick it into
• 6:06
  writing malware. They layer their requests and
• 6:09
  confusing logic to force a jailbreak. The Grey
• 6:12
  Swan benchmark measures how often these jailbreaks
• 6:14
  actually succeed. A lower score is much better
• 6:17
  here. Gemini 4 Argon records an absurdly low
• 6:21
  0 .7%. That is measured at 15 different attack
• 6:26
  attempts. That number really stood out to me.
• 6:30
  Does that tiny 0 .7 % attack success rate mean
• 6:34
  it's inherently safer to deploy? It absolutely
• 6:38
  points to highly aggressive defensive tuning.
• 6:40
  The model is trained to recognize incredibly
• 6:42
  complex malicious intent. It acts as a highly
• 6:45
  paranoid digital filter. It simply refuses to
• 6:48
  execute harmful code chains. Right. It doesn't
• 6:51
  fall for the layered logic tricks. Right. A tiny
• 6:55
  success rate makes it a highly defensive tool.
• 6:57
  Precisely. It is built to protect the system
• 7:00
  at all costs. If it is this paranoid about malicious
• 7:03
  code, I am curious, how does it handle something
• 7:06
  much messier? Let's talk about long context and
• 7:09
  its multimodal mastery. That is where it really
• 7:13
  flexes its muscles. How does it digest massive
• 7:16
  amounts of mixed media? Gemini has always been
• 7:19
  a powerhouse in long context, but Gemini 4 pushes
• 7:22
  these boundaries to a completely new level. Look
• 7:25
  at the GraphWox benchmark score. It holds incredibly
• 7:28
  strong at massive data scales. It maintains an
• 7:31
  84 % accuracy right up to the 1 million token
• 7:34
  mark. That is a staggering amount of data retention.
• 7:38
  You are talking about feeding it hundreds of
• 7:40
  dense manuals. And then we have the LVBench score.
• 7:43
  That specifically tests long video understanding
• 7:46
  capabilities. It reaches nearly 92 % on that
• 7:50
  test. That is the highest score across the entire
• 7:53
  evaluation table. Whoa. Imagine scaling to a
• 7:57
  billion queries across an hour -long video. That
• 8:00
  completely changes how you can interact with
• 8:02
  media. If you work with long videos or massive
• 8:05
  architectural documents, these specific multimodal
• 8:08
  scores matter way more than a generic intelligence
• 8:11
  score. Yeah, you need the AI to actually remember
• 8:14
  every single detail, not just the first 10 minutes
• 8:17
  of the video. How does it manage to not lose
• 8:19
  the thread when processing a million tokens?
• 8:22
  If we connect this to the bigger picture of AI
• 8:24
  design, it uses highly advanced, persistent attention
• 8:28
  mechanisms. Older models often suffered from
• 8:30
  the needle -in -a -haystack problem. Think of
• 8:33
  a detective reading an 800 -page novel. Right.
• 8:36
  Older models forgot the clue on page 1 by page
• 8:39
  500. Gemini 4 actively links related concepts
• 8:42
  across the entire massive dataset. It never drops
• 8:45
  the magnifying glass. So it holds the thread
• 8:48
  of the story without losing the plot. Yes. It
• 8:51
  keeps the entire context window fully active
• 8:54
  and connected. Now, we reached the most exciting
• 8:56
  part of this deep dive. Benchmarks are really
• 8:59
  just paper promises at the end of the day. We
• 9:02
  need a practical, brutal test of these systems
• 9:04
  working together. I absolutely love this part
• 9:07
  of the analysis. The reviewer wanted to completely
• 9:10
  bypass the generic safe tests. So, they prompted
• 9:14
  the AI to build a full video game. A complete
• 9:17
  browser -based 3D survival game from scratch.
• 9:21
  It had to use HTML, CSS, JavaScript, and 3 .js.
• 9:25
  It had to run directly in the browser window.
• 9:28
  No backend systems or external game engines were
• 9:30
  allowed at all. The core concept was incredibly
• 9:33
  detailed and strict. The players trapped in an
• 9:35
  abandoned industrial area. The primary goal is
• 9:38
  to survive and collect energy cells. The physics
• 9:42
  requirements alone were very complex. It needed
• 9:45
  standard WASD third -person movement controls,
• 9:48
  and it needed smooth acceleration and deceleration
• 9:50
  curves. It also required a functional sprinting
• 9:53
  system using the shift key. That sprinting had
• 9:56
  to tie into a working stamina system. And your
• 9:58
  stamina had to recover naturally when you stopped
• 10:00
  moving. The camera work was another huge technical
• 10:04
  challenge here. It needed a smooth third -person
• 10:06
  follow camera. And the camera could not clip
• 10:09
  badly through large environmental objects. That
• 10:11
  is a nightmare to code from scratch. Then there
• 10:13
  was the actual enemy AI logic. Enemies had to
• 10:16
  dynamically spawn and actively detect the player.
• 10:19
  They needed slightly different randomized movement
• 10:22
  speeds. They even needed separation behavior.
• 10:25
  So they didn't just overlap into one polygon.
• 10:28
  It also required a full health and dynamic scoring
• 10:30
  system. Collecting energy cells increased your
• 10:33
  overall score. The game's difficulty had to increase
• 10:36
  smoothly over time. The environment needed atmospheric
• 10:39
  fog and dynamic nighttime lighting. It needed
• 10:42
  real -time shadows and clear contrast between
• 10:44
  safe and dangerous areas. It required precise
• 10:48
  visual feedback, like red screen flashes for
• 10:50
  taking damage. It needed a clean user interface
• 10:53
  showing your health and survival time. It even
• 10:55
  needed a completely functional state -driven
• 10:57
  game over screen. Doing all of this in one shot
• 11:00
  is conceptually insane. The result was... Genuinely
• 11:04
  surprising to read about. It wasn't just a massive
• 11:07
  broken bug collection. It actually loaded a beautifully
• 11:10
  polished start screen. The 3D industrial yard
• 11:13
  was fully visible and rendered. Once the game
• 11:16
  actually started, the systems simply worked.
• 11:18
  The player could move smoothly around the map.
• 11:21
  The camera followed correctly without breaking
• 11:23
  the geometry or glitching out. The HUD showed
• 11:26
  your health and stamina perfectly syncing up.
• 11:28
  It's incredibly impressive state management.
• 11:32
  It's not just generating a brick. It's building
• 11:34
  the whole house and making sure the plumbing
• 11:36
  works. That is the perfect way to frame the achievement.
• 11:40
  State management is notoriously difficult for
• 11:42
  language models. The AI has to remember where
• 11:45
  the player is. It has to track how much stamina
• 11:47
  is left. It has to know where every single enemy
• 11:49
  is located. And it has to update all of this
• 11:52
  30 times a second. If it forgets a variable,
• 11:55
  the game crashes. If it mixes up the enemy's
• 11:57
  speed, the logic breaks. Jim and I forked all
• 12:00
  the internal logic perfectly intact. The reviewer
• 12:04
  used a very strict, highly technical specification.
• 12:07
  They didn't just say, build me a cool 3D survival
• 12:10
  game. No, they were incredibly methodical. Why
• 12:13
  was giving it a highly technical specific prompt
• 12:16
  so critical here? It completely removes the AI's
• 12:19
  ability to guess or improvise. Vague prompts
• 12:22
  let models take lazy architectural shortcuts.
• 12:25
  By defining the exact physics, lighting and logic,
• 12:28
  you lock it in. It forces the model to actually
• 12:31
  engineer the solution properly. It tests pure
• 12:34
  capability, not just creativity. Giving strict
• 12:37
  boundaries meant it couldn't cheat with easy
• 12:39
  shortcuts. Exactly. It had to prove it could
• 12:41
  actually build complex interactive systems. We're
• 12:45
  going to take a brief pause here. We'll be right
• 12:47
  back. And we are back. Now that we know it works
• 12:51
  practically in the real world, how does the cost
• 12:53
  and reliability justify actually using it over
• 12:56
  Claude or GPT? Let's look at the broad intelligence
• 12:59
  index scores first. They measure raw general
• 13:02
  capability across the board. GPT 6 .1 Sol sits
• 13:06
  right at a score of 54. Gemini 4 Argon is right
• 13:09
  behind it at 53. And GPT 6 Astra is also sitting
• 13:13
  at 53. They are all incredibly close in raw generic
• 13:17
  intelligence. But the hallucination rate is the
• 13:19
  massive glaring differentiator here. Before we
• 13:23
  compare, let's define hallucination. When the
• 13:26
  AI confidently makes up fake or incorrect information.
• 13:30
  That is exactly what we want to avoid. This is
• 13:32
  where Gemini 4 really pulls away from the pack.
• 13:35
  Gemini 4 is sitting at an industry -low 15 %
• 13:37
  hallucination rate. Well, GPT -6 Astra is way
• 13:40
  up at a 51 % rate, and GPT -6 .1 Sol sits at
• 13:45
  an astonishing 54%. That is a staggering difference
• 13:49
  in actual daily reliability. Let's break down
• 13:51
  the pricing model for a second. Gemini 4 is $2
• 13:54
  per 1 million input tokens. It is $10 per 1 million
• 13:59
  output tokens. It is worth noting that cashed
• 14:01
  input is 95 % cheaper. After the initial promo
• 14:04
  period, it moves up slightly. It goes to $4 .20.
• 14:08
  Right now, it averages about $1 .99 per task.
• 14:11
  Let's compare that directly to the major competitors.
• 14:14
  It is definitely more expensive than GPT -6 .1
• 14:18
  Sol. That model is only 72 cents per task. But
• 14:21
  Gemini is vastly cheaper than GPT -6 Astra. Astra
• 14:24
  costs $3 .26 per task. And Claude Opus 5 .5 is
• 14:29
  way up at nearly $6. So it sits perfectly in
• 14:33
  the middle financially. It gives you Opus -level
• 14:35
  intelligence without paying that massive Opus
• 14:37
  -level premium. Let me push back just a little
• 14:40
  bit here, though. For pure web development, Claude
• 14:43
  might still be the absolute go -to, but Gemini
• 14:47
  is clearly carving out a totally new space. It
• 14:50
  is becoming the reliable heavy data middle ground.
• 14:53
  Is the dramatically lower hallucination rate
• 14:56
  the actual killer feature here rather than the
• 14:58
  price? I absolutely believe that it is the deciding
• 15:01
  factor. Think about using an AI for heavy financial
• 15:05
  research. If it makes up a fake earnings number,
• 15:07
  that is catastrophic. You have to double -check
• 15:10
  every single figure it gives you. That verification
• 15:12
  time completely destroys the efficiency gains.
• 15:15
  It is a massive hidden tax on your workflow.
• 15:17
  A cheap model that lies to you is incredibly
• 15:19
  expensive. You spend hours manually verifying
• 15:22
  the false outputs. That 1 .5 % rate makes it
• 15:26
  a truly viable, trustworthy research partner.
• 15:30
  Yeah, trusting the output matters much more than
• 15:32
  saving a few cents. It is the stark difference
• 15:34
  between a neat toy and a professional tool. So
• 15:38
  we have covered a massive amount of ground today.
• 15:41
  Let's quickly recap the shifting landscape we
• 15:43
  just explored. Gemini 4 Argon isn't a flawless
• 15:47
  sweep of the entire leaderboard. But it is completely
• 15:51
  impossible to ignore right now. It brings absolutely
• 15:54
  dominant knowledge work capabilities to the table.
• 15:57
  It has that massive 1 million token output limit
• 15:59
  for complex builds. The industry -low 1 .5 %
• 16:03
  hallucination rate builds actual verifiable trust.
• 16:06
  And it offers highly competitive pricing against
• 16:09
  the legacy major players. It places it right
• 16:12
  toe to toe with the very best out there. It seriously
• 16:15
  rivals GPT 6 .1 Sol and Claude Opus 5 .5. Google
• 16:20
  finally has a frontier model that competes without
• 16:22
  any hesitation. The AI wars are clearly entering
• 16:26
  a brand new phase. We are moving away from flashy
• 16:29
  paper specs to genuine, reliable utility. It
• 16:32
  is a genuinely exciting time to be building new
• 16:35
  things. I want to leave you with one final thought
• 16:38
  today. If an AI is hallucinating this rarely
• 16:41
  now, and it can build a functioning 3D game environment
• 16:45
  from a single prompt. That raises a very profound,
• 16:50
  slightly terrifying question for the future.
• 16:53
  Thank you for taking this deep dive with us today.
• 16:55
  Keep questioning the tools, and always keep exploring.
• 16:59
  Take care.