Latest / AI Fire Daily / #660 Neil: Gemini 4 Argon Tested In 3 Tough Real-World AI Challenges

Transcript

#660 Neil: Gemini 4 Argon Tested In 3 Tough Real-World AI Challenges · AI Fire Daily

• 0:00
  Imagine dictating a video game's rules to your
• 0:02
  computer. And then, three seconds later, you
• 0:06
  are actually playing it. It is wild to even think
• 0:09
  about. Google is clearly tired of being the runner
• 0:12
  -up in the AI race. Yeah, they really are. And
• 0:14
  with their latest drop, they are turning that
• 0:16
  sci -fi concept into a daily reality. Absolutely.
• 0:20
  They just released a model that can code a functional
• 0:23
  game from scratch. Right. And then, pivot to
• 0:26
  analyze a 50 -page commercial lease. All in one
• 0:29
  breath. It's a fascinating showdown. The landscape
• 0:33
  is shifting right beneath our feet. We have seen
• 0:35
  incremental updates for months, but this feels
• 0:38
  different. It really does. Welcome to the deep
• 0:41
  dive. I am so glad you're here with us today.
• 0:43
  We are exploring Google's brand new Gemini 4
• 0:46
  Argonne. If you've been watching this space,
• 0:48
  you know the stakes are high. Very high. So today
• 0:52
  we're going to look at its raw benchmark scores
• 0:54
  first. Then we're putting it through three real
• 0:57
  world trials. Building a game from scratch. Synthesizing
• 1:01
  messy research. And finally, coding a complex
• 1:04
  interactive web dashboard. After all that, we'll
• 1:08
  compare its aesthetics and its wildly fluctuating
• 1:10
  costs. Right, against Anthropix's Claude Opus
• 1:13
  5 .5. To know if the hype is actually justified,
• 1:17
  we always have to start with the wrong numbers.
• 1:19
  We do. Google is claiming some pretty massive
• 1:21
  victories on the benchmark battleground. They
• 1:23
  certainly are. Gemini 4 Argon is specifically
• 1:26
  built for long context windows. And agentic coding.
• 1:30
  Which is AI that writes, tests, and fixes its
• 1:33
  own software automatically. Exactly. It doesn't
• 1:36
  just autocomplete a line of code, it acts like
• 1:38
  a junior developer. And the benchmark results
• 1:41
  are genuinely surprising. It wins 13 out of 19
• 1:44
  standard tests. Yeah, and it ties for first and
• 1:48
  another. Overall, it's beating Claude Opus 5
• 1:50
  .5 and GPT -6 Astra. The knowledge work scores
• 1:54
  are what really caught my eye. If you look at
• 1:57
  the VALS index, it's a huge leap. Right, the
• 1:59
  VALS index. This evaluates complex multi -step
• 2:02
  reasoning. It tests how well an AI can juggle
• 2:05
  multiple constraints over a long period. Gemini
• 2:07
  hit 68 .9%. While Claude was trailing at 67%.
• 2:12
  Which sounds close, but in this specific test,
• 2:16
  every percentage point is a battle. It really
• 2:18
  is. It's kind of like stacking Lego blocks of
• 2:20
  data. If the model misinterprets a core definition
• 2:23
  on page one, Every single block stacked on top
• 2:27
  of it is going to be misaligned. Eventually toppling
• 2:29
  the whole structure. Exactly. And honestly, I
• 2:33
  still wrestle with prompt drift myself when I'm
• 2:35
  working with just a few paragraphs. Right. Keeping
• 2:38
  the model focused over massive documents is incredibly
• 2:41
  tough. It naturally tends to forget its initial
• 2:44
  instructions as the context window fills up.
• 2:46
  That's exactly why the coding scores matter so
• 2:48
  much here. DeepSWEv1 .1 measures long software
• 2:53
  engineering tasks. It is a notoriously brutal
• 2:56
  benchmark. Because software engineering isn't
• 2:58
  just about writing syntax. It's about understanding
• 3:00
  the underlying architecture. And Gemini scored
• 3:03
  77 .9%. It completely crushed Claude and Astra.
• 3:07
  It suggests Gemini is remarkably strong for multi
• 3:10
  -step coding projects. Context is everything
• 3:12
  there. But we should be clear, it doesn't win
• 3:14
  absolutely everything. No, Claude still beats
• 3:17
  it on Terminal Bench 4 .0. You can't just blindly
• 3:20
  trust it yet. That's a fair point. But I have
• 3:22
  to say, the legal reasoning benchmark surprised
• 3:25
  me more than anything else. Oh, Harvey's legal
• 3:27
  agent benchmark? Yeah. Legal tasks require rigid,
• 3:30
  unforgiving logic. AI usually hates that. It's
• 3:34
  true. Legal text is essentially kryptonite for
• 3:36
  large language models. A commercial lease isn't
• 3:38
  just words. It's a web of dependencies. Section
• 3:42
  4 references Section 12, which modifies a definition
• 3:44
  in Appendix A. Exactly. If an AI loses the thread
• 3:48
  for even a sentence, its analysis is compromised.
• 3:52
  Yet Gemini scored 19 .6%. Competitors are still
• 3:57
  stuck in the single digits. GPT -6 Astra scored
• 4:01
  5 .4%. And Claude's scored just 3 .8%. That is
• 4:06
  a massive leap forward. A 19 % success rate doesn't
• 4:10
  sound like a passing grade to a human. But it's
• 4:12
  huge for automated legal context. We are talking
• 4:16
  about parsing highly complex contracts. Identifying
• 4:19
  hidden liability clauses. Tracking specific definitions
• 4:23
  across 50 pages of dense text. It's incredibly
• 4:26
  tedious work for junior lawyers. If an agent
• 4:29
  can automate even 20 % of that document review
• 4:32
  reliably, that fundamentally changes the legal
• 4:34
  industry. They also tied Astra in cybersecurity.
• 4:38
  It scored 68 % on CWEbench v1. That is just one
• 4:43
  single point over clawed. That benchmark evaluates
• 4:46
  how well a model handles security vulnerabilities
• 4:48
  in raw code. It tests whether the AI can spot
• 4:51
  where a hacker might sneak in a cross -site scripting
• 4:54
  error. Or stop a SQL injection before it's deployed.
• 4:58
  But I have to push back a little on Google's
• 5:00
  victory lap here. The margins are razor -thin.
• 5:03
  Yeah, a 1 % lead in cybersecurity is nice for
• 5:06
  marketing, but it's basically a tie in the real
• 5:09
  world. Do these isolated laboratory benchmark
• 5:12
  victories actually translate into a better daily
• 5:14
  user experience? Not necessarily. Benchmarks
• 5:18
  are sterile laboratory conditions. Daily use
• 5:20
  involves messy human prompts and confusing edge
• 5:23
  cases. Raw potential is great, but daily usage
• 5:27
  is still messy and unpredictable. Exactly. Benchmarks
• 5:30
  are just numbers on a page. The real test is
• 5:33
  how it feels to build. Let's look at vibe coding
• 5:36
  next. That means creating software using natural
• 5:39
  language descriptions instead of writing code.
• 5:43
  Vibe coding is fascinating. The author of our
• 5:45
  source material tested it out. They asked Gemini
• 5:48
  to build a game directly in the browser. A 2D
• 5:50
  top -down HTML and JavaScript paintball capture
• 5:54
  the flag game. The prompt was incredibly detailed.
• 5:57
  They laid out the exact rules. Two teams. You
• 6:01
  play as the blue player against red opponents.
• 6:03
  It asks for standard WASD movement controls,
• 6:07
  mouse aiming, shooting mechanics. And it specifically
• 6:10
  demanded smooth collision detection. If you've
• 6:13
  ever tried to program a game, you know, collision
• 6:15
  detection is a nightmare. The math gets very
• 6:18
  messy. You aren't just telling a pixel to stop.
• 6:21
  You're constantly calculating the exact boundary
• 6:23
  of two moving objects on a grid at 60 frames
• 6:26
  per second. It also requested an enemy AI that
• 6:29
  actively patrols and chases the player. It specified
• 6:32
  the entire visual design. A symmetrical battlefield
• 6:35
  with two bases and a specific color scheme. And
• 6:38
  the model built it perfectly. It was fully playable
• 6:41
  on the very first try. The interface was remarkably
• 6:44
  clean. The mechanics worked flawlessly. The paintballs
• 6:47
  disappeared exactly upon hitting an obstacle.
• 6:50
  The score tracked correctly. The game instantly
• 6:53
  recognized when a flag was captured and returned.
• 6:55
  But the gameplay itself was apparently a little
• 6:58
  too challenging. The AI enemies were absolutely
• 7:00
  ruthless. They were basically terminators. They
• 7:03
  chased the player down immediately. So the author
• 7:05
  sent a second prompt. They asked the model to
• 7:07
  add a difficulty slider. They wanted easy, normal,
• 7:10
  and hard modes. They needed the AI to adjust
• 7:12
  enemy speed, accuracy, and reaction time dynamically.
• 7:16
  Now, building a fresh game is neat. It's a great
• 7:18
  parlor trick. But the real magic is that it successfully
• 7:21
  modified the existing game. Right, without breaking
• 7:24
  the core loop. Updating existing logic is the
• 7:26
  actual hard part. Exactly. Modifying state management
• 7:30
  is notoriously hard for AI. Usually, when you
• 7:33
  ask an AI to change a core mechanic, it hallucinates
• 7:36
  and shatters the entire codebase. But Gemini
• 7:39
  handled it flawlessly. Just two prompts for a
• 7:42
  working, highly adjustable game. The visual style
• 7:44
  was a bit basic, though. You'd need much more
• 7:47
  detailed prompts for specific artistic styles.
• 7:49
  But the underlying functionality was rock solid.
• 7:52
  The easy mode successfully slowed the enemies
• 7:54
  down. The hard mode made them hyper -aggressive
• 7:57
  again. The user interface updated correctly.
• 8:00
  It proves the model deeply understands state
• 8:02
  management. It can hold the entire logic of the
• 8:04
  game in its memory window simultaneously. And
• 8:07
  it applies surgical changes to specific functions.
• 8:09
  It doesn't just rewrite the whole file randomly.
• 8:12
  If vibe coding goes wrong, how much technical
• 8:15
  knowledge does a user actually need to fix it?
• 8:17
  Quite a bit. Honestly, if the AI hallucinates
• 8:21
  a broken function, a non -coder will just stare
• 8:24
  at gibberish. You still need fundamental logic
• 8:26
  skills. So fixing errors still requires a basic
• 8:29
  level of computational thinking. Absolutely.
• 8:32
  Now coding is highly logical and structured.
• 8:34
  Sifting through messy human news is a completely
• 8:37
  different cognitive task. It really is. So the
• 8:40
  author ran a research test to see how it handles
• 8:43
  ambiguity. They asked Gemini to act as a senior
• 8:46
  tech analyst. The goal was summarizing AI agent
• 8:49
  developments from the last 30 days. They specifically
• 8:52
  wanted updates from Google, OpenAI, and Anthropic.
• 8:56
  The prompt explicitly asked the model to prioritize
• 8:58
  facts over general AI hype. The output was a
• 9:04
  highly structured, incredibly dense report. It
• 9:07
  was beautiful to read. It found six major product
• 9:09
  updates. Each finding included specific announcement
• 9:12
  dates, practical use cases, and direct source
• 9:15
  links. The standout features were the timeline
• 9:18
  and a comparison table. The timeline arranged
• 9:21
  all the announcements chronologically. A timeline
• 9:23
  is crucial for humans. AI news moves so incredibly
• 9:27
  fast right now. Seeing the exact sequence of
• 9:29
  events provides massive clarity. You just don't
• 9:32
  get that from a bulleted list. The comparison
• 9:35
  table broke down the competing ecosystems. It
• 9:37
  compared their capabilities, their pricing structures,
• 9:41
  execution environments, and security protocols
• 9:43
  side by side. It completely separated the signal
• 9:46
  from the noise. It ignored the marketing fluff.
• 9:49
  And that is the true value here. It isn't just
• 9:52
  regurgitating search results. It's the synthesis.
• 9:56
  It's structuring raw data so a human can instantly
• 9:58
  make sense of it. Synthesis is much harder than
• 10:01
  search. Search is just fetching data. Synthesis
• 10:05
  requires judgment. It requires weighing the importance
• 10:08
  of conflicting data points against each other.
• 10:10
  But the author noted they would still double
• 10:12
  check the claims. You really cannot blindly trust
• 10:15
  an AI report yet, no matter how good it looks.
• 10:18
  My hallucination is always a lingering threat,
• 10:21
  especially with obscure sources or nuanced technical
• 10:24
  details. To combat that, the prompt was very
• 10:27
  strict. It demanded direct links to original
• 10:29
  sources. For every single major factual claim.
• 10:32
  And it told the model to state its limitations
• 10:34
  clearly. If you couldn't verify something, it
• 10:37
  was explicitly instructed to say so. Having that
• 10:40
  kind of meta -awareness is a huge step forward
• 10:42
  for AI research tools, knowing what it doesn't
• 10:45
  know. What is the actual risk of trusting AI
• 10:48
  to verify its own sources in research? The risk
• 10:51
  is massive confirmation bias. AI might invent
• 10:55
  a plausible sounding source, so if you don't
• 10:57
  manually click the links, you present fiction
• 11:00
  as fact. AI invents sources, so human verification
• 11:04
  remains completely essential. It really does.
• 11:07
  Now, let's take a quick break. Support for today's
• 11:10
  deep dive comes from our trusted partners. Please
• 11:12
  check the show notes for more information and
• 11:14
  special offers. OK, so taking the synthesis skills
• 11:17
  from the research test and the complex coding
• 11:20
  skills from the game test, the author decided
• 11:23
  to combine them. It was a massive final boss.
• 11:27
  Building a complex, highly interactive web app
• 11:30
  from scratch. They called it Pulse Analytics.
• 11:34
  It was a marketing dashboard mimicking a premium
• 11:36
  SaaS product. It used React, TypeScript, and
• 11:40
  Tailwind CSS. Now, if you've ever tried to build
• 11:43
  a data dashboard, you know the pain here. It
• 11:45
  wasn't just a simple static page. It included
• 11:48
  raw CSV data uploads. Using PapaParse. It featured
• 11:53
  interactive visualizations using recharts. You
• 11:56
  have to parse a raw file, map that messy data
• 11:59
  to a state manager, and feed it into a charting
• 12:01
  library. That's a huge headache for developers.
• 12:03
  But Gemini handled it. It calculated return on
• 12:06
  ad spend. It figured out cost per acquisition.
• 12:09
  It handled real messy marketing data schemas.
• 12:12
  The prompt demanded strict mathematical accuracy.
• 12:15
  It had to handle missing values safely. And division
• 12:17
  by zero. Which usually crashes simple web apps
• 12:19
  instantly. Gemini accounted for it automatically
• 12:22
  without being handheld. The result was stunning.
• 12:24
  You get this highly polished dark mode interface.
• 12:27
  complete with working date filters, sorting functions,
• 12:30
  and accurate calculations right out of the gate.
• 12:32
  It even provided a downloadable sample CSV template,
• 12:35
  so users knew exactly how to format their data.
• 12:37
  That is incredible attention to user experience.
• 12:40
  It processed realistic sample data perfectly.
• 12:42
  Wow. Imagine scaling to a billion queries. It
• 12:46
  is mind -bending. A single well -crafted prompt
• 12:49
  generated a functional tool. Something that would
• 12:52
  normally take a human developer days or even
• 12:54
  weeks to build perfectly. The campaign performance
• 12:57
  table looked fantastic. It calculated conversions,
• 13:00
  total revenue, and total spend instantly. Users
• 13:03
  could hover over specific data points in the
• 13:05
  line charts to see exact numbers. They could
• 13:07
  switch metrics on the fly. It behaved exactly
• 13:09
  like a paid software product. And it stored all
• 13:12
  the user preferences locally. It required no
• 13:14
  external API keys. Or expensive paid backend
• 13:17
  services to function. That's the real power of
• 13:20
  modern client -side web development. The user's
• 13:22
  browser does all the heavy lifting. The author
• 13:24
  did note they still need to test extreme edge
• 13:26
  cases. Like importing heavily corrupted CSV files
• 13:29
  just to see how it handles catastrophic data
• 13:32
  failure. But as a minimum viable product, it's
• 13:35
  shockingly complete. The visual hierarchy was
• 13:37
  clean and professional. It really demonstrates
• 13:40
  the raw power of combining complex logic, interface
• 13:43
  design, and data processing. Do tools like this
• 13:46
  replace data analysts or just give them a much
• 13:49
  faster starting point? They definitely don't
• 13:51
  replace analysts yet. You still need a human
• 13:54
  to decide which metrics actually matter and interpret
• 13:56
  the data. They provide a massive head start,
• 13:59
  but human interpretation is essential. Now, Jim
• 14:02
  and I proved it can build highly functional logical
• 14:04
  tools. But aesthetics and cost are a completely
• 14:07
  different battleground. How does it actually
• 14:09
  stack up against Claude Opus 5 .5 in a head -to
• 14:13
  -head creative mashup? It's a wild comparison.
• 14:16
  They ran something called the Gummy Bear Challenge.
• 14:19
  I absolutely love this test. It highlights their
• 14:21
  creative differences perfectly. Claude created
• 14:23
  these realistic, glossy, highly translucent gummy
• 14:27
  bears. It cost $7 .20 in API credits. And Jim
• 14:31
  and I... It created blocky, somewhat cartoonish
• 14:34
  bears. And it cost a shocking $22. That's three
• 14:38
  times the price for a noticeably worse visual
• 14:42
  image. But then they flipped the script and did
• 14:44
  the 3D ship in a bottle challenge. Right. Claude
• 14:47
  made a stunning, complex HTML scene. Dolphins
• 14:51
  swimming, seabirds flying, gorgeous translucent
• 14:53
  water effects. And that cost $12 .06. Gemini
• 14:57
  was much less polished. The graphics were rudimentary.
• 15:01
  But it only cost 19 cents. 19 cents. It's a bizarre,
• 15:06
  unpredictable trade -off. Claude is clearly the
• 15:09
  undisputed winner for visual polish and artistic
• 15:11
  consistency. Well Gemini is highly variable.
• 15:15
  Sometimes it's shockingly expensive for simple
• 15:17
  tasks. Sometimes it's incredibly cheap for complex
• 15:21
  ones, but it is always highly functional. The
• 15:25
  aesthetic gap is very real though. Claude just
• 15:28
  understands beauty better. It understands lighting,
• 15:31
  composition, and visual hierarchy natively. Gemini
• 15:34
  feels much more utilitarian. It gets the job
• 15:37
  done reliably, but it lacks that artistic flair.
• 15:41
  The API pricing model is also Pretty tricky right
• 15:44
  now. Google is offering a steep discount. Until
• 15:47
  November 30th, 2026. Right. Right now, it's $2
• 15:51
  per million input tokens. And $10 per million
• 15:55
  output tokens. They also charge $0 .10 per million
• 15:57
  cashed input tokens. Cashing saves developers
• 16:00
  a massive amount of money on repeated prompts.
• 16:03
  But on December 1st, those prices literally double
• 16:06
  overnight. $4 for input, $20 for output. That
• 16:10
  changes the math completely for startups. $20
• 16:13
  output fee is incredibly steep. Especially for
• 16:16
  large -scale consumer applications. And remember,
• 16:19
  output pricing includes thinking tokens. When
• 16:21
  the model takes extra time to reason through
• 16:23
  a problem before answering, you pay for that
• 16:25
  internal thought process. Which perfectly explains
• 16:27
  the $22 gummy bears. The model probably spun
• 16:30
  its wheels for ages thinking about the design
• 16:32
  logic, driving the cost up. How are developers
• 16:35
  supposed to budget for apps when token costs
• 16:37
  fluctuate this wildly between prompts? It is
• 16:40
  a massive headache. Developers will likely have
• 16:43
  to build hard caps and route simple queries to
• 16:45
  cheaper models. Developers must build hard limits
• 16:47
  and route queries to manage costs. Exactly. We've
• 16:50
  covered a huge amount of ground today. If we
• 16:53
  synthesize everything we just explored, the broader
• 16:56
  narrative is extremely clear. Google is officially
• 16:59
  back in the heavyweight fight. They are not messing
• 17:01
  around anymore. Gemini 4 Argon isn't just a laboratory
• 17:05
  benchmark champion, it's a deeply capable agentic
• 17:08
  coder. And a surprisingly nuanced researcher.
• 17:10
  It's a heavy lifter. It handles massive context
• 17:13
  windows with surprising grace, holding entire
• 17:16
  code bases in its memory. While Claude might
• 17:18
  still firmly hold the crown for artistic visual
• 17:21
  polish, Chimney is an absolute powerhouse. It
• 17:24
  really is. It excels at multi -step logic and
• 17:27
  functional app development. It effectively bridges
• 17:30
  the gap between generating raw code and building
• 17:33
  actual usable user interfaces. And those legal
• 17:37
  reasoning scores are a major signal we can't
• 17:39
  ignore. Major law firms are definitely paying
• 17:42
  attention right now. That brings us to the end
• 17:44
  of today's deep dive. But I want to leave you
• 17:46
  with a final thought to mull over. If models
• 17:50
  like Gimini 4 Argon can build custom interactive
• 17:52
  software on the fly for mere pennies, are we
• 17:56
  approaching a future where we stop downloading
• 17:58
  apps altogether? Will you just prompt your device
• 18:01
  to generate a temporary single use app for whatever
• 18:04
  highly specific task you need in that exact moment,
• 18:07
  and then delete it when you're done. It changes
• 18:08
  everything about how we interact with computers.
• 18:11
  Thank you for joining us on this deep dive. Take
• 18:14
  care of yourselves out there.