Latest / Neural Newscast / Google Gemini 3.7 Flash Benchmarks and the Pro Delay [Model Behavior]
Transcript
- [00:00] Thatcher Collins: From Neural Newscast, this is Model Behavior, AI-focused news and analysis on the models [00:05] Thatcher Collins: shaping our world. [00:11] Nina Park: I'm Nina Park. [00:13] Nina Park: Welcome to Model Behavior. [00:15] Nina Park: This program examines how artificial intelligence systems are built, deployed, and operated [00:21] Nina Park: in real professional environments. [00:24] Nina Park: We look beyond the headlines to understand the underlying infrastructure and the actual [00:29] Nina Park: impact on industry workflows. [00:31] Nina Park: We'll see you next time. [00:31] Thatcher Collins: I am Thatcher Collins. Today is Monday, August 17th, 2026. [00:37] Thatcher Collins: Just a few days ago, on August 13th, Google released Gemini 3.7 Flash. [00:43] Thatcher Collins: This release marks the third significant update to the Flash model family in under three months, [00:48] Thatcher Collins: showing a clear shift in how Google is prioritizing their production release train model. [00:54] Nina Park: According to reporting from Coding Beauty, the 3.7 flash model has reached a score of 56 on the Artificial Intelligence Intelligence Index. [01:03] Nina Park: What stands out here is the operational speed. It is generating roughly 340.1 tokens per second. [01:11] Nina Park: Thatcher, help us contextualize this performance relative to the previous flash versions we saw earlier this summer. [01:18] Thatcher Collins: It is a substantial jump, Nina. [01:20] Thatcher Collins: To look at the progression, Gemini 3.5 Flash launched in May with an intelligence score of 52 and 172 tokens per second. [01:30] Thatcher Collins: The 3.6 update in July increased speed to 225 tokens, but kept the intelligence flat. [01:37] Thatcher Collins: This 3.7 release is the first to actually push both metrics significantly higher, even jumping 11 places in the AI arena leaderboard. [01:46] Nina Park: The benchmarking for software engineering is where the gap between these versions becomes most apparent. [01:52] Nina Park: Gemini 3.7 Flash achieved 65.3% on Deep SWE version 1.1. [01:59] Thatcher Collins: That is a dramatic increase from Gemini 3.6 Flash's 48.6%. [02:04] Thatcher Collins: It places the model ahead of Claude Sonnet 5, which scored just under 50%, [02:09] Thatcher Collins: and GPT 5.6 Terra, which is estimated to be in the low 50s. [02:14] Thatcher Collins: That focus on the workhorse model is interesting, Nina, but there is a missing piece in this narrative. [02:20] Thatcher Collins: Tort's AI points out that when Gemini 3.5 Flash launched in May, Google promised the Pro version would arrive next month. [02:28] Thatcher Collins: We are now in mid-August, and we've seen three Flash updates while that supposed flagship remains absent from the public product line. [02:35] Nina Park: It suggests a focus on the models Google can reliably put in front of millions of users right now. [02:42] Thatcher Collins: As Casper Bannick noted, Flash has become a production release train. [02:46] Thatcher Collins: If the flagship Pro model is still a moving target, Google is clearly choosing to sharpen the efficiency of the mid-tier. [02:53] Thatcher Collins: The introductory cost of $0.40 per intelligence index task makes it highly attractive for scaling. [03:00] Thatcher Collins: Efficiency is one thing, Nina, but is the reasoning actually better? [03:05] Thatcher Collins: The technical reports describe something called diligent thinking. [03:09] Thatcher Collins: Instead of immediately generating a response, the model spends more time planning and coordinating tools. [03:15] Thatcher Collins: It is an interesting move for a flash model, which is usually marketed for speed. [03:20] Thatcher Collins: They're trying to bridge the gap between low latency and sophisticated multi-step workflows. [03:26] Thatcher Collins: This forcing function for developers is quite real, Thatcher. [03:31] Thatcher Collins: Do you wait for a flagship name like Gemini Pro that lacks a clear release date? [03:35] Thatcher Collins: Or do you build on a model that is finishing multi-step workflows up to 40% faster today? [03:41] Thatcher Collins: The data suggests the latter is the more pragmatic path for enterprise deployment, [03:46] Thatcher Collins: where time per intelligence is becoming a critical metric. [03:50] Thatcher Collins: Exactly. When you look at the progression from 172 tokens per second in May to 340 now, the engineering gains are tangible, even if the flagship is stalled. [04:01] Thatcher Collins: It highlights a shift from chasing raw parameter counts to optimizing the intelligence per dollar metric for agentic systems. [04:08] Thatcher Collins: Nina, it seems the FLASH designation is becoming the primary testbed for their most frequent and impactful updates. [04:15] Thatcher Collins: I'm Nina Park. [04:17] Thatcher Collins: Thank you for listening to Model Behavior, [04:19] Thatcher Collins: a Neural Newscast editorial segment. [04:21] Thatcher Collins: mb.neuralnewscast.com [04:24] Thatcher Collins: Neural Newscast is AI-assisted, human-reviewed. [04:27] Thatcher Collins: View our AI transparency policy at neuralnewscast.com. [04:31] Announcer: This has been Model Behavior on Neural Newscast. [04:34] Announcer: Examining the systems behind the story.