Latest / Neural Newscast / Microsoft MAI Models Beat Claude 4.6 in Blind Evals [Model Behavior]
Transcript
- [00:00] Announcer: From Neural Newscast, this is Model Behavior, [00:02] Announcer: AI-focused news and analysis on the models shaping our world. [00:12] Nina Park: Welcome to Model Behavior. This program examines how AI systems are built [00:17] Nina Park: and operated in professional environments. Today is June 8th, 2026, and we're witnessing [00:23] Nina Park: a consequential week for model infrastructure. Between Microsoft Build and Apple's WWDC, [00:30] Nina Park: the industry's largest players are shifting from third-party reliance to deeply integrated [00:36] Nina Park: in-house systems. This transition [00:38] Nina Park: marks a new era in the competition for foundational model control across the enterprise landscape. [00:45] Thatcher Collins: It is a massive strategic pivot, Nina. For years, the narrative focused on who could sign [00:50] Thatcher Collins: the biggest partnership deals. Now we are seeing the technical fallout as companies like Microsoft [00:56] Thatcher Collins: and Apple try to reclaim control over the model layer. We have a significant amount of data [01:02] Thatcher Collins: to unpack from the Microsoft Build announcement specifically, particularly regarding [01:06] Thatcher Collins: the new MAI family. It seems [01:08] Thatcher Collins: the reliance on external providers is finally reaching a breaking point for these tech giants. [01:13] Nina Park: Microsoft has officially unveiled its in-house MAI models. The standout is MAI Thinking One, [01:20] Nina Park: a reasoning model designed to cut costs and reduce dependency on partners like OpenAI and Anthropic. [01:26] Nina Park: According to Microsoft, in blind evaluations run by their partner Surg, [01:31] Nina Park: MAI Thinking One was preferred over Anthropic's Claude Sonnet 4.6. They are also claiming it matches [01:38] Nina Park: Claude Opus 4.6 on coding benchmarks, [01:41] Nina Park: which would be a significant technical achievement if it holds up under wider scrutiny. [01:46] Thatcher Collins: I have to stop you there on the Surg evaluations, Nina. We need to be careful [01:50] Thatcher Collins: with the phrase "preferred" in a blind evaluation. Microsoft commissioned this study, [01:55] Thatcher Collins: and while Surg is an independent partner, human preference evaluations are notoriously sensitive [02:00] Thatcher Collins: to the specific prompts and the task mix chosen for the test. Matching Oppus on coding is [02:06] Thatcher Collins: a bold claim, [02:07] Thatcher Collins: but we have to ask if this is actually translating to a better real-world developer experience. [02:12] Nina Park: That is the central question. Early reports are mixed. While Microsoft is pitching these [02:18] Nina Park: as frontier competitive, some third-party testing tells a different story. [02:23] Nina Park: PCMag tested the full suite, including Thinking One and Voice Two, and described them [02:29] Nina Park: as largely mediocre. Their reviewer Ruben Sorcele noted that Mei Thinking One lacks Internet access, [02:36] Nina Park: which is a significant handicap when you compare it to a connected model like CloudeSonnet 4.6. [02:42] Nina Park: Without that live data retrieval, its utility in dynamic environments is limited. [02:48] Thatcher Collins: The lack of Internet access is a major friction point for a reasoning model in 2026. [02:54] Thatcher Collins: " And the criticism didn't stop at the reasoning. The testing of MAI Voice II [02:58] Thatcher Collins: apparently landed it right in the uncanny valley, with reviewers describing it as robotic [03:03] Thatcher Collins: and breathy. In a market where we have highly realistic voices from several competitors, [03:08] Thatcher Collins: just being fine might not be [03:10] Thatcher Collins: enough to move enterprise customers away from their existing and more polished stacks, Nina? [03:15] Nina Park: True, Thatcher, but look at the strategic logic. Even if MAI thinking one is only good enough [03:22] Nina Park: for most tasks, it gives Microsoft massive leverage by offering a model that is cheaper to run [03:28] Nina Park: and owned outright inside their Foundry infrastructure. They are no longer hostage [03:33] Nina Park: to a partner's pricing or roadmap. It is essentially a dependency hedge dressed up [03:38] Nina Park: as a benchmark win, allowing them to provide a baseline level of intelligence at a much lower cost. [03:44] Thatcher Collins: That makes sense from a business standpoint. If you were a developer already using [03:48] Thatcher Collins: Microsoft's infrastructure, good enough and cheaper is a very compelling default win. [03:53] Thatcher Collins: But if the goal is to beat anthropic at the frontier, there is clearly a gap [03:58] Thatcher Collins: between the marketing benchmarks and the consumer perception of the product's actual polish. [04:02] Thatcher Collins: We should keep an eye on whether third-party replication confirms that Opus-level coding parity [04:07] Thatcher Collins: or if it falls short in complex scenarios. Moving to the other side of the big tech divide, [04:13] Thatcher Collins: Google has introduced a new primitive in the Gemini API called Managed Agents. [04:18] Thatcher Collins: This is a significant operational shift for developers building autonomous coding tools. [04:23] Thatcher Collins: Essentially, a single API call can now boot a 16 GB Linux virtual machine on Google's site [04:28] Thatcher Collins: to execute model-generated code. This allows for a much tighter integration [04:33] Thatcher Collins: between the model's reasoning and the actual execution environment. This is a direct shot [04:38] Thatcher Collins: at sandbox providers like E2B or Modal. Traditionally, if you wanted an LLM to run code safely, [04:44] Thatcher Collins: you had to manage the isolated environment yourself. Google is collapsing that entire stack. [04:50] Thatcher Collins: You call the interactions API and Gemini 3.5 Flash [04:54] Thatcher Collins: handles the reasoning while the system provisions a Linux sandbox to run Python or Bash scripts. [04:59] Nina Park: It simplifies [05:00] Thatcher Collins: the developer workflow by removing the need for third-party infrastructure for code execution. [05:05] Nina Park: They are using an agent called anti-gravity-preview-05-2026 which is the same harness powering [05:13] Nina Park: their anti-gravity IDE in early tests it was able to install libraries like pandas write a script [05:20] Nina Park: and return [05:20] Nina Park: the results in seconds during the current preview the virtual machine compute is essentially free [05:27] Thatcher Collins: you only pay for the Gemini 3.5 flash tokens which makes it an attractive option for developers [05:33] Thatcher Collins: looking to prototype agentic workflows without high overhead but that free compute comes with [05:39] Thatcher Collins: a caveat The one API call framing is a bit deceptive because of the token meter since the agent [05:46] Thatcher Collins: is doing both the reasoning and the environment management those context windows can fill up fast [05:52] Thatcher Collins: I'm curious Nina if we're seeing a trend where the infrastructure becomes the loss leader [05:57] Thatcher Collins: just to keep developers locked into the specific model ecosystem it is a classic platform play [06:03] Thatcher Collins: to ensure long-term adoption it certainly looks that way and speaking of Gemini Google [06:09] Thatcher Collins: is also nearing the June launch of Gemini 3.5 pro we're looking at [06:14] Thatcher Collins: a two million token context window and something they call deep think reasoning It seems [06:20] Thatcher Collins: they are trying to match the reasoning capabilities we're seeing in the My Thinking [06:24] Thatcher Collins: and Clawed Lines while maintaining that massive context advantage. They want to be the one-stop shop [06:30] Thatcher Collins: for both long context retrieval and complex logical reasoning. The competition [06:36] Thatcher Collins: for the reasoning label is getting crowded. It is also interesting to note [06:40] Thatcher Collins: that Anthropic is reportedly filing for an IPO. This move suggests they believe their current lead [06:46] Thatcher Collins: with Claude 4.6 is sustainable enough to go public, even as Microsoft and Google move to commoditize [06:52] Thatcher Collins: the reasoning layer. They are betting that their focus on safety and frontier performance [06:56] Thatcher Collins: will keep them ahead of the integrated platform offerings from the larger incumbents. [07:01] Thatcher Collins: Let's pivot to OpenAI. They haven't been quiet while Microsoft moves into their territory. [07:06] Thatcher Collins: On June 4th, they walled out Dreaming V3. This isn't just a model update; it's a memory revolution. [07:14] Thatcher Collins: ChatGPT now runs a background synthesis process after your conversation ends to catalog [07:19] Thatcher Collins: your preferences and project context. This allows for a much more personalized assistant [07:24] Thatcher Collins: that can anticipate [07:25] Thatcher Collins: your needs based on past interactions without requiring manual inputs every single time. [07:31] SPEAKER_03: The technical detail that caught my eye was the 5X reduction in compute cost [07:37] SPEAKER_03: for this memory architecture. By moving the synthesis to a background process rather than doing it [07:42] SPEAKER_03: live during the chat, they can offer this to free users. It changes the dynamic from a chatbot [07:48] SPEAKER_03: that takes notes to one that actually understands your long-term goals. [07:53] SPEAKER_03: It is a clever way [07:54] SPEAKER_03: to increase user retention by making the system more useful the more you interact with it. [07:58] Thatcher Collins: However, the privacy implications are raising eyebrows. A recent study found that 96% [08:04] Thatcher Collins: of chat GPT memories were created unilaterally by the system, not by explicit user request. [08:11] Thatcher Collins: With the EU AI Act transparency rules hitting in August, OpenAI might be heading toward [08:17] Thatcher Collins: a significant legal reckoning regarding how these behavioral profiles are built. [08:21] Thatcher Collins: They will need to prove [08:22] Thatcher Collins: that this automated synthesis respects user privacy and complies with the new regulatory standards. [08:28] SPEAKER_03: It's a delicate balance. Users want the assistant to remember that they finished their trip [08:33] SPEAKER_03: to Singapore, so it stops suggesting restaurants there, but they might not want the system [08:38] SPEAKER_03: silently building a detailed profile of their professional habits. It will be interesting to see [08:42] SPEAKER_03: if OpenAI adds more granular controls before those August regulations take effect, Nina. [08:48] SPEAKER_03: The tension between personalization [08:50] SPEAKER_03: and privacy is only going to increase as these systems become more autonomous. [08:55] Thatcher Collins: Finally, we have to mention Apple's WWDC 2026. They've introduced iOS 27 [09:02] Thatcher Collins: and the headline is clearly the Siri overhaul. Apple is finally moving Siri away from basic command [09:07] Thatcher Collins: and control toward a context-aware assistant powered by Apple Intelligence. [09:12] Thatcher Collins: They are integrating this across the entire ecosystem, including the Vision Pro. [09:17] Thatcher Collins: This represents a fundamental shift in how Apple views the role [09:20] Thatcher Collins: of AI within their hardware and software ecosystem, moving toward a more holistic integration. [09:27] SPEAKER_03: Apple is playing the long game with on-device AI. By making AI the foundation [09:31] SPEAKER_03: of the operating system, they are betting that privacy and ecosystem integration will beat [09:36] SPEAKER_03: the raw reasoning power of cloud-only models. It puts them [09:39] SPEAKER_03: in a three-way battle — Apple Intelligence vs. Gemini vs. ChatGPT. This week has shown us that none [09:46] SPEAKER_03: of these players [09:47] SPEAKER_03: is willing to concede the model layer to their rivals, Nina. It is an all-out race for dominance. [09:52] Nina Park: It is a fascinating shift from partnership to pure competition. Thank you for listening to [09:59] Nina Park: Model Behavior, a Neural Newscast editorial segment. You can find more of our reporting [10:05] Nina Park: and our AI transparency policy at mbnneuralnewscast.com Neural newscast [10:13] Nina Park: is AI-assisted, human-reviewed. View our AI transparency policy at neuralnewscast.com [10:20] Announcer: This has been model behavior on Neural Newscast. Examining the systems behind the story