Latest / AI Fire Daily / #547 Neil: Grok 4.5 Review Puts Premium Coding Models Under Pressure
Transcript
- 0:00You read the glossy press releases? Oh yeah.
- 0:02They always promise the world. But then you sit
- 0:05down at two in the morning, your coat is completely
- 0:07broken, you're exhausted. We have all been there.
- 0:09You don't want magic in that moment, you just
- 0:11want something that actually works. Welcome to
- 0:16the Deep Dive. I'm really glad you're here with
- 0:18us. Today, our mission is exploring your sources
- 0:21on XAI's newly announced Grok 4 .5. Thanks for
- 0:25having me. I've been digging through these numbers
- 0:27all week. It's an incredibly fascinating data
- 0:29set. It really is. So here is our roadmap. We're
- 0:32going to unpack its core specs first. Right.
- 0:34Then we'll decode the benchmark reality check.
- 0:37We'll explore the hidden economics of token efficiency.
- 0:40That part is super interesting. Yeah. Then we'll
- 0:42look at real world bug hunts and 3D app generation.
- 0:45Finally, we'll define exactly where Grok 4 .5
- 0:49hits a glass ceiling. We'll compare it to heavyweight
- 0:51rivals like Fable 5, GPT 5 .5, and Opus 4 .8.
- 0:56It's a packed journey today. We definitely have
- 0:57a lot of ground to cover. So Grok is cheap and
- 1:00efficient on paper. But let's look closely at
- 1:02this first segment. What exactly is Grok 4 .5
- 1:06under the hood? We need to start with the spec
- 1:08check. Yeah, let's dive into it. The sources
- 1:10say it was developed directly with Cursor. Cursor
- 1:13is a very popular AI -powered code editor. They
- 1:16built this model specifically for software engineering.
- 1:19They deeply optimized it for knowledge work and
- 1:22agent workflows. Agent workflows are multi -step
- 1:26tasks, and AI manages entirely on its own. Right.
- 1:29And that context is crucial here. It isn't just
- 1:32a generic chatbot in a browser. Yo. It's designed
- 1:35to act like a highly focused junior developer.
- 1:37It sits natively inside your actual coding environment.
- 1:40Yeah, that makes a big difference. Exactly. It
- 1:42isn't trying to write poetry or summarize novels.
- 1:44Yeah. It wants to write functions and fix your
- 1:47syntax errors. The technical specs are pretty
- 1:50staggering to think about. It has a 500 ,000
- 1:52token context window. Wow. Yeah. A context window
- 1:56is the amount of text the AI can remember at
- 1:58once. Half a million tokens is a massive cognitive
- 2:00leap. It fundamentally changes how you approach
- 2:03comp - Flexcoding projects. Well, you aren't
- 2:05just feeding it a single file anymore. You can
- 2:08feed it entire code repositories all at once.
- 2:10It really is mind -bending. It's like holding
- 2:12an entire library in your short -term memory
- 2:15without dropping a single book. That is a great
- 2:18way to put it. And the output speed is around
- 2:2080 tokens per second. Yeah. And that sheer speed
- 2:23is absolutely crucial. Because it keeps you moving.
- 2:26Right. 80 tokens a second is far faster than
- 2:29human reading speed. It feels almost instantaneous
- 2:32when you are working. a highly complex architectural
- 2:36question. The code just pours onto your screen
- 2:39like water. It keeps you in a state of deep flow.
- 2:42Let's look at the baseline performance numbers.
- 2:44XAI is claiming some very strong scores across
- 2:47the board. It hits 54 points on the Artificial
- 2:50Analysis Intelligence Index. That is a solid
- 2:53baseline for general reasoning. And it scores
- 2:5576 on the coding agent index. Those are highly
- 2:58respectable numbers for a specialized model.
- 3:00They really put Grok up there near the frontier
- 3:01models. But the pricing structure is where it
- 3:04gets genuinely interesting. Exactly. It costs
- 3:07just $2 per million input tokens and $6 per million
- 3:12output tokens. That is extremely aggressive pricing
- 3:16for this market. They're clearly targeting the
- 3:18daily grind of offer development. Because you
- 3:20don't always need the smartest model. Exactly.
- 3:22You don't always need a genius level AI for every
- 3:25task. Sometimes you just need a fast, heavily
- 3:28caffeinated, competent assistant. To sex silence.
- 3:32Which raises an important foundational question
- 3:35here. Does raw token speed actually translate
- 3:38to better problem -solving or just faster typing?
- 3:41It's really a mix of both, honestly. I mean,
- 3:43fast typing without logic just creates bad bugs
- 3:45much faster. Right. But Grok 4 .5 has the baseline
- 3:49intelligence to back it up. The unprecedented
- 3:51speed means you can iterate much faster, you
- 3:53can test a hypothesis, fail, and try again in
- 3:56seconds. So speed is useless if the underlying
- 3:59logic is flawed. Beat. Moving on. Exactly right.
- 4:02We need to look much closer at the actual performance
- 4:04data. We know the on -paper stacks are impressive,
- 4:06but how does it actually perform in sterile testing
- 4:09environments? And why does the test setup matter
- 4:12so much? Benchmarks are notoriously tricky in
- 4:15the AI space. They're basically just highly controlled
- 4:18laboratory experiments. Right. They don't always
- 4:21reflect the messy reality of production code.
- 4:23Let's look at DeepSW 1 .0. This is a tough repository
- 4:27-level software engineering test. Grok scored
- 4:3062 % on this benchmark. It was just behind Fable
- 4:33at 66 .1 % and just behind GPT 5 .5 at 64 .31%.
- 4:40That is a very tight cluster of top scores. Grok
- 4:44is hanging right there with the absolute heavyweights.
- 4:46It proves the foundational logic is fundamentally
- 4:48sound. But then we look at the SWE Marathon benchmark.
- 4:52Grok actually led the pack on that specific test.
- 4:55Yeah, that surprised some people. It had a solid
- 4:5729 % resolution rate. It outright beat Opus 4
- 5:00.8 in Fable. That is a critical data point for
- 5:03developers to understand. SWE Marathon tests
- 5:06much longer complex software tasks. Not just
- 5:08quick edits. Right. It isn't asking for quick
- 5:11isolated code edits. It demands sustained attention
- 5:13over a long period of time. I have to admit something
- 5:16here. I still wrestle with prompt drift myself
- 5:18when trying to manage long automated workflows.
- 5:21Oh, we all do. Prompt drift is absolutely brutal
- 5:24for productivity. It really is. And the AI starts
- 5:27strong, but slowly forgets the main goal over
- 5:30time. Grok leading the SWE Marathon suggests
- 5:33it fights off that drift quite well. It stays
- 5:35locked onto the original objective. It also scored
- 5:3883 .3 % on Terminal Bench 2 .1. That's almost
- 5:42level with the top tier premium models, and it
- 5:44scored 64 .7 % on SWE Bench Pro. SWE Bench Pro
- 5:49is a tough benchmark testing AI on real world
- 5:52software engineering tasks. but there is a cautionary
- 5:56tale hidden in the data. Ah, yes, the notorious
- 5:59testing harness issue. Exactly. On Debsys WE
- 6:021 .1, Datacurve used the mini -suite agent harness.
- 6:06A testing harness is the software framework delivering
- 6:08the test instructions to the AI. Right. Grok's
- 6:11score dropped dramatically to 53%. Yeah, a huge
- 6:14drop. Meanwhile, Fable hit 70%, GPT 5 .5 hit
- 6:1767%. This is exactly why you can't trust a single
- 6:21chart. So why does the testing harness change
- 6:23the score so drastically? It comes down to how
- 6:25the model expects to receive instructions. A
- 6:27harness is just the framework delivering the
- 6:29standardized test. If a model was trained on
- 6:32one specific prompt structure, changing the harness
- 6:35breaks its rhythm. Fable and GPT -5 .5 are highly
- 6:39generalized intelligence models. They adapt much
- 6:42better to weird unexpected testing environments.
- 6:45That makes a lot of sense. Let's talk about the
- 6:47underlying token economics now. This specific
- 6:50detail completely blew my mind. Yeah, this is
- 6:52wild. According to XAI, Grok 4 .5 averaged about
- 6:5615 ,900 output tokens per SWE bench protest.
- 7:01This shows incredible restraint. But Opus 4 .8
- 7:04used over 67 ,000 tokens for the exact same tasks.
- 7:08That is roughly 4 .2 times more tokens consumed.
- 7:11Why does Opus 4 .8 burn through four times as
- 7:14many tokens to solve the exact same problem?
- 7:16Opus naturally tends to think out loud. Oh, interesting.
- 7:19Yeah, it generates massive amounts of internal
- 7:21reasoning text before it writes the final code.
- 7:24GROG is a much more direct utilitarian model.
- 7:27Just skips the monologue and outputs the final
- 7:29solution. This saves you a massive fortune in
- 7:31API costs. over a long coding session. Basically,
- 7:34Grok gets straight to the point instead of rambling.
- 7:37On to real projects. Yes, exactly. Theoretical
- 7:40benchmarks only take our understanding so far.
- 7:43Benchmarks are just theory. Let's look at how
- 7:45Grok 4 .5 performs when the gloves finally come
- 7:48off. We have fascinating data from actual messy
- 7:51development environments. This is where we see
- 7:53its true colors emerge. The sources mention extensive
- 7:57automated code audits. Grok successfully reviewed
- 8:00massive code bases for hidden security risks.
- 8:02Right. It also handled quick GitHub pipeline
- 8:05reviews. A GitHub pipeline is automated scripts
- 8:08that test and deploy your software code. Though
- 8:11we should note a very important limitation here.
- 8:13What's that? The sources don't prove it can handle
- 8:15complex pull request comments entirely on its
- 8:17own yet. A pull request is a formal request to
- 8:20merge new code into a project. That's a fair
- 8:22point to keep in mind, but the autonomous bug
- 8:24hunting is truly wild. In one personal test,
- 8:27a developer gave Grok a repository with 45 hidden
- 8:30bugs. That is a total nightmare scenario for
- 8:32any human developer. Grok successfully fixed
- 8:34five of them and found two more. It finished
- 8:37the entire process in 16 minutes and 30 seconds.
- 8:40Wow. The total compute cost was just $5 .82.
- 8:44Think about the staggering economics of that
- 8:46result. Yeah. Hiring a human engineer to hunt
- 8:49down seven obscure bugs would take hours. It
- 8:52would easily cost hundreds of dollars in labor.
- 8:56Grok did it for the price of a fancy coffee.
- 8:58Here is where it gets really interesting for
- 9:00visual creators. 3D app generation. Oh, this
- 9:03is a fantastic eye -opening example. One test
- 9:06asked Grok 4 .5 to rebuild a 3 .js 3D globe dashboard.
- 9:123 .js is a popular library used to create 3D
- 9:14web graphics. The user provide only one prompt
- 9:17and one reference image. Just a single image,
- 9:20no other context at all. Two sec silence. Whoa.
- 9:24Imagine watching a complex 3D glass globe render
- 9:27perfectly on the first try from just one image.
- 9:30It's visually stunning to watch it happen. The
- 9:32dynamic lighting matched perfectly, the intricate
- 9:34glass panels matched. Unbelievable. Yeah, the
- 9:36depth and spacing were almost completely identical
- 9:38to the reference image. Another prototyping text
- 9:40was even crazier. They used Grok 4 .5 to build
- 9:43a playable first -person shooter game. They did
- 9:46it in Grok building under an hour? They just
- 9:48ask for a basic game design document and use
- 9:51free web assets. It's an absolute prototyping
- 9:53monster. But we do need a serious reality check
- 9:58here. Right. Prototyping is great, but how dangerous
- 10:01is it to push that AI -generated game code straight
- 10:04to production without human review? It is incredibly
- 10:07dangerous to be blunt. These models write highly
- 10:10confident -sounding code. But confident doesn't
- 10:13necessarily mean secure. It doesn't mean the
- 10:16database is scalable. Right. You still have to
- 10:18test the gameplay mechanics. You have to carefully
- 10:20review the software architecture. You have to
- 10:22ensure it won't crash under a heavy user load.
- 10:25Right. Prototypes still need human oversight
- 10:26before going live. Beat sponsor. Welcome back
- 10:30to the Deep Dive. We're exploring the ground
- 10:32level realities of XAI's Grok 4 .5. We've seen
- 10:36its impressive speed, we've seen its prototyping
- 10:38chops, but no AI model is perfectly flawless.
- 10:41Far from it. They all have distinct blind spots.
- 10:43Let's explore exactly where Grok 4 .5 needs adult
- 10:47supervision. It really comes down to handling
- 10:49complex orchestration. The sources highlight
- 10:52a specific shared workflow developed internally
- 10:55at X. Yeah, this is a great case study. In this
- 10:57workflow, Grok 4 .5 handles all the daily coding
- 11:01and debugging. But it heavily relies on Fable
- 11:045 for high -level planning and orchestration,
- 11:07and it relies on GPKI 5 .6 Sockhall for the hardest,
- 11:10most complex coding work. This represents a profound
- 11:13shift in how we actually use AI. It clearly struggles
- 11:17with deep thoroughness, right? Yeah. Grok is
- 11:19great for incredibly fast implementation. But
- 11:23Fable 5 is needed to review potential weak points.
- 11:27Fable actively looks for missed edge cases. It
- 11:29applies careful, systemic reasoning to the whole
- 11:32project. Exactly. Grok is a brilliant sprinter.
- 11:35Fable is a patient chess player. Beat, listening
- 11:37to you describe this workflow, it almost sounds
- 11:39like managing a construction crew. Oh, definitely.
- 11:41Am I off base here, or is Grok essentially the
- 11:44incredibly fast brick layer? It throws up walls
- 11:46at lightning speed. But Fable 5 is the architect
- 11:49who makes sure the wall is actually straight.
- 11:51That is a perfect, highly accurate analogy. You
- 11:54definitely don't want the brick layer designing
- 11:56the foundation blueprints. And you really don't
- 11:59want the master architect laying every single
- 12:02brick. It's a massive waste of their highly specific
- 12:05talents. Does this mean the dream of one single
- 12:08god model doing everything is dead for now? I
- 12:12strongly believe so, yes. The entire industry
- 12:14is rapidly moving toward multi -agent systems.
- 12:18It's simply much more efficient to use specialized
- 12:21models for specialized tasks. One massive generalized
- 12:24model trying to do everything is just too slow
- 12:26and too expensive. Specialization is clearly
- 12:29winning over generalization. That brings us to
- 12:31our final showdown. The underlying economics
- 12:34of these multi -agent workflows. Understanding
- 12:36Grok's limitations leads us somewhere deeply
- 12:39important. It shows us exactly how smart developers
- 12:42are splitting the workload today. Right. They
- 12:44expertly balance Grok with more expensive premium
- 12:46models. Let's break down the specific model matchups.
- 12:49First, we have Grok versus Fable 5. Fable is
- 12:52primarily used for high -level design. It provides
- 12:55clear architectural direction and connects sequential
- 12:57steps. Grok happily takes over once requirements
- 13:00are crystal clear and actual implementation begins.
- 13:03They're highly complementary tools. They aren't
- 13:05direct zero -sum replacements for each other.
- 13:07Then we have Grok versus GPT 5 .5 and Opus 4
- 13:11.8. GPT 5 .5 is still vastly better for deep
- 13:15systemic reasoning. It is much better at careful,
- 13:19nuanced code review. But the financial cost factor
- 13:22changes the equation entirely here. Right. One
- 13:24direct comparison showed an average task cost
- 13:26of just 31 cents for Grok 4 .5. But Opus 4 .8
- 13:30Max cost a massive $1 .80 for the exact same
- 13:35task. That is nearly six times more expensive
- 13:37for the same output. But let me push back on
- 13:40this pure cost -saving narrative for a second.
- 13:42Sure. Is the 31 -cent task really cheaper in
- 13:45the grand scheme? What if you have to spend your
- 13:47own cognitive energy double -checking Grok's
- 13:49work? It inherently lacks GPT 5 .5's deep architectural
- 13:53reasoning. That is the hidden, often -ignored
- 13:56tax of cheaper models. Exactly. Your engineering
- 13:59time is actual money. If Grok saves you a dollar
- 14:01but costs you 20 minutes of frustrating debugging,
- 14:03you absolutely lost money on that exchange. Where
- 14:06is the tipping point where cheap API costs are
- 14:09outweighed by human debugging time? It entirely
- 14:11depends on task ambiguity. If the task is perfectly
- 14:15defined with clear boundaries, Grok wins easily.
- 14:18Right. If the task is highly vague and requires
- 14:21major assumption making, human debugging time
- 14:24will quickly destroy any API savings. You absolutely
- 14:28need the premium reasoning models there. Use
- 14:30Grok for clear blueprints and premium models
- 14:33for ambiguous problems. Precisely. That is the
- 14:36winning strategy for modern developers. We've
- 14:38covered a tremendous amount of ground today.
- 14:40It's time for our big idea recap. If we pull
- 14:43all these intricate threads together, a very
- 14:46clear picture emerges. Grok 4 .5 fundamentally
- 14:50changes the daily software coding grind. It offers
- 14:53an unprecedented, highly disruptive balance of
- 14:56pure speed and cost efficiency. It isn't trying
- 14:59to out -think Fable 5. It isn't trying to beat
- 15:01Opus 4 .8 on complex orchestration. Instead,
- 15:05it completely dominates the actual implementation
- 15:07phase. It dominates rapid prototyping once the
- 15:10instructions are crystal clear. It is the ultimate,
- 15:13highly efficient blue -collar worker of the AI
- 15:15world. I really love that specific framing. It
- 15:18is the ultimate blue -collar AI worker. Thank
- 15:20you for joining us on this deep dive today. I
- 15:22highly encourage you to test out these multi
- 15:24-model workflows in your own projects. See firsthand
- 15:27how the bricklayer and the architect work together.
- 15:29It's a total game changer for daily coding productivity.
- 15:32I want to leave you with one final lingering
- 15:34thought to mull over. If AI models are rapidly
- 15:38specializing into distinct roles planners, coders,
- 15:41and reviewers, how long until your main job as
- 15:43a developer is no longer writing code but acting
- 15:45as the HR manager for a team of specialized AI
- 15:48agents? Out TRO music.