Latest / AI Fire Daily / #507 Neil: GLM-5.2 Vs GPT-5.5 Vs Claude Opus 4.8 Coding Test
Transcript
- 0:00You hand an AI your tangled, broken code base.
- 0:04You stare at the blinking cursor in silence.
- 0:07Beat. Trusting a machine with that mess is, it
- 0:11is terrifying. It really is like a leap of faith.
- 0:14Yeah. You're wondering if it will actually fix
- 0:16the architecture or if it's just going to hallucinate
- 0:18a completely new disaster. It is a genuine moment
- 0:21of developer anxiety. Absolutely. I mean, we
- 0:24are essentially handing over the keys to our
- 0:27digital infrastructure. you are asking a statistical
- 0:29model to understand human intent. Right. And
- 0:32today on the deep dive, we are looking at the
- 0:34three models currently asking us to take that
- 0:36leap. Because we are in mid -2026 now, and the
- 0:40AI coding debate has essentially narrowed down
- 0:42to three titans taking up all the oxygen. We
- 0:45have got GLM 5 .2, Claude Opus 4 .8, and GPT
- 0:495 .5. Those three are... absolutely dominating
- 0:52the conversation right now. Every engineering
- 0:54team is having heated arguments about which one
- 0:56is actually the most capable. Okay, let's unpack
- 0:58this. We are going to look at what the rigid
- 1:00benchmarks actually say. We will uncover the
- 1:03real financial cost hiding behind supposedly
- 1:05cheap tokens. And most importantly, we're going
- 1:09to look at the results of two brutal real -world
- 1:12tests. The side -by -side ones. Exactly. A complex
- 1:15interactive game build and a massive frustrating
- 1:18bug hunt. We really have to look past the marketing
- 1:21hype and the pristine release notes. We need
- 1:23to understand how these models behave when things
- 1:26get messy. Exactly. But, you know, the hype cycle
- 1:29always starts with the headline numbers. People
- 1:31love a clean, simple metric. Oh, they do. We
- 1:34see a high score on paper, and it looks incredibly
- 1:37convincing. It makes a complex neural network
- 1:39feel quantifiable. It is human nature. We want
- 1:42a leaderboard. Yeah. But a high score on a standardized
- 1:44test does not guarantee real -world success.
- 1:48The mechanisms behind those tests are, well,
- 1:51they're often flawed. Right, which brings us
- 1:53to the DeepSW benchmark. Yes. DeepSWE is a test
- 1:56mirroring actual bugs in real code bases without
- 2:00internet access. It is designed to be rigorous.
- 2:02The isolation is the key factor there. The model
- 2:05runs inside a restricted container. It has to
- 2:08ingest the issue. locate the faulty file, and
- 2:11write the patch completely on its own. No human
- 2:14developer can guide it. The official stats for
- 2:16that test are pretty striking. GLM 5 .2 scored
- 2:2046 .2 % on DeepSWE. Wow. Yeah. That is currently
- 2:25the top result for any open -weight model. It
- 2:27is an impressive milestone for the open source
- 2:29community, but the landscape gets more complicated
- 2:32with long -horizon tasks. You mean tasks requiring
- 2:35sustained logic over multiple steps. Exactly.
- 2:38Frontier SW benchmark comes in. On those long
- 2:41horizon tasks, GLM hit 74 .4%. That puts it just
- 2:46behind Claude Opus 4 .8, which hit 75 .1%. And
- 2:50surprisingly, GLM actually edged out GPT 5 .5,
- 2:54which sat at 72 .6%. On paper, those margins
- 2:56are incredibly thin. You're looking at a statistical
- 2:59tie. It implies the open model has completely
- 3:01caught up. But I look at those numbers, and I
- 3:03think about my own projects. If the open model
- 3:05is supposedly tied with GPT, why does it lose
- 3:07the plot? with my code. Yeah. Why do models that
- 3:10ace these tests suddenly fail on messy projects?
- 3:13Because benchmarks are fundamentally controlled
- 3:15environments. They're tidy, sanitized sample
- 3:18problems. The AI knows exactly what the grading
- 3:21script wants, but real -world coding requires
- 3:26adapting to undocumented, unpredictable chaos.
- 3:30It is like watching a national spelling bee.
- 3:33These kids memorize thousands of incredibly complex
- 3:36root words. It is highly impressive under pressure.
- 3:39But spilling the word chrysanthemum perfectly
- 3:41does not mean you can write a compelling novel.
- 3:44Right. And a novel requires handling messy, unpredictable
- 3:47human elements. Real software projects are exactly
- 3:50like that. Yeah, they are. You have vague feature
- 3:52requests from product managers. You have strange
- 3:56legacy code written by a developer who quit three
- 3:58years ago. Oh, the worst. And you have comments
- 4:00that actively contradict what the function actually
- 4:03does. A benchmark never tests for human unpredictability.
- 4:06It can't. The AI simply cannot handle the...
- 4:09ambiguity. It was trained to find the mathematically
- 4:11optimal path, not the context -dependent path.
- 4:14So real life has no screw to follow. Since benchmark
- 4:17scores are largely an illusion, the next thing
- 4:20everyone looks at is the price. If they all score
- 4:23roughly the same on paper, we just want the cheapest
- 4:25option. But the sticker price hides massive practical
- 4:29hurdles. The pricing structures of these APIs
- 4:31are notoriously difficult to map to real -world
- 4:34budgets. You have to scrutinize the input costs
- 4:37versus the output costs. The token pricing per
- 4:40million is wild right now. GLM is $1 .40 for
- 4:45input and $4 .40 for output. Claude Opus jumps
- 4:49up to $5 for input and $25. for output, and GBT
- 4:535 .5 is $5 for input with a massive $30 for output.
- 4:57The gap there is staggering. GLM is roughly three
- 5:00and a half times cheaper on the input side. Yeah.
- 5:03And it is six to seven times cheaper when generating
- 5:05output. Mathematically, it feels like a no -brainer,
- 5:08but I know from painful experience it is not
- 5:10that simple. It rarely is. I still wrestle with
- 5:13prompt drift myself. Yeah. I will start with
- 5:15a very clear, specific instruction to save money
- 5:18on a cheaper API. But 10 prompts later, the AI
- 5:23has completely forgotten the core architecture.
- 5:26It is just writing broken spaghetti code. This
- 5:29is a classic example of the cheap token trap.
- 5:32A cheaper model often has a weaker attention
- 5:35mechanism over long contexts. It makes subtle
- 5:39logical mistakes. You have to constantly correct
- 5:41it. And every time you correct it, you are resending
- 5:44the entire context window. Precisely. If a cheap
- 5:47model needs 10 times the tokens to actually finish
- 5:49a job correctly, those massive savings evaporate.
- 5:52You burn through your budget just trying to steer
- 5:54the model back on track. Exactly. Which brings
- 5:56up a really confusing aspect of this entire landscape.
- 6:00GLM 5 .2 is an open -weight model. It has an
- 6:03MIT license. Right. If it's free to download,
- 6:06why are we still paying for it? Because open
- 6:08does not mean lightweight. We're talking about
- 6:10a model with hundreds of billions of parameters.
- 6:13Those parameters are essentially the complex
- 6:15mathematical weights that make up the neural
- 6:17pathways. You physically cannot load that into
- 6:19the local memory of a standard machine. Not even
- 6:22close. To run a model of that scale at an acceptable
- 6:26speed, you need a massive dedicated GPU cluster.
- 6:30Wow. We are talking about data center level hardware.
- 6:34So most independent developers still have to
- 6:36use an API service like OpenRotter to access
- 6:38it. You are paying for the massive infrastructure,
- 6:41not the intellectual property. Exactly. The licensing
- 6:44is free, but the electricity, the cooling systems,
- 6:47and the silicon certainly are not. Open means
- 6:50accessible data, not free hardware. That is the
- 6:54hard truth of where the technology is right now.
- 6:56You can't run these titans locally and you can't
- 6:59trust the benchmarks or the raw sticker price.
- 7:01You are left in the dark. Yeah, you really are.
- 7:04The only truth lies in high effort side -by -side
- 7:07server testing. You have to strip away the marketing
- 7:09material completely. The only way to know how
- 7:11they behave is a head -to -head match -up. Right.
- 7:14Here is how you set up a rigorous real -world
- 7:16test. You give all three models the exact same
- 7:18prompt. You ask them to generate a step -by -step
- 7:21plan first before writing any code. And you force
- 7:24them into their absolute highest effort modes.
- 7:27That means utilizing codecs for GPT. You use
- 7:31open code via OpenReader for GLM. And you use
- 7:34Claude code for Opus 4 .8. You are forcing the
- 7:38models to dedicate maximum compute to the problem.
- 7:41Exactly. But we need to clarify a huge misconception
- 7:44about how that compute actually works. A lot
- 7:46of people buy highly expensive laptops thinking
- 7:49it will make their AI coding assistance faster.
- 7:51It is a very common misunderstanding of the client
- 7:54server architecture. Your high -end laptop's
- 7:57RAM does not speed up the model's analytical
- 7:59thinking whatsoever. Right. Your machine is basically
- 8:02just a dumb terminal requesting data. The heavy
- 8:05lifting is happening hundreds of miles away on
- 8:07the provider server. Your local machine only
- 8:09matters for the local preview rendering. Right.
- 8:11If you ask the AI to build a WebGL demo or a
- 8:14complex 3D game, your laptop needs the power
- 8:17to render those pixels smoothly. Makes sense.
- 8:19If your local GPU is weak, the preview will lag.
- 8:22But the AI itself isn't slowing down. And when
- 8:24you use these high effort modes, you are going
- 8:26to wait. One public test showed GPT Pro responding
- 8:29in about 10 minutes. 10 minutes for a single
- 8:32architectural response. is a lifetime in traditional
- 8:35development. It is. But another test took close
- 8:38to 60 minutes for a full complex game build.
- 8:41Yeah. Whoa. Two -second silence. Imagine it processing
- 8:45hundreds of billions of parameters just to find
- 8:48one missing semicolon. The scale of computation
- 8:51is genuinely breathtaking. We are routinely spinning
- 8:54up supercomputers to do the digital equivalent
- 8:57of proofreading. So buying a faster laptop won't
- 9:00make the AI code any faster. No. The server processes
- 9:04the logic. Your laptop just renders the resulting
- 9:06pixels on the screen. Servers do the thinking.
- 9:09Your laptop just displays the result. Precisely.
- 9:12Oh, okay. So you take all that massive server
- 9:15power we just talked about and we apply it to
- 9:17creation. Right. Generating a function is one
- 9:19thing. But turning a single text prompt into
- 9:21a fully interactive game is something else entirely.
- 9:24It is the ultimate test of synthesis. You are
- 9:27asking the model to build an interactive experience
- 9:29of from nothing but language. Let's break down
- 9:32Connor Dart's public test on X. He gave the models
- 9:36one single prompt and asked them to build a working
- 9:39marble puzzle game. Video games are a phenomenal
- 9:42stress test for AI. Because they require multiple
- 9:46heavily interconnected systems running at the
- 9:48exact same time. You need a physics engine, you
- 9:51need visual rendering, you need user input controls,
- 9:55and you need core win state logic. It all has
- 9:58to happen simultaneously at 60 frames per second
- 10:00without crashing. It forces the model to juggle
- 10:03parallel logic tracks. If the physics update
- 10:05loop conflicts with the rendering loop, the whole
- 10:08thing falls apart. The results of this game build
- 10:10were incredibly telling. GPT structured the code
- 10:13beautifully, is mathematically sound and highly
- 10:16organized, but the game itself lacked a playable
- 10:18feel. It was technically correct, but experientially
- 10:22rigid. Right. Now, GLM was honestly impressive
- 10:25for an open model. It managed to build a functioning
- 10:27game, but those token costs we warned about really
- 10:30piled up. It required a massive amount of trial
- 10:33and error and reprompting to get the physics
- 10:35right. Which leaves us with the clear winner
- 10:37of the interaction. test. Claude Opus 4 .8 won
- 10:41this hands down. It was the safest choice for
- 10:44smooth controls, the gameplay was stable, and
- 10:47the physics actually felt right to the human
- 10:49hand. This points to a fascinating divergence
- 10:52in how these models are being trained and optimized.
- 10:55Here's where it gets really interesting. GPT
- 10:58is like a highly technical, uncompromising architect.
- 11:02It builds a mathematically perfect, structurally
- 11:04sound house where all the angles are exactly
- 11:0790 degrees. But Claude is the interior designer.
- 11:11It actually makes the house comfortable to live
- 11:13in. That is a great way to visualize it. Claude
- 11:15focuses heavily on human -computer interaction.
- 11:18But I don't understand how a massive matrix of
- 11:20numbers even knows what smooth gameplay feels
- 11:23like. It doesn't have hands. True. It doesn't
- 11:25play games. Why does Claude excel specifically
- 11:28at the interactive parts? It comes under the
- 11:30underlying training mechanisms. Claude appears
- 11:33to be heavily tuned via reinforcement learning
- 11:35from human feedback. Exactly. And specifically
- 11:38tailored toward user experience. Its training
- 11:41data likely prioritizes front -end stability,
- 11:43React components, and user interface frameworks.
- 11:47It learned the statistical patterns of what humans
- 11:50rate as a good interface. Claude prioritizes
- 11:53how the code actually feels to a human. And that
- 11:57makes it an exceptional partner for front -end
- 11:59developers. But creating from scratch on a blank
- 12:02canvas is only one type of skill. Navigating
- 12:05someone else's messy, undocumented logic requires
- 12:08an entirely different type of intelligence. Reading
- 12:11code is notoriously much harder than writing
- 12:13it. It is the most agonizing part of software
- 12:15engineering. We are looking at Pawar Heran's
- 12:17test next. This was a brutal 21 bug hunt. He
- 12:21planted 21 highly complex hidden bugs inside
- 12:25a real existing code base. This is the perfect
- 12:27real world scenario. You aren't building a shiny
- 12:30new toy. You're acting as a mechanic trying to
- 12:32fix a broken machine without a manual. They tested
- 12:35the model's ability to find these bugs, specifically
- 12:37measuring the impact of giving them maximum reasoning
- 12:40time. Claude Opus 4 .8 was a solid code reviewer.
- 12:44When given more time to think, it actually improved
- 12:47significantly. It used that extra compute time
- 12:51to trace variables carefully through different
- 12:53files. But GLM 5 .2 stayed completely flat. Turning
- 12:57on higher reasoning did not help it catch a single
- 13:00extra bug. It just hit an immediate plateau.
- 13:03This exposes a critical architectural weakness
- 13:06in current open -weight models. GPT 5 .5, however,
- 13:09absolutely dominated this bug hunt. Mac's reasoning
- 13:12made a massive difference. It systematically
- 13:15hunted down the errors, making it the clear undisputed
- 13:19winner for deep debugging. This highlights a
- 13:21deep philosophical difference between generating
- 13:23code and debugging code. When you generate new
- 13:26code, you control all the variables. You are
- 13:28the master of the logic you are creating. But
- 13:31debugging isn't creation, it is forensic accounting.
- 13:33You have to track down someone else's embezzlement
- 13:36of memory. You have to extract intent from someone
- 13:39else's chaos. Exactly, because you are looking
- 13:42for the logic that is missing. Or the logic that
- 13:46silently contradicts itself. GBT clearly possesses
- 13:50a superior analytical engine for tracing those
- 13:53long convoluted threads of state and memory.
- 13:55Why did the open model hit a wall when given
- 13:58more time to think? Why couldn't GLM use that
- 14:01extra time to trace the variables like Claude
- 14:03did? It comes down to reasoning ceilings. Giving
- 14:06a model more time only helps if it has the architectural
- 14:09depth to form new analytical pathways. Right.
- 14:12If the model's core reasoning engine is shallow,
- 14:15extra time just means it repeats the exact same
- 14:17flawed analysis in a longer loop. It doesn't
- 14:19magically develop deeper comprehension. GLM hits
- 14:22a strict reasoning limit no matter the time given.
- 14:25It is a fundamental limitation that open source
- 14:27researchers are still desperately trying to solve.
- 14:30So what does this all mean for you, the developer,
- 14:32sitting at your keyboard today? Let's synthesize
- 14:34the ultimate takeaway from all these conflicting
- 14:36metrics and tests. The biggest realization is
- 14:39that there is no universal winner. The idea of
- 14:43one model to rule them all is dead. You have
- 14:45to match the specific model to your specific
- 14:48workflow. If you are building interactive UIs,
- 14:51if you are rapid prototyping games or throwing
- 14:53together visual demos, you should be using...
- 14:56Claude Opus 4 .8. Definitely. It intrinsically
- 14:58understands human feel, smooth interaction, and
- 15:02front -end stability. It is a clear, unrivaled
- 15:05choice for any user -facing application. But
- 15:07if you are doing deep logic work, if you are
- 15:10reading heavy legacy code bases, migrating databases,
- 15:14or doing complex forensic bug hunting, you need
- 15:17to use GPT 5 .5. Its analytical reasoning and
- 15:20ability to trace variables across files is currently
- 15:23unmatched. It acts as your senior backend engineer.
- 15:25and your most pedantic code reviewer. And what
- 15:27about GLM 5 .2? It is not useless. You should
- 15:30use it as a powerful API testing ground. Right.
- 15:32If you have high -volume, low -complexity tasks,
- 15:35it is a great tool. Just make sure you carefully
- 15:37monitor your token count so you don't fall into
- 15:39the cheap token trap. It is pushing the boundaries
- 15:42of what open models can achieve, even if it hasn't
- 15:46completely conquered the reasoning ceilings yet.
- 15:48I want to leave you with a final thought to mull
- 15:50over. We have seen that these models can build
- 15:53entire physics engines from a single prompt.
- 15:55We have seen them hunt down 21 complex bugs simultaneously
- 16:00across dozens of files. But all of this relies
- 16:03on humans grading the outcome. What happens when
- 16:06these systems become so advanced that humans
- 16:09can no longer comprehend the code they write?
- 16:12That is a scary thought. What happens when the
- 16:14AI starts writing the benchmarks to test itself?
- 16:17That is the singularity of testing. When they
- 16:20start grading their own homework, our entire
- 16:22definition of a benchmark fundamentally changes.
- 16:25We won't even know what they're optimizing for
- 16:26anymore. It is a wild frontier. We encourage
- 16:28you to run your own side -by -side cage match
- 16:30this week. Causally do it. Take your most frustrating,
- 16:33tangled current project, feed it to all three
- 16:35models, and see what happens. You might be deeply
- 16:38surprised by who actually fixes the mess and
- 16:40who just hallucinates a bigger one. We will see
- 16:43you next time. Out to your own music.