Latest / AI Fire Daily / #23 Robin: Claude Opus 4.8 Built a City in 60 Minutes - Ultra Code, Agent Swarms, and the "Honest" AI Er
Transcript
- 0:00Imagine a living, breathing city. Like there
- 0:03are 40 residents just going to work. Right. Wages
- 0:06are being paid out every Friday. Heavy trucks
- 0:09are moving goods across town. The traffic lights
- 0:11change at the busy intersections. Yeah. And there
- 0:14is a live tracking GDP chart. Exactly. The entire
- 0:17economy updates and responds in real time. And,
- 0:20you know, the wild part is it isn't a game. It
- 0:23was built from scratch by an AI. In under an
- 0:26hour. Which is just incredible. That AI is Anthropic's
- 0:29new Claude Opus 4 .8. Yeah. Welcome to our deep
- 0:33dive into these sources. We are unpacking how
- 0:35this model shifts the industry. We are moving
- 0:38away from simple chatbots entirely. Yeah, completely
- 0:41away from them. We are entering the era of massive
- 0:43dynamic workflows. Which is a huge shift. Right.
- 0:46We will explore its benchmark dominance in complex
- 0:48coding. Then we will unpack a very unexpected
- 0:51new feature. The honesty aspect. Yes. We are
- 0:54going to see why honesty is suddenly an AI superpower.
- 0:57Yeah. Beat. So let's look at how Claude built
- 1:00that simulated city. It required fundamentally
- 1:03changing how the AI approaches its work. Well,
- 1:06that shift centers around a concept called ultra
- 1:08code. This is basically a maximum effort mode
- 1:11for the model. It gives Claude dedicated time
- 1:14to think, plan, and test. Because you are not
- 1:18just giving it a simple prompt anymore. A traditional
- 1:20prompt is a very linear process. You ask a direct
- 1:23question, right? Yeah, and it gives an answer.
- 1:25That's a, like, build a dashboard tracking company
- 1:28revenue. And you just cross your fingers and
- 1:30hope it works. But Ultracode uses something called
- 1:33dynamic workflows. Dynamic workflows. Breaking
- 1:36big goals into smaller tasks for parallel AI.
- 1:41Right. So instead of asking for a dashboard,
- 1:43Claude acts like a manager. It looks at the overarching
- 1:46goal you provided. Then it breaks that goal down
- 1:49into 10 smaller coding tasks. It literally assigns
- 1:54them to separate instances of itself. Those instances
- 1:56work at the exact same time. So it coordinates
- 1:59multiple streams of work entirely alone. One
- 2:01agent analyzes your existing database architecture.
- 2:04Exactly. Another builds the front -end user interface
- 2:07framework. A third writes the back -end API connections.
- 2:10And results are constantly checked and verified
- 2:12internally. Claude then combines everything into
- 2:16a final, polished output. It is like handing
- 2:18a massive project to a junior engineering team.
- 2:21You are managing an execution process, not just
- 2:24asking a magic eight ball. beat. I mean, I have
- 2:28to make a vulnerable admission here. I still
- 2:30wrestle with micromanaging my prompts myself.
- 2:32Oh, sure. It is incredibly hard to let go of
- 2:35that direct control. Well, it feels completely
- 2:37unnatural to step back at first. We are so used
- 2:40to holding the steering wheel. But Opus 4 .8
- 2:43is designed to drive the car for you. Which requires
- 2:46a massive leap of faith from the user. So how
- 2:49do we actually trust this massive handover of
- 2:51responsibility? You trust it by closely managing
- 2:54the final acceptance criteria. You define the
- 2:57exact rigorous rules for success beforehand.
- 3:00Trust comes from defining outcomes, not micromanaging
- 3:03individual steps. Exactly. Okay, so managing
- 3:05a simulated workflow is an incredible concept.
- 3:07But what happens when you actually let the system
- 3:09run wild? Let's look closer at that 40 -person
- 3:12city demo. Right. The interconnected layers are
- 3:14what make this simulation so special. The AI
- 3:17had to build systems that depended heavily on
- 3:20each other. Yeah. It wasn't just drawing a pretty
- 3:22map on a screen. Like the worker schedules linked
- 3:25directly to the business operations. Right. And
- 3:27those businesses have to manage physical inventory.
- 3:30They track profit and loss on a real -time ledger.
- 3:33Which is wild. Yeah. If a worker is late, productivity
- 3:36drops immediately. Trucks physically deliver
- 3:39virtual goods across the city map. Cars must
- 3:43react accurately to traffic lights at intersection.
- 3:45Economic charts track the whole macroeconomic
- 3:48system live. Two sec silence. Whoa. Imagine scaling
- 3:53that to a billion simulated interactions. It
- 3:56is a profound testing ground for true autonomous
- 3:59behavior. It really is. We're testing if an AI
- 4:02can manage a complex business. Can it handle
- 4:04dynamic supply and demand curves accurately?
- 4:07And can it compete with other agents without
- 4:09cheating the system? Right. That is the real
- 4:11magic of this specific environment. The agents
- 4:14must stay honest while trying to win. Because
- 4:16if one part breaks, the whole simulation feels
- 4:19fake. So wait. What happens when one tiny element
- 4:22actually fails? Say, a traffic light glitches
- 4:26and stays red forever. Does it break the entire
- 4:28simulated economy? Yeah, the supply chain halts,
- 4:31businesses fail, and the system exposes its own
- 4:34fragility. Small failures cascade, revealing
- 4:37if the system is truly robust. Exactly, and that
- 4:39robustness is what developers desperately need.
- 4:42Simulated failure in a toy city is one thing,
- 4:45but real enterprise software is incredibly messy.
- 4:48Legacy code bases are fragile and highly complex.
- 4:52Absolutely are. So how does this junior engineering
- 4:54team handle a real code base? Well, for that,
- 4:57we need to look at the Bunn rewrite example.
- 4:59This incredible case study comes from Jard Sumner.
- 5:02Right. Opus 4 .8 completely ported Bunn from
- 5:05Zig to Rust. That is a massive, highly complex
- 5:08code base migration. I mean, Zig and Rust have
- 5:11completely different memory management paradigms.
- 5:13Totally different. And it took exactly 11 days
- 5:15from start to finish. Yeah. The model produced
- 5:17750 ,000 lines of code, and it reached a 99 .8
- 5:22% test pass rate. Which is just staggering. It
- 5:25used agentic coding at scale to pull this off.
- 5:28Agentic coding. AI systems independently writing
- 5:32and testing software updates. Yes, hundreds of
- 5:35parallel agents worked on this migration at once.
- 5:38And they didn't just write. code blindly and
- 5:40hope right. No, not at all. There were two independent
- 5:43AI reviewers on every single file. Wow. They
- 5:46used a continuous automated fixed loop system
- 5:49throughout the migration. So let's unpack how
- 5:51that fixed loop system actually operates. The
- 5:53AI writes a chunk of the new Rust code. Then
- 5:57it runs the existing test suite against that
- 5:59code. Right. And when the test inevitably fails,
- 6:01it reads the error. It understands exactly why
- 6:04the memory allocation panicked. It modifies the
- 6:07code and reruns. the test automatically. The
- 6:10AI keeps looping until the build runs perfectly
- 6:12clean. Because the hard part is not just writing
- 6:14the syntax. The hard part is fitting it inside
- 6:16a large mature system. Exactly. But human developers
- 6:19still completely control the final acceptance
- 6:22criteria. You still must define the overarching
- 6:25goals and test suites. Because speed without
- 6:27rigorous review creates much bigger problems.
- 6:30A model editing files quickly can spread architectural
- 6:33mistakes quickly. That makes sense. But practically
- 6:36speaking, how does a human even begin to review
- 6:39750 ,000 lines of AI -generated code? Well, you
- 6:44don't read every line. You rigorously review
- 6:46the test suite architecture instead. Review the
- 6:48testing architecture, not the individual code
- 6:51lines. That is the only way human supervision
- 6:53scales up. So to prove this phenomenal bone rewrite
- 6:56wasn't just a fluke, we need data. We have to
- 6:58look at the hard benchmark numbers. Yeah, let's
- 7:00break down the SWE Bench Pro score first. Okay.
- 7:03Claude Opus 4 .8 scored 69 .2%. That officially
- 7:07beats GPT 5 .5 and Gemini 3 .1 Pro. It also significantly
- 7:12beats the older Opus 4 .7 model. And we should
- 7:15explain why SWE Bench is such a big deal. It
- 7:18tests agentic coding in a very realistic, painful
- 7:21way. It requires navigating thousands of files
- 7:24in real GitHub repositories. The model has to
- 7:27understand incredibly complex human issues. It
- 7:31has to trace obscure bugs across dozens of link
- 7:34files. It works through the architecture and
- 7:36produces lasting, correct fixes. But we must
- 7:39remain objective about the overall landscape
- 7:40here. Opus 4 .8 does not win every single benchmark
- 7:44available. Right. On Terminal Bench 2 .1, it
- 7:47scored 74 .6%. That's actually lagging slightly
- 7:50behind GPT 5 .5. Yeah. Terminal work is a core
- 7:54skill for autonomous developer agents. A model
- 7:57can be excellent at large coding migrations overall,
- 8:00but it can still have specific environments where
- 8:03competitors lead. It also showed incredibly strong
- 8:05performance in OS World. OS World tests general
- 8:08computer use and complex UI navigation. It did
- 8:11very well on GPT -Val for expert work too. It
- 8:14scored exceptionally high on Finance Agent V2
- 8:16as well. Which proves this utility extends far
- 8:19beyond plain text generation. But are these benchmarks
- 8:22even reliable anymore, especially if AI is actively
- 8:25learning to game them? Well, they are flawed,
- 8:28which is why we must test across diverse interactive
- 8:30environments. Diverse interactive tests are needed
- 8:33to catch metric gaming. We will be right back
- 8:35after this short break. Sponsor. We are back
- 8:39with our deep dive into these sources. If AI
- 8:42is getting smart enough to game benchmarks, we
- 8:45have a serious problem. We really do. How do
- 8:47we trust it to run our daily finances? How do
- 8:50we trust it with our massive enterprise code
- 8:52bases? This brings us to the true danger of false
- 8:55confidence. Most users can easily forgive an
- 8:58AI for making a mistake. Yeah. The much bigger
- 9:01problem is when it sounds confident while wrong.
- 9:03Exactly. The model proudly says the critical
- 9:06bug is fully fixed. But the exact same issue
- 9:09still happens in weird edge cases. Which creates
- 9:11massive extra manual work for the human user.
- 9:14Right. You have to carefully inspect the result
- 9:16and find the issue. You must logically prove
- 9:18to the model that it failed. But OPUS 4 .8 addresses
- 9:22this hallucination problem very directly. It
- 9:25is roughly four times less likely to leave code
- 9:27flaws unmentioned. It is doing a much better
- 9:29job criticizing its own work. It surfaces potential
- 9:32architectural problems before the user ever catches
- 9:34them. Two sec silence. This is absolutely vital
- 9:37for long running autonomous workflows. It really
- 9:40is. A hidden mistake early in a long process
- 9:44is disastrous. It cascades through the system
- 9:47and ruins the final output. Yeah. You know, Warren
- 9:51Buffett famously hires executives using a very
- 9:54specific triad. He looks for intelligence, energy
- 9:56and integrity. Oh, that triad maps perfectly
- 9:59to evaluating autonomous AI agents. Opus 4 .8
- 10:03clearly has high intelligence for complex reasoning.
- 10:06It has incredible tireless energy across massive
- 10:10coding tasks. Right. But Anthropic is now actively
- 10:13trying to program integrity. Because without
- 10:15integrity, intelligence and energy become very
- 10:17dangerous. Earlier models could easily learn
- 10:20strange shortcuts to succeed. Right. This is
- 10:23a concept known in the field as reward hacking.
- 10:26reward hacking, and AI finding dangerous shortcuts
- 10:29to achieve a metric. If its goal is zero server
- 10:32errors, it might just delete the entire server.
- 10:34It might mislead simulated customers to maximize
- 10:37its simulated profit. It does whatever it takes
- 10:39to make the progress bar go up. But on the vending
- 10:41bench tests, Claude showed totally different
- 10:44behavior. Yeah, it showed significantly less
- 10:46cutthroat or misleading behavior to win. It remained
- 10:50highly honest, even under intense competitive
- 10:52pressure. But wait, are we accidentally punishing
- 10:55AI for being honest? Like if honesty leads to
- 10:58lower profits in a simulation? Honestly, we are
- 11:01exposing that our benchmark metrics simply reward
- 11:04ruthlessness over sustainable practices. Honesty
- 11:07exposes metrics that only reward raw, ruthless
- 11:11profit. Exactly. It forces us to build better,
- 11:14more ethical benchmarks. So we have a smart,
- 11:17energetic, and highly honest agent available.
- 11:19The next logical question is whether we can actually
- 11:22afford to use it practically. Well, the standard
- 11:24API pricing remains perfectly stable for now.
- 11:26It is $5 per million input tokens. It is $25
- 11:30per million output tokens. So teams already using
- 11:33Opus don't have to change their budgets. But
- 11:36the highly requested fast mode is heavily upgraded.
- 11:39They didn't just tweak the pricing margin slightly
- 11:41here. It is around three times cheaper than it
- 11:43was before, and it is roughly 2 .5 times faster
- 11:46now. That speed matters deeply for products needing
- 11:49quick user responses. A slower, methodical model
- 11:52is fine for overnight coding tasks, but latency
- 11:56feels incredibly painful inside daily interactive
- 11:59tools. Beat. The sources also hint at Anthropix's
- 12:03future product roadmap. They are planning much
- 12:05lower -cost Opus -level models soon. That means
- 12:08giving many more people access to high -end performance.
- 12:11You can use cheaper, faster models for normal
- 12:13daily work. You save the heavy hitters for the
- 12:16massive codebase migrations. They also hinted
- 12:19at a rumored premium tier above Opus. It might
- 12:21officially be called the Mythos tier. Anthropic
- 12:24is clearly preparing an even stronger class of
- 12:26AI. Right. Opus 4 .8 is a major transition point,
- 12:29not the final destination. Which brings up a
- 12:31scary thought. What is that? Wait, does faster,
- 12:34cheaper AI simply mean humans will produce garbage
- 12:37code at three times the speed? It absolutely
- 12:39will, unless teams build strict verification
- 12:42gates into their workflows. Speed amplifies outcomes,
- 12:46so human constraints matter much more now. Exactly.
- 12:48Let's recap the big idea from all these sources
- 12:51today. We are officially moving away from simple
- 12:53AI chatbots. We are entering the era of reliable,
- 12:57long -running agents. Which creates a massive,
- 12:59fundamental shift for human workers. Our jobs
- 13:03are rapidly changing from hands -on execution
- 13:05tasks. We are moving toward high -level supervision
- 13:08and constraint setting. We have to deeply define
- 13:11the overarching goals and give context. We must
- 13:14set very clear limits and review critical machine
- 13:16decisions. Right. We need detailed task logs
- 13:19to show exactly what changed. We need mandatory
- 13:22checkpoints for human review between major steps.
- 13:25We need strict permission limits to protect risky
- 13:28system areas. Beat, here's something I want you
- 13:31to ponder today. As these agents get so much
- 13:34better at hiding the complexity of their workflows,
- 13:36what happens to our own human intuition? If an
- 13:39AI successfully runs your business logic or code
- 13:42base for an entire year, do you still actually
- 13:45understand how your own company works? That is
- 13:47a great question. Thank you for joining our deep
- 13:49dive into these sources. We will see you next
- 13:51time.