Latest / AI Fire Daily / #651 Neil: Claude Haiku 5.5 Built My OS but Struggled With Games

Transcript

#651 Neil: Claude Haiku 5.5 Built My OS but Struggled With Games · AI Fire Daily

0:00
We usually think of massive data centers when
0:02
we talk about advanced AI. We picture these billion
0:05
dollar supercomputers crunching impossible numbers.
0:08
Yeah, totally. But Claude Haiku 5 .5 is out here
0:11
building entire operating systems. It's wild.
0:14
Right? It's coding 3D video games entirely from
0:16
scratch. And it did all of this using just 1
0:20
% of a standard $200 API limit. Wow. Opus level
0:24
ambitions? Coffee budget prices. Well, welcome
0:27
to the deep dive. I'm really glad you're joining
0:29
us today. We have a genuinely fascinating mission
0:32
ahead of us. We really do. We're looking at raw
0:34
tests, benchmarks, and complex project files.
0:37
Basically, we're testing if Anthropic is deliberately
0:40
crashing the market price of artificial intelligence.
0:42
Yeah, so we'll explore the raw performance benchmarks
0:45
first. Then we test its ability to build a fully
0:48
functional browser -based operating system. After
0:51
that, we'll push its limits with raw C++ game
0:54
development. Which gets messy. Oh, very messy.
0:57
And we'll see what happens when it pairs up with
0:59
real professional tools like Godot. Finally,
1:01
we break down the actual token math behind the
1:03
curtain. So let's start with the fundamental
1:05
promise of this model. Claude Heikoop 5 .5 is
1:09
officially Anthropic's smallest model. Right.
1:12
It's designed to be their most affordable, you
1:13
know, entry level option. But the benchmark jumps
1:16
here are, frankly, they're staggering. Let's
1:18
look closely at the computer use benchmark. Yeah,
1:20
which basically tests how well an AI clicks around
1:23
a digital screen. Exactly. And it scored an impressive
1:25
72 .4%. Wait, really? Yeah. That's up from just
1:29
15 .7 % for the previous Haiku 4 .5 model. That
1:33
is a massive generational leap for a budget tier
1:35
model. It really is. And on frontier code 1 .1,
1:39
it hit 46 .4%. Now, that is a benchmark measuring
1:42
advanced coding capabilities, right? Exactly.
1:45
And it actually beat GPT -6 Luna. Oh, wow. Yeah.
1:49
Luna only scored 42 .4%. I mean, Sonnet 5 .5
1:53
still leads the computer use benchmark at 83
1:55
.9%, but Haiku 5 .5 is getting surprisingly competitive
1:59
in some very complex areas. Well, the cost efficiency
2:01
is the real story we need to unpack here. Yeah.
2:04
Anthropic introduced this new reasoning effort
2:06
scale for these models. You essentially choose
2:09
how hard the AI thinks before it answers. So
2:12
at low effort, Haiku 5 .5 scores around 42 %
2:16
on computer use. Right. But that climbs past
2:18
62 % when you shift to high effort. And it reaches
2:21
roughly 72 % at the absolute max effort setting.
2:25
OK, but higher effort means you're paying slightly
2:27
more for each individual attempt, doesn't it?
2:30
You are. But it hits that max effort for just
2:32
$0 .60 per attempt. $0 .60. Exactly. That's nothing.
2:35
Sonnet 5 .5 needs a significantly higher budget
2:38
to reach its maximum reliability. So if you run
2:40
thousands of automated tasks, that price difference
2:43
adds up quickly. The real world knowledge tasks
2:46
tell a very similar story. Like, let's look at
2:48
GDPVal. Right, which is a test measuring complex
2:53
economic reasoning. On that test, Haiku 5 .5
2:55
starts around 1130 ELO with low effort. And ELO
3:00
is simply a relative skill rating system. Right.
3:04
But it climbs to an impressive 1620 ELO at max
3:07
effort. Which is huge. It is. That gets much
3:10
closer to the premium Sonnet 5 .5 model, which
3:13
scores around $18 .40. And Haiku reaches that
3:16
$1 ,620 score for roughly $0 .80 per task. Sonnet
3:20
5 .5 costs considerably more to reach its highest
3:23
potential score. So, wait. If it's so cheap,
3:27
why wouldn't I just use the premium model anyway?
3:29
You know, just to be totally safe. Because if
3:31
you're running massive batches of automated tasks,
3:34
Sonnet will absolutely bankrupt you. Ah, I see.
3:37
Haiku gives you premium reliability on a coffee
3:40
budget, but only if you shift it into that high
3:42
effort gear. I mean, it feels exactly like shifting
3:45
gears in a manual car. Yeah, totally. You pay
3:48
just a few pennies more for gas. That lets the
3:50
AI engine rev higher and, you know, think longer
3:53
about the problem. But you're still undercutting
3:56
the luxury premium models entirely. That's a
3:58
great analogy. You get way more RPMs without
4:01
buying a whole new engine. So why is the max
4:03
effort cost scaling the secret weapon here? Well,
4:06
allocating extra compute specifically for reasoning
4:09
effectively prevents the model from hallucinating.
4:12
Right. It gives the AI time to double check its
4:15
own internal logic. It essentially buys you premium
4:18
reliability and bridges that gap between budget
4:21
and flagship. So it thinks longer to avoid making
4:24
cheap, rushed mistakes. Exactly. It's paying
4:27
for patients. But you know, benchmarks are really
4:29
just static numbers on a page. Yeah. They only
4:32
tell you so much. We really need to see what
4:34
happens in a truly complex project. Because it's
4:37
one thing to ace a static coding benchmark, but
4:39
real -world software development is deeply interconnected
4:43
and messy. Right. We need to force it to build
4:46
from a single, massive prompt. So the source
4:49
tested this by asking it to build Nova OS. Which
4:53
is a fully functional browser -based operating
4:55
system. Yeah. And this wasn't just a static visual
4:58
mock -up of a user interface. No. It had to code
5:01
a functional backend to actually support the
5:03
visuals. It built a complete Windows -style desktop
5:06
environment from scratch. It had a working taskbar.
5:09
It had a fully functional start menu. It even
5:12
displayed the current date and time correctly.
5:14
Yeah. You could drag, resize, minimize, and maximize
5:18
multiple application windows simultaneously.
5:21
It built a file manager for organizing virtual
5:23
folders and a notes app. with fully adjustable
5:26
font sizes. It built a simulated terminal for
5:29
navigating directory structures. It even built
5:31
a paint application for saving virtual drawings.
5:34
The crazy part is the interconnected logic between
5:37
these apps. Yeah, this blew my mind. Right. The
5:40
file manager actually saves the notes you type
5:42
in the notes app. Which means the AI had to hold
5:45
multiple independent programs in its head simultaneously.
5:49
The prompt also required a realistic mail inbox
5:52
with sample messages. It had to display pop -up
5:55
notifications for new incoming messages. And
5:58
it included highly interactive playable games.
6:00
Oh, the games. It built a GTA -inspired driving
6:04
game right on the desktop. You had controllable
6:07
vehicles, moving traffic, and active police cars.
6:10
Though, hilariously, the player got completely
6:12
trapped inside the car. Oh, yeah. A transparent
6:15
map accidentally covered the entire screen. Right.
6:18
Apparently, NovaOS wanted you to live in the
6:20
car permanently. Just forever driving. It also
6:23
built a surprisingly fun retro space arcade game.
6:26
That space shooter had really decent visual effects
6:29
when taking damage. But my absolute favorite
6:31
part of the whole build is the mood engine. Yes,
6:34
the mood engine is amazing. It continuously monitors
6:37
your keyboard activity, mouse movement, and idle
6:40
time. It converts your physical user activity
6:42
into an energy value between zero and one. Then
6:45
it makes the desktop wallpaper dynamically react
6:47
to changes in that energy. I mean, just imagine
6:51
an AI generating an entire working OS with a
6:54
mood tracker in one go. That is genuinely mind
6:57
-blowing. It's truly incredible. But, you know,
7:00
it wasn't a totally flawless victory. No. It
7:02
hit a massive wall during the development process.
7:05
Right. It threw a critical output limit error
7:08
during the process. The paint application suddenly
7:11
showed a storage full error. The entire session
7:14
stopped dead in its tracks. The response had
7:17
completely exceeded the API's standard output
7:20
limit. And usually when an AI session crashes
7:23
like that, you lose everything. It just breaks
7:25
the continuity of the code entirely. Exactly.
7:28
This is where the recovery prompt becomes incredibly
7:31
important. They needed to recover as much completed
7:33
work as mathematically possible. So the prompt
7:36
told the AI to list all existing project files.
7:39
It had to carefully identify completed, incomplete,
7:43
and missing components. It had to run available
7:45
build commands and validation checks. It needed
7:48
to specifically identify errors preventing the
7:51
application from starting. And it was explicitly
7:53
told not to rewrite any working code unnecessarily.
7:57
It had to resume precisely from the last verified
8:00
working state. And remarkably, ClothiQ 5 .5 had
8:04
saved absolutely everything. Really? Yeah. All
8:07
36 complex source files were still there, perfectly
8:10
intact. Wait, how did it manage to recover from
8:14
the output limit crash without corrupting the
8:16
code? Well, the recovery prompt forced the AI
8:19
to inspect the project directory first. It had
8:22
to methodically verify functionality before it
8:25
was allowed to write another single line of code.
8:28
So it essentially checked its own saved work
8:30
before writing new code? Precisely. It demonstrated
8:34
a highly methodical, self -healing approach to
8:38
software development. NovaOS was deeply impressive
8:41
for web development tasks. It was. But web frameworks
8:44
handle a lot of the heavy lifting behind the
8:46
scenes, right? Right. They provide a safety net
8:48
for the AI to fall back on. The web games inside
8:51
NovaOS had some obvious fatal flaws. So what
8:55
happens if we strip away that web framework completely?
8:58
Like, what if we ask it to build raw 3D physics
9:01
from absolute scratch? Well, the source challenged
9:04
it to build a 3D skateboarding game. And it had
9:06
to use raw C++ code. without any game engine.
9:10
No Unity. No Unreal Engine. No Godot to hold
9:13
its hand. Just pure C++ and standard system libraries.
9:17
They asked for a detailed skate park with roads,
9:20
sidewalks, and ramps. They wanted a dynamic third
9:23
-person camera that smoothly follows the player.
9:26
It had to support complex movement mechanics
9:28
using the W, A, S, and D keys. It needed to implement
9:32
acceleration, braking, turning, and jumping physics.
9:36
And surprisingly, it successfully coded all of
9:39
that mathematical movement perfectly. Really?
9:42
Yeah. It even coded functional kickflips using
9:45
raw math. That's impressive. The game had moving
9:47
cars, pedestrians, and a real -time speed display.
9:51
It even had a clear interface showing the current
9:53
trick score. It built a cinematic replay system
9:56
using the Z key, too. It showed your completed
9:59
tricks from a totally different camera angle.
10:02
But there was a rather fatal flaw in the visual
10:06
design. Right, it completely failed the visual
10:08
physics test. The character animations were incredibly
10:12
stiff and unnatural. It thinks human legs are
10:14
made of wooden sticks. Yeah, especially when
10:17
the skater was flying in midair. The physical
10:19
logic just didn't translate visually at all.
10:21
The writer looked totally rigid. So they tried
10:24
to improve it with a specific reference image.
10:27
They fed it a screenshot from Tony Hawk's Pro
10:29
Skater 3 plus 4. Right, they wanted to improve
10:32
the environment and the character proportions,
10:34
but they explicitly wanted to keep the existing
10:36
low -poly art style. They specifically asked
10:39
it to make the skateboard move independently
10:41
from the rider. The rider's legs needed to stay
10:43
in natural bending positions. The prompt clearly
10:46
said feet must separate from the board during
10:48
tricks, but the visual upgrade was a complete
10:51
and utter failure. It added some basic geometric
10:54
skate park ramps, but the graphics looked terrible
10:57
and looked like it was hastily built from basic
10:59
geometric shapes. Haiku 5 .5 apparently only
11:02
noticed the ramps in the reference image. It
11:05
completely ignored the complex character posing
11:07
and bodily proportions. So why could it code
11:10
complex physics but fail so badly at updating
11:12
graphics from a reference image? Because translating
11:14
two -dimensional vision into three -dimensional
11:16
spatial logic is incredibly difficult. Makes
11:19
sense. It processes pixels as statistical patterns,
11:23
not physical objects in simulated space. It remains
11:26
a massive blind spot for budget -tier AI models.
11:29
So it sees the shapes, but lacks the artistic
11:32
translation skills. Exactly. It can calculate
11:35
the math of a jump perfectly, but it cannot visualize
11:38
how a human knee naturally bends during that
11:40
jump. So since raw C++ was a visual disaster,
11:44
they kind of pivoted their approach? Let's pair
11:46
this cheap analytical brain with professional
11:49
development tools. Right. We want to see if structural
11:52
guardrails help the model succeed visually. They
11:55
asked it to build a 3D backyard pool party game.
11:58
It used Blender for generating assets and Godot
12:01
for the gameplay logic. The prompt asked for
12:04
a highly detailed backyard with a swimming pool.
12:07
It wanted a colorful, vibrant game inspired by
12:10
casual party games. It specifically asked for
12:13
a casual Wii Sports style vibe, and the environment
12:16
had amazing details built right in. It really
12:19
did. It included beautifully animated water moving
12:21
in the pool. It had poolside furniture, glowing
12:24
string lights, and even a friendly jumping dog.
12:27
A dog. Yeah. That's great. Yeah. The prompt demanded
12:31
several selectable characters with totally unique
12:33
appearances. It wanted smooth, responsive, and
12:36
intuitive character movement. And the gameplay
12:39
mechanics were surprisingly solid and genuinely
12:41
fun this time. Oh, yeah. You used the space bar
12:43
to carefully control your jump power. There was
12:46
a timing -based meter with distinct yellow and
12:48
green success zones. Right. And you used the
12:51
A and D keys to physically steer midair. You
12:54
used W and S to carefully adjust your spin midair.
12:57
Practice mode was definitely the real standout
13:00
feature here, though. Oh, absolutely. It provided
13:02
highly specific, actionable feedback after every
13:06
single dive into the pool. Right. It would calculate
13:08
and say, you scored 58 out of 100. It would tell
13:11
you that you landed exactly 1 .3 meters away
13:14
from the target. It even told the user they jumped
13:16
way too hard. You know, I have a vulnerable admission
13:19
to make right now. Oh, yeah? If you're an indie
13:21
developer listening, you probably know this struggle.
13:23
I still wrestle with getting basic collision
13:26
boxes to work in Godot myself. Oh, same. And
13:29
this budget AI just whipped up a dynamic scoring
13:31
system in seconds. It's genuinely wild to see
13:34
that level of structural competence. The backyard
13:37
felt alive and the gameplay was truly interactive.
13:41
Which leads me to a really obvious question about
13:43
these tests. Why was the Godot test so vastly
13:46
superior to the C++ test? Well, game engines
13:49
provide an established underlying mathematical
13:52
framework and physics engine. Right. The AI understands
13:55
those native rules instantly, rather than having
13:58
to invent physics entirely from scratch. So engines
14:01
give it the physics guardrails it desperately
14:03
needs. Exactly. It leans on the engine's built
14:06
-in rules incredibly effectively. It doesn't
14:09
have to imagine how gravity works anymore. So
14:12
it can build amazing things if you give it the
14:15
right tools. Right. But how much did all this
14:18
prompting and failing actually cost? Let's look
14:21
closely at the bottom line on pricing. The API
14:24
pricing breakdown is highly specific and honestly
14:27
very surprising. It costs just 10 cents per million
14:31
input tokens. Yep. And it costs 50 cents per
14:35
million output tokens. That pricing structure
14:38
makes it 90 % cheaper than Haiku 4 .5. Wow. But
14:42
there is a massive catch you have to understand.
14:45
Okay, what is it? Those prices only apply if
14:47
your prompts are under 100k tokens. Anthropic
14:51
says most requests easily fall within this specific
14:54
limit. Right. Even cash reads are incredibly
14:57
cheap under that specific threshold. And cash
15:00
reads basically mean accessing previously saved
15:03
conversation data. They cost just 1 cent per
15:06
million tokens and cash writes cost about 12
15:09
.5 cents. The model features a massive 1 million
15:13
token context window. Which is how much text
15:16
the AI can remember at one time. Right. It supports
15:19
up to 128k output tokens per single request.
15:23
And it currently has a knowledge cutoff of June
15:25
2026. But that 100k pricing tier threshold is
15:29
the critical detail here. Absolutely. It completely
15:32
changes the financial math of using this budget
15:34
model. Yes. What happens if you exceed that 100k
15:37
token limit during a long coding session? The
15:40
price jumps dramatically across the board without
15:42
warning. Oh, no. Yeah, it goes to 50 cents for
15:45
input tokens and it skyrockets to two dollars
15:48
and 50 cents for output tokens. So keep your
15:51
project files small or the budget vanishes. Precisely.
15:54
It completely erodes the budget advantage of
15:56
the model. Cash reads jump to five cents. Cash
15:59
writes jump to over 60 cents. Which brings us
16:02
to the core takeaway from all these complex coding
16:05
tests. Claude Haiku 5 .5 is clearly not just
16:09
a cheap toy. No, it is a highly capable developer
16:12
for complex software projects. But it requires
16:15
a very specific approach from the human user.
16:18
Right. You absolutely must use established frameworks
16:21
like Godot for structural guardrails. And you
16:25
must keep your context windows carefully managed
16:27
to avoid bankruptcy. If you respect those boundaries,
16:31
it forces us to completely rethink what budget
16:33
AI actually means. It fundamentally changes the
16:37
entire landscape of basic software development.
16:39
It really makes you wonder about the premium
16:41
models. Like, what exactly are those expensive
16:43
models charging extra for right now? If Haiku
16:46
provides this much reliability at the max effort
16:49
setting, the premium tier might be overkill for
16:52
everyday coding tasks. True. It really lowers
16:55
the barrier to entry for indie developers everywhere.
16:58
I want to leave you with a provocative thought
17:00
today. Think about the economic implications
17:03
of what we just discussed. The sheer scale of
17:06
the industry disruption here. If a budget AI
17:09
model from 2026 can build a playable 3D video
17:12
game, complete with a dynamic scoring system,
17:15
for less than the cost of a gumball, what happens
17:19
to the value of basic software development over
17:21
the next five years? That is a truly staggering
17:24
question to ponder. The floor for technical skills
17:27
is rising incredibly fast. We encourage you to
17:30
test this out for yourself this week. Try pushing
17:32
a budget AI model with a complex multi -step
17:35
prompt. See those advanced reasoning capabilities
17:38
with your own eyes. Push it to the max effort
17:40
setting and just watch it work. Thanks for taking
17:43
this deep dive with us today. We'll catch you
17:45
on the next one.