Latest / AI Fire Daily / #532 Neil: GPT-5.6 Sol Tested Against Fable 5 For Serious AI Tasks
Transcript
- 0:00Imagine you spend months building up to a massive
- 0:03product launch. Right. The confetti barely hits
- 0:06the floor. The tech world is cheering loudly.
- 0:08Yeah, everyone is celebrating. And then your
- 0:10biggest rival. drops a next -generation model,
- 0:13they drop it right on top of your news cycle.
- 0:15Oh, it is brutal. That is exactly what OpenAI
- 0:18just did to Anthropic. Fable 5 had barely launched.
- 0:21Jiri, out the door. Right. And then GPT 5 .6
- 0:24saw under the chat, it feels like a sudden clash
- 0:27of AI titans. It really does. Which one actually
- 0:29takes the crown right now. Welcome to the deep
- 0:32dive. Today we are exploring a real head -to
- 0:36-head performance comparison. We are looking
- 0:38closely at GBT 5 .6 Sol and Fable 5. We will
- 0:42rely strictly on the hard bench per data. Just
- 0:45the data. We are also looking at OpenX coding
- 0:47demos. And we will review direct chat GPT chat
- 0:52interface tests. Yeah. We are not taking sides
- 0:54here. Not at all. We just want to unpack the
- 0:56data for you. We want to help you figure out
- 0:57the best tool. It is all about fitting your specific
- 1:00daily workflow. Okay, let's unpack this. Yeah,
- 1:03the timing from OpenAI was definitely not an
- 1:05accident. Fable 5 was getting a ton of immediate
- 1:08attention everywhere. OpenAI clearly wanted to
- 1:11shift that spotlight back. They wanted it back
- 1:13on their own ecosystem. But marketing timing
- 1:16does not automatically equal better performance.
- 1:18No, it definitely doesn't. We have to look past
- 1:21the shiny launch announcements. We need to see
- 1:23how these models actually handle work. We need
- 1:25to understand the real underlying mechanics.
- 1:27Exactly. We need to see what OpenAI actually
- 1:30built first. Let us look at the new GPT 5 .6
- 1:34model family. It is actually broken down into
- 1:37three distinct models. You have Sol, which is
- 1:40the massive flagship model. Sol is built for
- 1:42complex, multi -step professional work tasks.
- 1:45It handles heavy coding and deep research workflows.
- 1:49Then you have Terra sitting right in the middle.
- 1:51Terra gives you a highly balanced mix of performance
- 1:53and cost. It is designed for everyday tasks needing
- 1:57reliable logic. Finally, there is Luna for the
- 2:00lighter, rapid workloads. Right, the fast one.
- 2:03Exactly. Luna is basically built for blazing
- 2:05fast data crunching. It handles large scale text
- 2:09processing at a much lower cost. Inside Chat
- 2:12GPT itself, Sol has different reasoning levels
- 2:15available. You can choose medium -high or extra
- 2:17-high reasoning. Which is pretty cool. It depends
- 2:19entirely on how difficult your specific task
- 2:22is. There is also a Sol Pro version available
- 2:24now. Sol Pro gives the model extra compute to
- 2:27check its work. Right, and when a task requires
- 2:30multiple complex steps, checking work is critical.
- 2:33Yeah. It allows the model to spot its own logical
- 2:36errors. Honestly, I still wrestle with prompt
- 2:38drift myself. Absolutely. You know, when the
- 2:40AI slowly forgets your instructions over a long
- 2:43chat. So navigating this many model modes just
- 2:46feels daunting. Yeah, I totally get that. I have
- 2:48to play devil's advocate here. As a user, staring
- 2:51at a massive drop -down menu feels overwhelming.
- 2:54You have medium -high, extra -high, pro, max,
- 2:57and ultra. It is a lot. Is this multimode setup
- 3:00unnecessarily complicated for the average user?
- 3:03Are they over -complicating this just to monetize
- 3:05different compute tiers? Well, it definitely
- 3:07looks like choice paralysis at first glance.
- 3:10But it is really about maximizing your overall
- 3:13efficiency. OK. You match the specific tool to
- 3:16the size of the task. Using Sol Pro for a simple
- 3:19summary is just wasteful. Right. It burns expensive
- 3:22compute power and waste your valuable time. By
- 3:25picking the right tier, you save money and energy.
- 3:28Makes sense. Also, they included a max mode for
- 3:31deep reasoning. Max mode just gives Sol much
- 3:34more time to think. It allows the model to map
- 3:37out complex logic trees. And then there is the
- 3:40highly anticipated ultra mode. Right. Ultra mode
- 3:43uses parallel subagents for massive tasks. Right.
- 3:46And subagents are just tiny digital workers splitting
- 3:49up a task. Think of it like a bustling restaurant
- 3:52kitchen instead of one master chef cooking a
- 3:54five -course meal sequentially. You have a head
- 3:57chef instantly delegating to a dozen sous chefs.
- 4:00They chop, boil, and bake simultaneously. That's
- 4:02a great analogy. It is highly efficient for massive
- 4:05workflows and data analysis. They can process
- 4:08information much faster than a single worker.
- 4:10We should also mention the specific API costs
- 4:12involved. Through the API, Sol costs $5 per million
- 4:18input tokens. Yeah. And it costs $30 per million
- 4:21output tokens. That can add up incredibly fast
- 4:24if you run heavy automation. Exactly. That is
- 4:27why the different compute tiers are actually
- 4:29helpful. They let developers control their exact
- 4:31server spending. Right. So don't use a sledgehammer
- 4:34to crack a tiny nut. Precisely. Do not pay for
- 4:36a supercomputer to write an email. OK. So Sol
- 4:39has this brute force parallel processing power.
- 4:42But massive compute power does not always equal
- 4:45true intelligence. No, it doesn't. How does Sol
- 4:48actually perform against Fable 5 in blind testing?
- 4:51Does that extra compute translate to higher benchmark
- 4:54scores? What's fascinating here is where Sol
- 4:56clearly pulls ahead. Sol dominates in long professional
- 5:00tasks and direct execution. Take the agent's
- 5:03last exam benchmark, for example. This test measures
- 5:06if an AI can stay entirely focused. It tests
- 5:09long workflows without getting distracted or
- 5:11confused. Sol scored 52 .7 % on this exam. Fable
- 5:155 only reached 40 .5%. That is a... pretty significant
- 5:19gap in overall performance. Solol also leads
- 5:21in the important coding agent tests. On the artificial
- 5:24analysis coding agent index, Sol scored 80. Yeah.
- 5:27Fable 5 was close, but trailed slightly at 77
- 5:30.2. But here is the really interesting mechanical
- 5:33detail. Sol actually used far fewer output tokens
- 5:36to finish the job. That matters because it looks
- 5:39beyond just the final code. It means Sol is not
- 5:42just babbling to find an answer. Right. It writes
- 5:44concise, efficient code without wasting your
- 5:47money. We see a similar result on DeepSWE version
- 5:501 .1. Sol scored 72 .7 % against Fable 5 at 69
- 5:55.7. Sol excels at handling long sequential software
- 5:59engineering tasks. What about command line work
- 6:02and terminal testing benchmarks? This is where
- 6:04the AI types raw code into a black box. Right.
- 6:07No safety net. There was no pretty user interface
- 6:09to guide it. Sol performed incredibly well on
- 6:11Terminal Bench 2 .1. Sol scored 88 .8 % against
- 6:14Fable 5 at 83 .1%. Wow. And when they turned
- 6:17on Sol Ultra Mode, it hit an impressive 91 .9
- 6:20% on that raw test. That Ultramode score proves
- 6:23the real value of subagents. Breaking a large
- 6:26complex task into smaller parts really works.
- 6:29It helps the model coordinate different tools
- 6:31and plan automatic retries. Sol also has a clear
- 6:34advantage in deep web research. Browse Comp tests
- 6:38complex searches across multiple different web
- 6:40sources. It requires the model to connect information
- 6:43from several isolated places. Sol scored 90 .4
- 6:48% on this specific test. That is high. And Sol
- 6:51Ultra reached an amazing 92 .2%. According to
- 6:54the chart, Fable 5 only managed 84 .3%. Interestingly,
- 6:5916 agent setups reach high scores much faster.
- 7:02Single agents need way more time to get anywhere
- 7:04close. This proves that parallel processing is
- 7:07the future of deep research. But we have to look
- 7:10at the other side of the coin. Right. Fable 5
- 7:12still wins some very important standardized benchmarks.
- 7:15On the artificial analysis intelligence index,
- 7:17Fable 5 scored 59 .9. Sol is just slightly behind
- 7:20at exactly 58 .9. So Fable 5 still edges out
- 7:23in general. Broad intelligence. And the gap becomes
- 7:26absolutely huge on SWBench Pro. Fable 5 scored
- 7:29an incredible 80 % on this test. Wow. Sol hit
- 7:33a much lower 64 .6%. SWBench uses real world
- 7:38repository issues for its tests. Imagine dropping
- 7:41an AI into a 10 ,000 line existing code base.
- 7:45It has never seen this specific code before.
- 7:48You give it a vague bug report and tell it to
- 7:50fix it. That sounds like an absolute nightmare
- 7:52for any developer. It is. But Fable 5 dominates
- 7:55when dealing with these massive codebases. Why
- 7:57does Sol crush coding agents but lose so badly
- 8:00to Fable 5 on SWBench repository coding? Well,
- 8:04it comes down to their underlying architectural
- 8:06differences. Sol is optimized for step -by -step
- 8:09logic gates. It excels at sequential tool use
- 8:12and direct execution. It is great at taking the
- 8:15next immediate logical step, but sometimes it
- 8:18forgets what room of the house it is building.
- 8:20Fable 5 handles broad reasoning much better.
- 8:22It's training heavily emphasized wider context
- 8:25window retrieval. And a context window is just
- 8:27the AI's short -term memory for reading files.
- 8:29Right. Fable 5 can hold the blueprint of an app
- 8:32in memory. It understands how a small change
- 8:36affects an entire existing code base. It sees
- 8:39the big picture much better than Sol does. So
- 8:42Sol is the marathon runner, while Fable is the
- 8:45architect. That is the perfect way to look at
- 8:47it. Sol will flawlessly run the miles, but Fable
- 8:51draws the map. Benchmarks are sterile and highly
- 8:53controlled testing environments. What happens
- 8:56when real developers throw messy tasks at them.
- 9:00We looked at some open -ended community tasks
- 9:02shared on X. These coding demos show us the models
- 9:06in real action. This is where we see how they
- 9:08actually behave under pressure. One user had
- 9:11Saul build a complete Replet -style coding app.
- 9:14It was entirely generated from just one single
- 9:17text prompt. That is mind -blowing. The project
- 9:19included a database and isolated code sandboxes.
- 9:23It even had an app building agent inside the
- 9:24main app. That is basically an AI creating another
- 9:27AI to build things. Sol had to connect all these
- 9:30moving product parts together. Exactly. It had
- 9:32to keep the same goal across a long build. That
- 9:34is far more complex than a simple landing page.
- 9:37It requires maintaining context over a very long
- 9:40generation period. Another amazing test involved
- 9:43building a complex spaceship game. Sol worked
- 9:46continuously on the spaceship game for 87 minutes.
- 9:49Yeah. The previous model, GPT -5 .5, stopped
- 9:53working after just 35 minutes. Whoa. Imagine
- 9:56an AI staying actively focused on coding a single
- 9:59spaceship for 87 minutes straight without losing
- 10:02the plot. Two -sec silence. It is wild. Usually
- 10:06an AI's context window degrades over time. It
- 10:08starts hallucinating or writing bad code loops.
- 10:11Right. But Sol kept fixing its own physics engine
- 10:13bugs continuously. That level of sheer persistence
- 10:16is a huge leap forward. Yeah. It matters heavily
- 10:18when coding requires constant tedious checks.
- 10:21There was also a head -to -head Minecraft clone
- 10:23test. This is a massive undertaking for both
- 10:25AI models. Sol finished building the clone in
- 10:2870 minutes via Devon. Fable 5 took 90 minutes
- 10:31using Claude code. and in a completely different
- 10:33development environment. Different setups. So
- 10:35it is not a perfectly clean speed test comparison,
- 10:39but it still shows incredible stamina for both
- 10:41of these models. Definitely. But I have to ask
- 10:43about the actual output quality. Does a longer
- 10:46runtime, like that 87 -minute session, actually
- 10:49guarantee better code? No. Persistence is great
- 10:53for fixing immediate surface -level errors. But
- 10:57runtime alone does not prove the final code is
- 10:59clean. It does not mean the architecture is structurally
- 11:02reliable. It just means the AI keeps blindly
- 11:05trying to solve the problem. A bad architectural
- 11:08foundation will not improve just by running longer.
- 11:11Longer focus is great, but quality still needs
- 11:14a human check. Always. You can never fully remove
- 11:16the human element. The developer still has to
- 11:18verify the final code structure. Mid -roll sponsor,
- 11:21read placeholder. Those are massive developer
- 11:23tasks we just talked about. But how does Sol
- 11:25handle granular, everyday chat requests. What
- 11:29happens when you throw a daily research task
- 11:31at it? They tested Sol on researching conflicting
- 11:34evidence from different sources. This is a very
- 11:37common task for managers and developers today.
- 11:39Very common. They asked the AI a very specific,
- 11:42highly debated question. They asked if AI coding
- 11:45assistants actually speed up delivery. And they
- 11:48asked if they cause more production bugs over
- 11:50time. A simple web summary is definitely not
- 11:52enough for that. Saul had to check original evidence
- 11:55and explain the key differences. It successfully
- 11:58prioritized original academic studies over marketing
- 12:01fluff. It cited specific numerical claims from
- 12:04the actual GitHub commit data. It completely
- 12:07separated company blog posts from measured scientific
- 12:10findings. It also explicitly stated what the
- 12:14evidence cannot actually prove. That kind of
- 12:16nuance is incredibly valuable for deep research.
- 12:19Yes, it is. It does not just blindly agree with
- 12:21the user's initial prompt. It actively looks
- 12:24for public data sets and engineering reports
- 12:26to verify claims. Here's where it gets really
- 12:28interesting. Sol is also genuinely great at deep
- 12:31file analysis. You can upload complex spreadsheets
- 12:34or dense PDF reports. It finds data errors without
- 12:38guessing any missing values. Right. It completely
- 12:40refuses to confuse basic correlation with actual
- 12:44causation. It stays incredibly close to the facts
- 12:46presented in the file. It does not fill in the
- 12:49blanks with reasonable sounding assumptions.
- 12:51That is huge. That makes it a much safer tool
- 12:54for sensitive beta analysis. It clearly labels
- 12:57anything that is only a statistical possibility.
- 13:00They also tested Saul on rebuilding a basic visual
- 13:02chart. The original chart did not have any exact
- 13:05value labels. Sol rebuilt it into a clean, professional
- 13:08presentation slide. It added branding, footers,
- 13:12and a nice professional page number. It even
- 13:15moved the market labels so they did not cover
- 13:17data. The new version is much easier to read
- 13:20and compare. Wow. But there is a very crucial
- 13:23mechanical caveat here to remember. The original
- 13:26image lacked exact numerical values for the bars.
- 13:31OK. So Saul most likely visually estimated the
- 13:33bar heights. It was literally guessing numbers
- 13:35based on pixels and proportions. If it's just
- 13:38estimating chart values based on bar heights,
- 13:40can we trust the final slide it generates? Absolutely
- 13:43not for final reporting to your boss. You must
- 13:46manually verify every single number before putting
- 13:49it in a presentation. It is just estimating values
- 13:52based on visual pixels. It is not actually calculating
- 13:55the underlying financial data. It makes things
- 13:57pretty, but you still have to verify the math.
- 13:59Right. It is a design assistant, not your certified
- 14:02accountant. You still own the final numbers on
- 14:04that slide. So what does this all mean for you?
- 14:06We have pulled together all these benchmarks
- 14:08and wild X demos. We have looked at the daily
- 14:11chat interface tests too. We need to answer the
- 14:14ultimate question for you today. Which model
- 14:17should you actually choose for your daily workflow?
- 14:21Well, if we connect this to the bigger picture,
- 14:24you should absolutely choose GPT 5 .6 Sol for
- 14:27a few specific things. OK. Use it for deep web
- 14:30research involving multiple conflicting sources.
- 14:33Use it for tasks requiring heavy tool use and
- 14:36terminal commands. It is phenomenal for long
- 14:38tasks with incredibly clear success criteria.
- 14:41And use it when you can check the work step by
- 14:43step. Sol is clearly built for rigorous step
- 14:47-by -step execution. It follows the precise path
- 14:50you set out for it perfectly. But you should
- 14:52choose Fable 5 for different reasons entirely.
- 14:55Use Fable 5 for open -ended planning and massive
- 14:57repository -level coding. It is better for tasks
- 15:01where deep judgment matters more than speed.
- 15:03Fable 5 has a much stronger sense of broader
- 15:06direction. It holds that structural blueprint
- 15:08in its context window. It understands what should
- 15:11logically happen next in a complex system. It
- 15:14is the much safer option for code -based work
- 15:16needing real judgment. Sol executes the specific
- 15:19steps. Fable 5 decides the overall direction.
- 15:22We want to leave you with a final thought today.
- 15:25Sol is becoming an absolutely unstoppable doer
- 15:28for software developers. It could work for 87
- 15:31minutes straight without ever quitting. Yeah,
- 15:33it's wild. Meanwhile, Fable 5 is becoming a master
- 15:36planner for system architecture. It can handle
- 15:39vast complex code bases with deep human -like
- 15:42judgment. Beat. What happens when developers
- 15:44figure out how to seamlessly chain these two
- 15:46models together? Oh, man. Imagine Fable 5's architectural
- 15:50planning guiding Sol's relentless 87 -minute
- 15:53execution. Yeah. That is when things will get
- 15:56truly wild. We might see completely autonomous
- 15:59software engineering teams very soon. Thank you
- 16:02for taking this deep dive with us today. We hope
- 16:04this helps you choose the right tool for your
- 16:06workflow. We will see you next time. Out to your
- 16:08own music.