Latest / In The Loop / GPT-5 Review: Everything You Need To Know

Transcript

GPT-5 Review: Everything You Need To Know · In The Loop

• 0:00
  Welcome back to In the Loop and after what feels like years of
• 0:04
  waiting, GPT 5 is finally here. So about a week on opinion is
• 0:10
  quite divided. You've got many people saying
• 0:12
  this is revolutionary, but just as many people saying it's
• 0:15
  disappointing. You had on the launch event open
• 0:18
  AI showing bar charts that were just straight out lies.
• 0:22
  And you have prediction markets betting against open AI just
• 0:25
  minutes after their launch. So if you're sat here thinking,
• 0:28
  I have no idea what's going on, well, that's probably the only
• 0:32
  reasonable response you could have.
• 0:34
  In today's episode, you're going to learn everything that is
• 0:36
  worth knowing about this new model release, what's new,
• 0:39
  what's improved and what was just a lot of good marketing.
• 0:43
  We've spent many hours collecting all the research and
• 0:45
  reviews so you don't have to. This is in the leap of Jack
• 0:49
  Horton. I hope you enjoyed the show.
• 1:03
  So before I get into the episode, I think it's important
• 1:05
  to set the scene and add just a little bit of context to this
• 1:09
  model release. For the past year, we've been
• 1:13
  living in basically model chaos. It feels like Open AI has been
• 1:17
  throwing spaghetti at the wall when it comes to language
• 1:20
  models. We've had GPT 44.5, the O series
• 1:24
  with different specialised models, and much of it felt
• 1:28
  sometimes like they were trying to justify a $200 per month
• 1:31
  price tag for their pro membership.
• 1:33
  And people were often so confused about which model they
• 1:36
  were going to use for this specific task of theirs that
• 1:39
  they had to even release white papers on it.
• 1:41
  And meanwhile, Anthropic has been completely eating away at
• 1:45
  their enterprise market. So particularly in coding, we've
• 1:49
  covered this in a number of episodes, but called code and
• 1:53
  Anthropic Clod language model has become the go to choice for
• 1:56
  developers across the world. And I guess the general take
• 1:59
  away has been quite clear, which is most people would go to
• 2:02
  ChatGPT for general use. But for most people, if you're
• 2:05
  going to go do a coding task, you're going to go and use clod.
• 2:08
  And in this context frames the release of GPT 5 really well.
• 2:13
  This release wasn't just a better model, it was all about
• 2:17
  cleaning up this chaotic mess of different language models and
• 2:21
  trying to take back market share that they've lost over the last,
• 2:24
  I'd say 69 and 12 months to clock.
• 2:27
  Now with that being said, let's get into it.
• 2:28
  And I think it's going to be really helpful to start with a
• 2:31
  high level overview of some of Chachi BT5's new features and
• 2:35
  abilities. So one of the first things
• 2:36
  you're going to notice since using the new Chachi BT5 model
• 2:39
  is that it's called a hybrid model.
• 2:41
  And the reason this is noticeable is because it's got
• 2:43
  both thinking and non thinking versions built into the exact
• 2:46
  same model. So previously if you asked a
• 2:48
  question, unless you specified the O series or the thinking
• 2:52
  model, you wouldn't have any reasoning and and thinking in
• 2:55
  any of the responses that you get from it.
• 2:57
  You'd have to specifically choose a reasoning model.
• 2:59
  And for those on a free tier, you've only got a limited number
• 3:02
  of queries on that. And in fact, initially, they
• 3:04
  actually called to deprecate many of their former models, so
• 3:08
  they completely took away Chachi, BT 44.1, the entire O
• 3:12
  series of models. There's been such a big uprising
• 3:15
  about this that Sam Altman's ended up releasing what is
• 3:18
  basically a small essay of tweets and open AI, actually
• 3:22
  bringing back many of the models that people loved into the
• 3:26
  subscription package. For those paying $200 a month,
• 3:28
  this could be an episode on its own just due to the amount of, I
• 3:32
  guess, emotional attachment that people have clearly had to some
• 3:35
  of these models and the fact that it was then taken away from
• 3:38
  them caused real upset online. Another new addition is that
• 3:42
  you're going to now be able to short circuit it's thinking when
• 3:45
  it goes into a thinking loop for a question that to be honest,
• 3:47
  was quite simple. It's also going to be available
• 3:49
  to everybody, so including all free users.
• 3:51
  I think they really learned from the Deep Sea Saga last year that
• 3:55
  everyone must experience their best models.
• 3:58
  And there are also three versions of Chat Chi BT5.
• 4:01
  We have the standard one, the Mini 1 and the Nano.
• 4:05
  So if you then hit usage limits on one model, it will
• 4:07
  automatically switch to another model.
• 4:09
  Also, it's a massive context window, which is for me a huge
• 4:13
  improvement. It's context window is now
• 4:14
  400,000 tokens, which is larger than all previous models.
• 4:18
  And if I put that into context, that's about 300,000 words.
• 4:23
  And again, if you listen to the episode on context engineering a
• 4:25
  few weeks ago, context is the ability for the agent to have a
• 4:30
  set of information when you ask it a question.
• 4:33
  So it could be a long part of a copy and paste of information.
• 4:37
  In document terms, that's about 600 to 800 pages of text it can
• 4:40
  keep in its short term memory when you're having a
• 4:44
  conversation with it. And previous models were about
• 4:46
  128,000 tokens, so that's 96,000 words.
• 4:49
  So that jumped from 96,000 words of information it can store in
• 4:53
  its memory when you're having a conversation with it to now.
• 4:55
  300,000 is a huge, huge leap. There's nothing more than more
• 5:00
  annoying than putting information into a model and
• 5:03
  hitting a wall because of a context of it.
• 5:06
  Another huge addition has been coding and front end creation
• 5:10
  just using ChatGPT. They've clearly gone all out to
• 5:13
  also try and take Liverpool's market share.
• 5:16
  It now has a really good understanding of spacing and
• 5:18
  typography and even white space and design.
• 5:21
  You can vibe code entire user interface really, really simply.
• 5:26
  And in fact, in the actual press release, which I really highly
• 5:29
  recommend you go and check out, they provide a load of prompts
• 5:32
  that can generate really cool applications.
• 5:35
  And this is the exact reason that Sam Altman recently came
• 5:38
  out and said that this is the fast fashion era of technology.
• 5:41
  We'll get into this a little bit more throughout the episode, but
• 5:44
  you want to keep it top of mind. They've also really lowered the
• 5:47
  barrier of price. It's now much, much cheaper.
• 5:49
  You're paying about $10 per 1,000,000 tokens.
• 5:52
  And output tokens, by the way, is I guess is the number of
• 5:56
  words it uses when it responds to you.
• 5:59
  For the same 1,000,000 tokens that you pay $10.00 for with
• 6:02
  Open AI, you're actually having to pay $75.00 for Opus 1.
• 6:05
  So that's Claude's good model. So that pricing is absolutely
• 6:08
  crazy. It's 10 times cheaper than
• 6:10
  Claude, which is, to be honest, really difficult to say no to.
• 6:14
  And it's actually been reported that API usage for companies
• 6:18
  using ChatGPT inside their application or existing
• 6:21
  workflows and systems has doubled in the last few days of
• 6:25
  the ChatGPT launch and kept on growing subsequently after that
• 6:30
  launch. It's been reported that the
• 6:32
  usage of Open AISAPI, so that's using the language models inside
• 6:36
  existing software actually doubled within just the first 48
• 6:39
  hours of ChatGPT 5 coming out. This is clearly an aggressive
• 6:43
  pricing strategy designed to win back market share, especially in
• 6:46
  that coding space where Anthropic has grown rapidly.
• 6:50
  And finally, they've launched a what they described as a unified
• 6:53
  system. But that basically means that
• 6:54
  they've added a router, which quickly decides which model is
• 6:58
  going to be used to perform that task.
• 7:01
  And that's going to be based on the complexity of the tool
• 7:04
  types. So if it needs to do a web
• 7:05
  search tool or maybe the type of question that's been asked, all
• 7:09
  of that sounds really cool. And to be honest, the main area
• 7:12
  they've invested in clearly is the coding space.
• 7:15
  And to be honest, if you just watched their entire launch
• 7:18
  video, if you had no context on what Open AI did, you would
• 7:22
  probably assume that they are predominantly A coding
• 7:25
  application. So as we go into what the
• 7:38
  independent benchmarks say, I think that's a really important
• 7:41
  thing to keep top of mind. Now, as you will know, I'm not a
• 7:44
  massive fan of these benchmarks, to be honest.
• 7:48
  I think it's a game that all of them play and they're all
• 7:50
  obviously going to try and make each other look as best they
• 7:53
  possibly can on launch day. But I think it helps gives us, I
• 7:56
  guess, an initial gauge of whether there's been a big leap
• 7:59
  or not. Artificial analysis, which is
• 8:00
  the gold standard for independent AI benchmarks, found
• 8:03
  that when ChatGPT was tested and it used its reasoning, it was
• 8:09
  the best model out there. However, when GPT 5 didn't have
• 8:12
  reasoning, it performs closer to GPT 4.1's level.
• 8:17
  This part is because actually what it shows us is again, the
• 8:21
  open air isn't selling US1 model.
• 8:23
  They're selling an entire new packaging and pricing system.
• 8:27
  This minimal version that most people, to be honest, will often
• 8:31
  get is basically just their old models, and their high reasoning
• 8:36
  version that gets the incredible flashy headlines about coming
• 8:39
  first place in the most important benchmarks is going to
• 8:43
  be rarely used. So through very clever pricing
• 8:46
  and packaging, they're trying to increase the perception of value
• 8:49
  across all their tiers. And if you think about what that
• 8:51
  means for say, competition, Anthropic might charge premium
• 8:54
  prices for Clawed because they position it as consistently high
• 8:58
  quality. But Open AI has built a new
• 9:00
  packaging and pricing system that enables them to undercut
• 9:04
  Clawed and other model providers, all by using that
• 9:08
  router system, which is automatically routing people to
• 9:11
  different models. And subsequently, Open AI has
• 9:13
  announced that they're going to bring out the ability for you to
• 9:16
  understand which models being used more often.
• 9:18
  Which for me is a really big win because I personally haven't
• 9:22
  been trusting the quality of responses.
• 9:25
  And I can't help but think, am I getting the rubbish models and
• 9:28
  still paying money for it? When it comes to coding
• 9:30
  benchmarks, Vellum's analysis found that Grok 4, GPT 5, and
• 9:34
  Cloud Opus are all performing a very similar level on the SWE
• 9:38
  bench, though ChatGPT did come first place in many of the other
• 9:43
  coding benchmarks like Design Arena.
• 9:45
  And so even though ChatGPT has led to the number one best
• 9:48
  performing model in many coding benchmarks, Group 4, So Elon
• 9:53
  Musk's language model still actually gets the best marks on
• 9:58
  some of the hardest types of assessments on the ARC AGI two
• 10:01
  test, which is designed basically to measure abstract
• 10:04
  reasoning. That's more like human problem
• 10:06
  solving, the kind of figure it out yourself thinking that
• 10:10
  humans have to do when they've never actually seen a specific
• 10:13
  problem before. Well, Grokforce scores highest
• 10:16
  on that assessment at 15.9%, while ChatGPT 5 actually lags
• 10:21
  behind it, less than 10%, so 9.9%, which is a huge gap.
• 10:26
  So in short, ChatGPT 5 is faster, it's better at coding,
• 10:30
  it's cheaper, but across many of the reasoning benchmarks, it's
• 10:35
  not as good. That's just the benchmarks.
• 10:47
  And as we all know with language models, it's just as much about
• 10:50
  the vibes. And certainly there's been a lot
• 10:52
  of noise and conversation about those vibes with ChatGPT 5, and
• 10:55
  that's included very divided opinions.
• 10:57
  So on Polymarket, which is a decentralised prediction market.
• 11:00
  So you can bet on whether Apple's going to bring out a
• 11:02
  foldable phone in 2026. They actually placed 80% odds on
• 11:08
  Open AI having the best leading AI model by the end of August
• 11:11
  before the demo. And then Google was lagging
• 11:14
  behind it, just 20%. But by the time Sam Altman and
• 11:18
  the Open AI team ward off that stage, the title completely
• 11:22
  turned and Google surged to 85% and Open AI to 14%.
• 11:27
  So that only 14% of betters believe that Open AI will have
• 11:31
  the best model in 2026 from 80%, 14% and this speaks volume on
• 11:37
  the divided opinion that people have got around this new model
• 11:40
  release During the ChatGPT 5 release, there was also let's
• 11:44
  call it chart crime Open AI showed off several charts on the
• 11:48
  screen as they spoke to demonstrate how good ChatGPT 5
• 11:52
  really is and it honestly left me baffled.
• 11:54
  So it's hard to explain, but if you want a podcast platform that
• 11:57
  has video will actually show the bar chart because it's it's
• 11:59
  hilarious. They had bars that were next to
• 12:03
  each other but drastically different percentage outcomes.
• 12:07
  So we had one bar that got a 69% score on coding deception,
• 12:11
  another one next to it that got 30.8, but they were displayed as
• 12:14
  identical bar heights, while the Chachi BT5 results were 52.8%.
• 12:20
  Yeah, was visually showed as being much, much bigger than the
• 12:25
  other bars that were at 69% and 30%.
• 12:28
  It's criminal. Someone addressed it afterwards
• 12:31
  and said it was a mega chart screw up from them, but I guess
• 12:34
  many people were claiming that they vibe coded it using
• 12:37
  ChatGPT. You have Professor Ethan Mollick
• 12:39
  predicting that you're going to see a lot of varied results
• 12:42
  posted online about chat GB 25 because it's using multiple
• 12:45
  models in its responses, some of which are very good and some of
• 12:49
  which are, to be honest, meh. And since the underlying model
• 12:52
  selection isn't very transparent, it becomes very
• 12:54
  confusing. And this is exactly my
• 12:56
  experience so far. They also had to do a massive
• 12:59
  patch just after their release because usage went through the
• 13:02
  roof and everything started to fail, including that auto
• 13:04
  Switcher. So switching to different models
• 13:07
  and as a result, it just started putting every single person on
• 13:09
  the worst possible model. This blog by Matt Schumer, I
• 13:12
  think, captures the broader market reaction really, really
• 13:15
  well. And he said, when I started
• 13:17
  using it, I wasn't blown away. And in fact, I felt very let
• 13:19
  down given all of this hype and a massive expectation around it.
• 13:23
  But what Schumer said was that if you're using Chachi BT5 for
• 13:27
  tasks that the previous O3 models or even the Cloud Sonic
• 13:31
  can do really well, then you're not really going to get a gauge
• 13:35
  of how good it is to see how good this truly is.
• 13:38
  You have to ask it to do things that AI simply can't do very
• 13:42
  easily. And in short, that is coding.
• 13:45
  And I think this gets the meat of what open AI I've clearly
• 13:47
  obsessed over getting right, because the vibe coding
• 13:50
  revolution is here, and that means it's the era of creating
• 13:55
  things all of the time. Before I go into some of the
• 13:57
  examples of what's been built on ChatGPT 5, I just want to
• 14:01
  quickly stop and acknowledge that we're growing incredibly
• 14:04
  fast. And according to the stats, only
• 14:06
  45% of you listening right now are following us or are
• 14:10
  subscribed. So if you could press that
• 14:12
  button on your podcast app, that would help so, so much.
• 14:16
  A shed load of effort goes into every episode and that number is
• 14:20
  part of what makes it worth it. So the Vive coding revolution
• 14:22
  really is here. And Matthew Burnham, who has
• 14:24
  480,000 subscribers on YouTube, actually focuses all on AI
• 14:30
  coding and he got access to the model quite early and he built a
• 14:33
  bunch of really, really cool things.
• 14:35
  So you had ChatGPT 5 generate several very specific
• 14:38
  application to test its vibe coding abilities.
• 14:41
  Again, if you're watching the video, we'll put them on the
• 14:43
  screen. But he made a jumping ball
• 14:45
  runner game. So a simple game that although
• 14:48
  is very basic, was functional. He created a pixel art tool, so
• 14:52
  an entire application that you can draw things on and it can
• 14:55
  change colours. He said it was a bit laggy, but
• 14:58
  not too bad. There was also a game he
• 15:00
  generated which measures your typing speed, so which is a
• 15:03
  functional typing test that essentially tracked the accuracy
• 15:06
  and speed of you typing out words.
• 15:08
  The game even included proper feedback and scoring systems.
• 15:12
  He also created a drum simulator, which is an
• 15:14
  interactive music application that you could literally make
• 15:17
  music on and finally a tool that visualises music and lo fi,
• 15:21
  which is also just really, really cool.
• 15:23
  So the fact that you don't need to go to another application
• 15:25
  now, ChatGPT literally does this for you from a single prompt.
• 15:30
  And this is the world of vibe coding.
• 15:31
  Every application is now a vibe coding application.
• 15:34
  I think another fantastic review and one that I wanted to know
• 15:36
  because I really, really enjoyed the reading was won by the
• 15:39
  Leighton Space and it was one of the more nuanced and
• 15:43
  philosophical pieces on ChatGPT 5 and they called it the Stone
• 15:47
  Age. This wasn't a criticism.
• 15:49
  This is a reflection of where they think we are in human
• 15:52
  civilisation right now, because this is one of the first model
• 15:55
  that excels at using tools during conversations, just as
• 16:00
  humans did a long, long time ago.
• 16:02
  So as they put it, the Stone Age marked the dawn of human
• 16:05
  intelligence, but what exactly made it so significant is the
• 16:09
  fact that we shaped tools and our tools shaped us.
• 16:21
  So I guess what does all this mean for Anthropic?
• 16:24
  So the threat to Anthropic is really interesting because
• 16:26
  apparently according to reports both Cursor and GitHub copilot
• 16:31
  drive approximately $1.2 billion of Anthropics 4 billion revenue.
• 16:36
  So that is just two applications built and using clawed every
• 16:40
  single day as part of their application that is driving most
• 16:42
  of their revenue while 30% of it anyway.
• 16:45
  And with Chat GPT's new model coming out and it's incredible
• 16:48
  abilities with coding and costing 7 times less than Claws
• 16:53
  models, well, it's going to be hard for an enterprise customer
• 16:56
  to say no. And lots of coding applications
• 16:58
  like Livable Cursor and others have made ChatGPT 5 their
• 17:02
  default model again because Claude has brought out Claude
• 17:06
  Code, which is actually a competitor to many of the
• 17:09
  companies that are using their language model.
• 17:12
  Many of those companies are now jumped straight to Open AI to
• 17:15
  make it their default model. This doesn't mean that
• 17:18
  developers will not choose Claude still, but it's certainly
• 17:21
  a massive risk to Anthropic. So I guess to conclude, ChatGPT
• 17:35
  represents both impressive technical achievements and
• 17:40
  really importantly, very aggressive competitive
• 17:43
  strategies against Anthropic. For coding, it appears to have
• 17:47
  made a huge leap forward, but for other applications the
• 17:50
  improvements feel incredibly incremental and to be honest,
• 17:53
  overall disappointing. This was just as much of A
• 17:56
  business model change disguised as technical leaps because Open
• 18:01
  AI have just built a system where they can offer cheaper
• 18:04
  language models to everybody with minimal reasoning, while
• 18:07
  still managed to claim that they have the very best model on the
• 18:10
  market. And obviously it's working.
• 18:12
  Cursor switched overnight, prediction markets lost
• 18:15
  confidence in Open AI and Anthropic is watching 30% of
• 18:18
  their revenue become potentially vulnerable to their pricing.
• 18:22
  And I guess anecdotally, the chart crimes, the very mixed
• 18:26
  reviews, the expectation gaps are probably a symptom of now a
• 18:29
  maturing market where these revolutionary AI narratives just
• 18:35
  don't quite work in the same way as they used to.
• 18:37
  The real story here is where the ChatGPT can write better code,
• 18:41
  write better content, and solve real problems for everyday
• 18:45
  people. Anyway, that's it for today.
• 18:47
  I hope you enjoyed that deep dive into ChatGPT 5.
• 18:51
  Thanks for listening and I will see you next week.