Latest / In The Loop / Everything You Need To Know About GPT-5.2 In 10 Mins

Transcript

Everything You Need To Know About GPT-5.2 In 10 Mins · In The Loop

• 0:00
  So as we predicted last week, Open Eye just brought out a new
• 0:03
  language model called GPT 5.2. What we described was something
• 0:08
  called GPT Garlic, which is their code name for what is now
• 0:12
  called 5.2. There's a lot of noise about it,
• 0:14
  especially because since writing and researching this episode and
• 0:18
  testing out the new model, they brought out another language
• 0:21
  model or a language model improvement, which was their
• 0:24
  image model generation on top of their 5.2 model.
• 0:28
  So for this episode, we're going to try and keep it short and
• 0:30
  sweet for the final episode of 2025.
• 0:33
  We're just going to break down exactly what it means for you as
• 0:36
  an everyday user and the things you should be looking out for.
• 0:39
  Now, this isn't a loop with Jack Horton.
• 0:42
  I hope you enjoyed the show. Where to start off, it feels
• 0:57
  like I am dragging myself to the end of this year.
• 1:00
  It has been an incredible year, but it's been incredibly tiring,
• 1:04
  especially this last month. So I was going to be the last
• 1:08
  episode of the year. I'm looking forward to a couple
• 1:10
  weeks break and we'll be coming back better and stronger in
• 1:14
  2026. So for context, on this episode,
• 1:18
  in early December, Sam Altman issued what was called a Code
• 1:22
  Red, which is what last week's episode really discussed in
• 1:25
  detail. And obviously we went through
• 1:27
  the memo he shared and the real problems that Open AI really
• 1:33
  face over the next 6/12/24 months.
• 1:37
  And as I said, we last week predicted that they were going
• 1:40
  to bring out a new language model and it was codenamed at
• 1:43
  the time GPT Garlic, which is now called GPT 5.2.
• 1:59
  And to cut to the chase, really, I think actually in many ways
• 2:03
  they've matched a lot of Gemini's biggest model updates.
• 2:07
  Whether that's something that people feel in the press, feel
• 2:11
  in the market, I can't really say.
• 2:13
  I don't think so yet. But from a benchmark perspective
• 2:17
  and an image generation model perspective, they've gone above
• 2:20
  and beyond what I to be honest, expected.
• 2:23
  So GPT 5.2 comes in three versions.
• 2:26
  Instant is their super fast one. As always, no thinking mode,
• 2:30
  just rapid responses. What they've called thinking is
• 2:33
  their standard model. As you come to expect from most
• 2:35
  models these days, it has reasoning.
• 2:38
  So it pauses, works through the problem, and then responds and
• 2:40
  it creates a plan for itself. And now you've got Pro extended
• 2:45
  thinking. So Matt Schumer, someone online
• 2:47
  that there's a lot of commentary on the space, had access to this
• 2:51
  model since November 25th, testing it out and apparently it
• 2:55
  was thinking for over an hour on some hard problems.
• 2:59
  But obviously Pro is only available on the $200 a month
• 3:03
  subscription and frustratingly for many businesses, it's not
• 3:06
  available via API. There's also a new reasoning
• 3:10
  effort setting within the thinking mode, so you have
• 3:13
  standard high or extra high, so that obviously the higher you
• 3:16
  set it, the longer it thinks and obviously you'd expect the
• 3:18
  better the output to be. From a technical perspective,
• 3:22
  the context window is now 400,000 tokens.
• 3:26
  To put that into context, GPT 5.1 had a context window of
• 3:29
  about 128,000. So it's a big substantial uplift
• 3:33
  here. And for those going, what the
• 3:35
  heck is context windows? That is the amount of
• 3:38
  information you can give to it and it be able to have within
• 3:43
  its context. When it answers questions or
• 3:45
  does tasks for you before it kind of gets poor and stupid and
• 3:49
  annoying as you can often find it doing or it just says sorry
• 3:52
  limit reach, move to a new chat. There's also now an auto setting
• 3:56
  that's supposed to and better choose between either instant or
• 4:00
  extended thinking. I'd recommend ignoring it.
• 4:02
  From what I've read online and some limited testings I've
• 4:05
  actually got rid of recently my ChatGPT license.
• 4:08
  The model often would just automatically think for a couple
• 4:10
  of seconds and often get an answer that is poor or wrong.
• 4:13
  And often you're going to need thinking for most professional
• 4:16
  work. The second big area of
• 4:17
  improvement is the area of professional work and activities
• 4:21
  and output. So let's talk spreadsheets and
• 4:23
  presentations and things like that because this is where
• 4:26
  Opening and Eye have clearly put a lot of their marketing energy
• 4:29
  into. And honestly, there's real
• 4:31
  improvements made here. Another early access user called
• 4:34
  Simon Smith said this is the first time that ChatGPT had made
• 4:37
  spreadsheets and presentations that were actually presentable.
• 4:40
  You know, on presentations, the YouTube reviewer Skill Leap gave
• 4:44
  a web link and asked it to create a full slideshow.
• 4:47
  And it took 28 minutes. But the output for him was
• 4:50
  really, really impressive. Really good layouts, information
• 4:53
  pulled correctly, slides that looked really professional and
• 4:57
  his words were that it was shockingly good compared to 5.1
• 5:00
  and another tester threw 10,000 rows of spreadsheet data and
• 5:04
  told it to make a PowerPoint with that and it created a
• 5:06
  really good set of slides. For those of you who have to do
• 5:09
  this sort of task all the time, this sort of thing is actually
• 5:12
  music to your ears. However, there are many
• 5:14
  fantastic tools like Gamma that do much the same thing.
• 5:17
  Now open AI is benchmark for this is called the GDP Val
• 5:21
  measuring essentially well specified knowledge work tasks
• 5:24
  across over 44 occupations. And they claim that 5.2 thinking
• 5:28
  mode ties on human expertise around they claim that 5.2
• 5:33
  thinking mode beats or at least ties with human experts on 70.9%
• 5:38
  of comparisons, up from 38.8%. So imagine it creating a task
• 5:43
  and having experts review it. Essentially that's 70% of the
• 5:47
  time it's either better or on par with humans.
• 5:50
  Now, I think there's nuances here.
• 5:52
  You know, well specified is doing a lot of work in that
• 5:56
  sentence. You know, it means a model gets
• 5:58
  handed everything it needs upfront.
• 5:59
  So super clear instructions, all the relevant context, define
• 6:04
  success criteria. And I think real professional
• 6:07
  work just isn't like that a lot of the time.
• 6:10
  You know, you often have to figure out what information it
• 6:12
  needs, you need to go find it. You need to make good judgement
• 6:15
  calls, you need to make good prompting.
• 6:18
  So that benchmark covers, I guess, well specified knowledge
• 6:22
  work and the fact that they're giving the perfect prompt to do
• 6:25
  that task to the language model, whereas most people are not
• 6:28
  giving such a well thought out structured prompt to a language
• 6:32
  model. So yeah, 70.9% doesn't mean that
• 6:35
  GPT 5.2 can certainly do 71 percent of a person's job.
• 6:39
  It means for tasks where everything is perfectly
• 6:42
  articulated, defined and essentially handed to a model on
• 6:46
  a plate, then it will perform at expert level most of the time.
• 6:50
  A third big areas of improvement has been cogeneration.
• 6:53
  As I said last last week, they're not trying to win in the
• 6:56
  code arena clearly. That being said, they've still
• 6:59
  made fantastic leaps in coding. Maybe they were working on
• 7:03
  making in a better coding model, realise they'd made a better
• 7:05
  model generally and just chucked it out there because most of
• 7:09
  their marketing is focused on professional work.
• 7:12
  On the SWE Bench Pro, which is a benchmark testing software
• 7:17
  engineering across basically four different languages for
• 7:20
  programming and 5.2 thinking scored 55.6% which is really a
• 7:26
  new state-of-the-art. On another benchmark that's very
• 7:28
  similar on called SWE benchmark verified it hit 80% essentially
• 7:33
  matching cords best model which was 80.9%.
• 7:36
  Another area of improvement has been vision and long context.
• 7:40
  So it's vision capabilities have improved a lot, you know, on
• 7:44
  chart understanding from scientific papers, for example,
• 7:47
  it's accuracy jumped from 80% to 88% on user interface
• 7:51
  understanding. So again that kind of agent mode
• 7:53
  of reading your screen and making clicks, it jumped from
• 7:56
  64% to 86% and error rates have been cut in half.
• 8:01
  On context windows, there's been a massive improvement as I
• 8:03
  mentioned, with 5.1 accuracy started to degrade as the amount
• 8:08
  of information you had given it throughout conversations or
• 8:11
  system prompts for those using APIs got a lot longer, around
• 8:15
  90% at about 8000 tokens, dropping to under 50% at about
• 8:20
  256,000 tokens. Whereas with GPT 5.2, accuracy
• 8:26
  stays at almost 100% across the entire context window, even when
• 8:30
  it's almost maxed out. This is one of the first models
• 8:33
  ever to actually achieve near perfect accuracy levels on the
• 8:38
  four needle challenge, which is essentially recalling 4 specific
• 8:41
  piece of information scattered across 200,000 words.
• 8:45
  Another major improvement here has been hallucinations.
• 8:47
  So Open AI claims that they've reduced hallucinations by 30%
• 8:52
  from 8.8% in 5.1 to 6.2% in 5.2. However, more independent
• 8:58
  benchmark reviews have, let's give, let's say given them more
• 9:01
  modest scoring. So Vectara found that GPT 5.2
• 9:06
  had an 8.4% hallucination rate, which trails DeepSeek at 6.3%.
• 9:12
  So a massive improvement, but still not a leading model.
• 9:15
  Now let's move on to still things that are still not very
• 9:17
  good. Speed is a still a real problem.
• 9:20
  You know, Matt Schumer said the standard 5.2 thinking is just
• 9:23
  extremely slow. You know, very, very, very slow
• 9:26
  for most questions, even straightforward ones, which
• 9:28
  changes how he works. So quick questions means that
• 9:32
  he'd go to Claude Opus. Deep reasoning now goes to 5.2
• 9:36
  pro, which is quite interesting because it used to be or
• 9:39
  certainly has been for me the other way around.
• 9:41
  However, for those who do, a lot of writing quality still lags
• 9:44
  behind Claude. Dan Shippers publications called
• 9:47
  every ran a load of systematic tests and found that Claude Opus
• 9:51
  4.5 scored 80% in writing quality, whereas GPT 5.2 scored
• 9:56
  74%, So still behind Claude. And many, many testers also
• 10:01
  noticed big personality changes. So Ali Miller, another big
• 10:04
  commentator in the AI space, said that a simple question
• 10:07
  turned into 58 bullet points and numbered points.
• 10:11
  And and then many people have been comparing 5.2 to a, a
• 10:15
  brilliant freelancer. So yeah, as you know with me,
• 10:18
  I've, I've said over and over again that benchmarks really
• 10:19
  are, are not the be all and end all.
• 10:23
  You know, comparing models is getting much harder and
• 10:26
  performance is scaling very iteratively.
• 10:28
  So as always, it's important to go and test this thing out and
• 10:31
  see for yourself and experience it and see if you like the
• 10:34
  improvements. But yeah, to conclude, I think
• 10:47
  opening eyes messaging on this release has been very focused,
• 10:51
  which is unusual because often they've been we're the best of
• 10:54
  everything for everyone or would you know that very scatter good
• 10:57
  messaging. Now, every single executive and
• 10:59
  interview and media has really focused on professional work for
• 11:03
  this release and economically valuable tasks.
• 11:06
  So they're clearly not trying to claim AGI breakthroughs, but
• 11:09
  they are trying to win at professional work, clearly going
• 11:13
  for the enterprise market. And obviously, as I said, the
• 11:15
  improvements are very real structured outputs.
• 11:17
  It's the most capable model GPT has ever produced.
• 11:21
  If your tasks involve slides and spreadsheets, then 5.2 is a
• 11:25
  serious upgrade. But I think if you zoom out, as
• 11:28
  I mentioned, you're seeing incremental progress, not
• 11:31
  massive leaps anymore. I think some people still hope
• 11:34
  for this big flash of inspiration, you know, one model
• 11:37
  that conquers it all. But what we're starting to see
• 11:39
  is a pattern of that not being quite true.
• 11:42
  So 5.2 is a much better tool, but it's not a new era.
• 11:46
  And the fact that Open AI is marketing better spreadsheets, I
• 11:49
  think, tells us a lot about where we are with AI right now.
• 11:53
  Anyway, that's it for this week and also as I said, for this
• 11:56
  year. I hope you find this episode
• 11:58
  interesting and I look forward to spending 2026 with you.
• 12:02
  Thank you and see you next year.