Latest / In The Loop / Five Things GPT-5.1 Does Better—Vibes Over Benchmarks

Transcript

Five Things GPT-5.1 Does Better—Vibes Over Benchmarks · In The Loop

• 0:00
  For this week, I am incredibly pleased to finally do an episode
• 0:03
  which isn't about macroeconomics and the meta state of AI in the
• 0:09
  world because of potential economic bubbles and moving on
• 0:12
  to something that's a bit more timely and relevant just this
• 0:15
  once. And this week we're going to be
• 0:16
  talking about GPT 5.1. So for those that haven't yet
• 0:22
  heard, Open AI just release a new language model.
• 0:25
  GBT 5.1 was released three months after their big flagship
• 0:29
  release of Five Point O, which is their fastest model upgrade
• 0:34
  so far. Three months in isn't a very
• 0:35
  long time, and they've released it with no benchmark data,
• 0:39
  graphs and charts showing how good it is versus other models.
• 0:42
  There's no technical metrics, no performance metrics, and what
• 0:46
  they're actually doing is now selling vibes.
• 0:49
  So yeah, the company that's built its reputation on big
• 0:52
  fancy charts and how clever it is compared to a PhD student is
• 0:57
  now selling quite literally, vibes.
• 0:59
  And I think the shift is going to tell us something about where
• 1:02
  AI is heading right now. And that is going to be the big
• 1:05
  topic of today's episode. We're going to be covering the
• 1:07
  five things Chachi BT 5.1 actually does better than its
• 1:11
  previous model and why it matters.
• 1:13
  This is in the loop with Jack Horton.
• 1:15
  I hope you enjoy the show. So First off, Chachi BT 5.1,
• 1:35
  what does it do better? It turns out it actually follows
• 1:37
  instructions. So if you ask it to respond with
• 1:42
  only 6 words, it will actually respond in just six words.
• 1:46
  For those that have tried to do any form of work with ChatGPT 5,
• 1:49
  you may have hit a wall of frustration.
• 1:52
  When you ask it to do very simple things.
• 1:53
  It says it's going to do that and seems to completely ignore
• 1:57
  you. Write this thing and make sure
• 1:59
  it has a word limit of say 500 and it comes out with an 8000
• 2:03
  word essay. Stop writing in short sentences
• 2:06
  and it continues to use short sentences and open AI releases
• 2:10
  prompting guide. So it's basically it's rules for
• 2:12
  how to work with ChatGPT 5.1 and in its prompting guide, it's
• 2:17
  asked the model to make assumptions about any missing
• 2:20
  details and do not ask the user for any clarification unless
• 2:24
  absolutely necessary. So it's going to be very
• 2:26
  decisive rather than trying to gather information from you,
• 2:31
  it's going to assume you already know what you want it to do and
• 2:35
  correctly prompted it to do that thing.
• 2:37
  So when you're really clear about what you want it to do,
• 2:38
  that's going to be fantastic. When you're kind of context
• 2:42
  dumping, it's not going to be that helpful.
• 2:44
  Number 2 is decisiveness. So, so GPT 5.1 is going to try
• 2:49
  and commit to recommendations more and more.
• 2:51
  So previous models often hedged. They said, on the one hand, do
• 2:55
  this, but on the other hand, this.
• 2:57
  And now it's just going to try and make recommendations much
• 2:59
  faster. In fact, the prompting guide by
• 3:01
  Open AI says quite literally have a bias for action.
• 3:05
  So there's been lots of experiments online on Twitter or
• 3:08
  actual I say, and people have managed to get it to write 2425
• 3:12
  word reports on strategies for improving supply chains just
• 3:16
  from a single prompt. Now personally, that's not
• 3:18
  really something I like, but I guess many people do.
• 3:22
  And really this is where vibes over benchmark starts to get
• 3:25
  quite interesting. A company called Surge AI ran an
• 3:29
  evaluation and found that in that study, 48% of people
• 3:33
  preferred a G PT4-O style language model that asked
• 3:38
  questions that didn't just assume you knew what you wanted
• 3:42
  and tried to help and guide you to the decision.
• 3:45
  And 52% of people wanted something that's just super
• 3:48
  decisive and that tells us a lot about society generally.
• 3:51
  And they've clearly gone for that latter group.
• 3:54
  Now you can go to settings and actually set personalized
• 3:56
  instructions for your GPT and it will follow them better.
• 4:00
  But still they've gone for that 52% of people that love decisive
• 4:05
  language models. I've one cancelled my license as
• 4:07
  a result of it constantly being direct and using constant short
• 4:11
  sentences and just being infuriating.
• 4:14
  And I actually switched to Claude as a result #3 is
• 4:17
  planning. So it's now going to be more
• 4:20
  verbose in its answers, which says a lot about how frustrating
• 4:24
  it was with its incredibly short responses.
• 4:27
  It's also going to be more explicit in how it got to a
• 4:30
  result as well. So again, not much of A
• 4:32
  technical improvement, but it's helping increase the feeling of
• 4:36
  vibes. And I think really that is the
• 4:38
  biggest pattern throughout this major releases.
• 4:40
  Their GPT 5 just didn't have the vibes to attract users.
• 4:44
  I for one, like I, as I said, cancelled my membership and I
• 4:48
  don't do things like that very lightly.
• 4:50
  I found it infuriating to work with #4 is the try to improve
• 4:54
  the writing. Now the big question here is
• 4:56
  improve for whom? And I'll get on to what I mean
• 4:59
  here. There's been a lot of
• 5:00
  independent analysis in this as always.
• 5:02
  And on the EQ benchmarks, Creative writing leaderboard,
• 5:06
  which is basically a benchmark that evaluates a language models
• 5:09
  ability to generate characters and character development,
• 5:13
  generate emotional engagement with readers or create plot
• 5:16
  structures. And lots of reviewers have said
• 5:18
  it no longer feels synthetic. And many people in the
• 5:20
  communities have been saying that it's the first model
• 5:23
  they've been excited to use for writing in a very long time.
• 5:28
  Now, Claude, especially Claude Sonnet completely beats every of
• 5:31
  the model on creative tasks, for example, in poetry or plot
• 5:34
  development fiction. But GPT 5.1 has obviously tried
• 5:39
  to gear towards the writers market.
• 5:42
  Again, it's been noted that it's especially good with technical
• 5:44
  writing because it loves having short sentences.
• 5:47
  Now, a little bit of caveat to this.
• 5:50
  There's a study that was released by Cornell that
• 5:53
  basically revealed something quite interesting about what
• 5:56
  happens with language models and the way it shapes our thought
• 5:59
  process and our interactions as a society.
• 6:03
  Researchers gave 118 people, so 60 from I58 from the US at
• 6:08
  writing tasks designed to understand and surface cultural
• 6:13
  expression. So things like describe your
• 6:16
  favorite festival or write about your favorite food.
• 6:20
  And now half wrote with AI assistance and the other half
• 6:22
  didn't. And what they found was quite
• 6:25
  interesting. When Indian participants used AI
• 6:28
  writing assistance, their writing suddenly shifted towards
• 6:32
  a more American cultural norms in the way they explained their
• 6:36
  favorite food or festival. So a good example here was an
• 6:39
  Indian participant wrote about Diwali and in it they wrote we
• 6:44
  worship Goddess Laxmi pot crackers and eat sweets.
• 6:48
  Yet with the AI assistance that was turned into each additional
• 6:52
  breakfast items and have a day filled with happiness and
• 6:56
  warmth. So those specific cultural
• 6:58
  references, the goddess Laxmi pot crackers, sweets got
• 7:02
  replaced for generic western descriptions.
• 7:05
  So traditional Indian breakfast items and happiness and warmth
• 7:09
  could be describing anything. And obviously AI has been
• 7:11
  trained on a lot of Western content, and as a result, it's
• 7:15
  been subtly pushing writing words with cultural norms in it.
• 7:21
  So I'm always a bit hesitant when they try to force changes
• 7:24
  to language models in the way that it writes and expresses
• 7:27
  itself because often it shapes the way that people express
• 7:31
  themselves online. So yeah, when we say GPT 5.1 is
• 7:34
  better than writing, I always wonder better for who, because
• 7:39
  obviously it's always optimized towards a certain audience.
• 7:42
  But anyway, that's just a small gripe with language models
• 7:45
  generally. This is exactly the whole point
• 7:46
  with vibes over benchmarks now, which is you can't really
• 7:49
  measure writing quality on just technical dimensions like
• 7:53
  clarity or structure. So yeah, vibes is going to
• 7:56
  become increasingly important with all language models moving
• 7:59
  forward. And finally, they had a warmer
• 8:01
  personality added to GPT. Open AI published something
• 8:05
  called a systems card. So essentially this is a safety
• 8:08
  assessment that they publish and it's a document for all major
• 8:11
  models and buried in that system card is key information about
• 8:15
  the model and it's safety. So apparently GPT 5.1 showed
• 8:20
  safety regression so got worse at safety benchmarks across 9 of
• 8:24
  13 categories compared to GPT 5. Specifically, anything to do
• 8:28
  with mental health metrics got worse and emotional resilience
• 8:32
  got worse. And obviously they knew this
• 8:35
  before they shipped it. You know, they documented it and
• 8:37
  they shipped it anyway because people wanted a model that felt
• 8:40
  more human and warm. Now, what does emotional
• 8:44
  resilience metrics got worse even mean?
• 8:46
  Well, open eyes data shows that nought point nought 7% of weekly
• 8:50
  active users, which is about 560,000 people per week show
• 8:55
  signals of psychosis or mania related to their chachi BT use.
• 8:59
  And another 0.15% which is about 1.2 million users shows
• 9:04
  heightened emotional attachment. So yeah, all those people are
• 9:08
  those that their safety benchmarks have regressed in the
• 9:12
  people that are particularly vulnerable.
• 9:14
  So there are pros and cons survives over benchmarks.
• 9:16
  I personally prefer a model that doesn't just use non-stop short
• 9:19
  sentences like it's a robot. And the reason they've done this
• 9:21
  is probably quite clear because over this last period, their
• 9:24
  enterprise market share fell from 50% to 25%, whilst Clawed
• 9:29
  captured 32% of enterprise market share.
• 9:33
  Claude code holds 42% versus open AI is 21%.
• 9:37
  And most people are choosing Claude over open AI.
• 9:40
  But ChatGPT has over 800 million users who aren't doing big
• 9:44
  evaluations into which package to choose.
• 9:47
  They're doing it based on feel and vibes.
• 9:50
  And so you could really say that we've reached a level of
• 9:52
  intelligence where things like personality matter more than
• 9:57
  just raw intelligence. So yeah, I think Open Air made a
• 10:00
  very deliberate choice to optimize for what those 800
• 10:03
  million people wanted, which is more warmth, personality, the
• 10:08
  ability to follow instructions better, for it to be a bit more
• 10:10
  verbose and explain its ideas better.
• 10:24
  So yeah, to conclude those those are the five things that GPT 5.1
• 10:29
  does better. And really it's quite obvious
• 10:31
  what they're doing. It's selling vibes over
• 10:33
  benchmarks. And I think we're going to start
• 10:35
  to see this as a pattern more and more and more.
• 10:38
  We've definitely hit an inflection point where technical
• 10:41
  capability has basically equalised across all the major
• 10:45
  providers and the competition has moved to something that's
• 10:47
  much harder to measure. How does this thing feel when I
• 10:52
  use it? Anyway, that's it for this week.
• 10:56
  I hope you enjoyed today's episode and I'll see you next
• 10:59
  week.