Latest / In the Loop AI / Why Claude Opus 5 is Our Favorite AI Model (For Now)

Transcript

Why Claude Opus 5 is Our Favorite AI Model (For Now) · In the Loop AI

0:00
Josh: So Anthropic just released a brand new model. It's not the best model.
0:03
Josh: It's not the smartest model.
0:04
Josh: It's not the cheapest or the fastest model. And yet this is possibly our most
0:09
Josh: favorite model release this entire year.
0:11
Josh: So what gives here? What's actually happening with Claude Opus 5?
0:15
Josh: This is the new iteration after Opus 4.8 and it sits just under Claude Fable 5.
0:19
Josh: Now a lot of people see this, they're like, hmm, I'm not that excited about
0:22
Josh: this model. It's just an iteration on another model that we've already had.
0:26
Josh: I think you're wrong. And I have some demos here. Ejaz, I know you've been playing
0:30
Josh: with this since the weekend, basically, talking to Opus 5, figuring out where
0:35
Josh: the edge cases are, figuring out how all of this works.
0:38
Josh: And the review so far is pretty great, right? Like Opus is a really fun model.
0:42
Josh: And I think that's what we're going to spend this episode unpacking,
0:44
Josh: is just how fun it is and how to get the best out of this model if you are someone
0:48
Josh: who's new coming to Opus 5.0.
0:50
Ejaaz: First and foremost, this model, as you mentioned, is not the cheapest,
0:54
Ejaaz: but it's as cheap as Opus 4.8.
0:57
Ejaaz: But it's a massively different type of model. And the first characteristic that
1:00
Ejaaz: it's very good at is visual intelligence.
1:03
Ejaaz: So what I mean by this is no clawed models can actually kind of see things.
1:07
Ejaaz: It kind of like when you send it an image, it kind of breaks down the pixels
1:11
Ejaaz: and reinvents what it's looking at from a code perspective.
1:14
Ejaaz: Now, what I'm about to show you on my screen, Josh, was one-shotted over one and a half hours, okay?
1:20
Ejaaz: It guzzled around 150,000 tokens.
1:24
Ejaaz: I didn't click anything after I sent the prompt and all of this is one shot. Are you ready?
1:30
Ejaaz: Let me tell you if you, let me ask you if you recognize what you're looking at right now.
1:34
Josh: Oh my goodness, this looks like home. I'm seeing the Williamsburg Bridge.
1:37
Josh: Oh my God, this is the East River.
1:40
Josh: Do you see this, Pat Boy? Oh, wait, dude, this looks awesome.
1:44
Ejaaz: This is a structurally accurate, of course, graphically rendered version of
1:50
Ejaaz: the Brooklyn Bridge live with trucks at sunset.
1:54
Ejaaz: All I said to Claude was, I want you to recreate the Brooklyn Bridge,
1:59
Ejaaz: dimensionally accurate, pull the architectural renderings, everything on your own.
2:05
Ejaaz: And then I want you to span out a bunch of sub agents.
2:09
Ejaaz: And I want you to get these agents to build different components of the Brooklyn
2:13
Ejaaz: Bridge and then review it.
2:15
Ejaaz: So take a screenshot, review it, and only feed me back the end result once you
2:20
Ejaaz: are satisfied with it. But it needs to look like something out of a really cool computer game.
2:26
Ejaaz: And this is what we ended up with. Here, let me show you the sunset that wide
2:29
Ejaaz: shot, if you want to have a look.
2:30
Ejaaz: Let me show you a tower over here.
2:32
Josh: Oh, that's fun. Okay, so it is pre-made shots for it as well.
2:35
Ejaaz: Yeah, maybe I'll show you the promenade if you notice the bridge is swaying,
2:38
Ejaaz: but maybe you want to look underneath the deck, make sure it's all structurally sound.
2:42
Ejaaz: Now, imagine the different use cases for this. Obviously, there's simulations,
2:45
Ejaaz: there's games, but if you're an architect that is coming up with a brand new
2:49
Ejaaz: idea and you want to kind of put things out in the wild, you can now have a
2:53
Ejaaz: physically and structurally accurate rendering of this model.
2:56
Ejaaz: Now, I know a lot of folks listening to this might be thinking,
2:58
Ejaaz: well, okay, maybe I don't want to be creating games and stuff like that.
3:01
Ejaaz: We'll get to the other good stuff. But what I want to point out is cold Opus
3:05
Ejaaz: can't see anything. I just want to reiterate that.
3:07
Ejaaz: It generated every single asset, every single visual that you're seeing on screen
3:11
Ejaaz: right now from scratch. It created a file on my computer.
3:15
Ejaaz: It did so with one single prompt in one and a half hours, and it only cost 100,000
3:19
Ejaaz: tokens. It's so, so impressive.
3:21
Josh: I think this is the thing that Opus 5 is really exceptional at that a lot of
3:25
Josh: people are figuring out is when we went to Fable we always loved the visual demos.
3:28
Josh: Fable was excellent but it's incredibly expensive. It takes a long time.
3:32
Josh: It is that leading flagship model if you want the best. Opus 5 gives you
3:36
Josh: Almost all of that, but with a fraction of the cost, a fraction of the time,
3:40
Josh: and other really unique capabilities. So we have a few other demos,
3:43
Josh: some that we found online that we didn't make ourselves.
3:45
Josh: And they blew my mind with the first one being this video that we're seeing
3:49
Josh: now. So this guy on X, he said he one-shotted this game.
3:54
Josh: And this game is a demo made of custom code without a single external asset.
3:59
Josh: And when you look at this game, it looks like you're watching a game of Call of Duty.
4:02
Josh: It's really impressive. This is Call of Duty. It's what it looks like,
4:05
Josh: right? I mean, the hands are a little funny and like there's definitely a lot of weird quirks here.
4:08
Josh: But for a one shot prompt on a model that is not a flagship model,
4:12
Josh: this is incredibly impressive.
4:14
Josh: And what I really appreciated about this model and what we're going to do right
4:17
Josh: now for the people who are watching is you could actually go and play this model
4:20
Josh: right now in the browser.
4:21
Josh: So please, if you would, I have the browser and let's see what your skills are
4:26
Josh: like. Now, for people who don't know, I've spent my entire life playing Call
4:29
Josh: of Duty, very familiar with the game.
4:31
Josh: And I'm here to criticize Ejaz on how well you are able to play this one-shot
4:36
Josh: demo from Opus 5. And we have the game loaded up here, Ejaz. Let's see what you got.
4:40
Josh: It's funny, this looks like a map from Call of Duty Modern Warfare.
4:43
Josh: I could see you're getting kills, you know, reasonably well.
4:45
Ejaaz: Oh my God, I'm getting cooked.
4:47
Josh: Honestly, of all the players, if I had to team up with, I'm not sure you're
4:51
Josh: going to be the one for Call of Duty.
4:53
Josh: I got to be honest with you, I'm not super impressed with the movement or the shots.
4:56
Josh: Fair enough. But this is like a really impressive demo. I mean,
4:58
Josh: the fact that this was created with one shot on opus 5 this is a game that costs
5:03
Josh: studios billions of dollars to develop and you have a lot of the textures like
5:06
Josh: the textures of the map look good
5:08
Josh: the design of the weapon looks good the hands are a little funky but it has
5:12
Josh: the skybox it has all these things
5:13
Josh: that you would look for if we had audio turned on you can actually hear
5:16
Josh: the sound of footsteps you can hear the hit boxes when someone gets hit by a
5:19
Josh: bullet you can hear the weapon firing it's really robust and it clearly has
5:23
Josh: this very deep understanding of what these games look like and that's what i
5:26
Josh: found and super interesting with this particular demo.
5:28
Josh: We have a couple other demos that I wanted to show before we got too deep into this.
5:32
Josh: One of which being a snowboarder game. So one of my favorite video games when
5:36
Josh: I was younger, SSX Tricky.
5:37
Josh: I'm not sure you've ever played that, Ejaz. It is a snowboarding game where
5:41
Josh: you go down a slope and you do tricks and you get points for doing these crazy tricks.
5:46
Josh: There is another one just like that. It doesn't have the tricks per se,
5:49
Josh: but it is a snowboard game indistinguishable from a game like SSX Tricky.
5:53
Josh: In fact, far more impressive in terms of visual integrity than a game like SSX
5:59
Josh: Tricky would be. And there's a quote post that Signal wrote.
6:02
Josh: I always love reading Signal's post on X.
6:04
Josh: And he had some commentary about this specifically. And he says,
6:07
Josh: if you put this game in the App Store in 2012, it would have potentially grossed
6:11
Josh: $5 to $10 million, at least over its lifespan, maybe even more.
6:15
Josh: Today, it can be one-shotted by roughly anyone with the correct physics,
6:18
Josh: graphics, gameplay, etc.
6:20
Josh: It's like $10 million of economic value wiped out in 15 years on only one game.
6:25
Josh: And I think that's kind of true. It's very impressive that we're able to get
6:28
Josh: things indistinguishable from leading AAA games from a single prompt on a model
6:33
Josh: that isn't even Frontier. And that's kind of the essence of Opus 5 as we've
6:38
Josh: been playing it. It's like the best
6:40
Josh: bang for your buck as it relates to getting high quality tokens for what you're able to pay for them.
6:45
Ejaaz: Before we go into what else Opus 5 is good at, and trust me,
6:49
Ejaaz: there are many things, I just want to talk about what it takes to create something
6:53
Ejaaz: that is as compelling visually as what you're seeing on screen as the games
6:57
Ejaaz: that we've showed you in one shot.
6:58
Ejaaz: Like, I am not exaggerating. I will show you the prompt that Matt Schumer used
7:02
Ejaaz: to create that FBS game that we just played badly.
7:06
Ejaaz: He goes, I want you to build a first person shooter at the level of the most
7:09
Ejaaz: recent Call of Duty games, it should be utterly perfect, visually beautiful,
7:13
Ejaaz: with every single thing done at AAA quality, from textures to physics to anything you can think of.
7:19
Ejaaz: Then he goes on to say, and this is the key part, so if you're listening to
7:22
Ejaaz: this and you're wondering, how can I do this for a game or for any other task
7:25
Ejaaz: that you're trying to do using your AI model, he says the following,
7:29
Ejaaz: fan out sub-agents and have sub-agents tackle each one individually so that
7:34
Ejaaz: the game is utterly perfect.
7:35
Ejaaz: You should slash loop on each item and have a separate sub-agent check it visually
7:41
Ejaaz: to ensure it looks AAA. That separate sub-agent should be a really harsh critic.
7:45
Ejaaz: And if it doesn't look AAA, it should keep going.
7:48
Ejaaz: Now, let me unpack why that's so important for not just this gaming prompt,
7:51
Ejaaz: but any prompt that you do for whatever you're trying to build using AI.
7:55
Ejaaz: He, number one, said, spin up multiple versions of yourself,
7:59
Ejaaz: Claude, so that you can work separately at the same time on different components of product.
8:05
Ejaaz: Now, if you think about a traditional company, they hire a ton of employees
8:08
Ejaaz: and they have different teams that all work together to build the same product.
8:11
Ejaaz: In this instance, it's the same thing, except there are various versions of
8:15
Ejaaz: Claude that tackle different tasks.
8:17
Ejaaz: Number two, he asked it specifically to spin up a manager role,
8:21
Ejaaz: which basically monitors these agents that are building these different things
8:24
Ejaaz: to make sure that they do a good job.
8:25
Ejaaz: And then number three, he said, use this loop function. The loop function is
8:29
Ejaaz: basically, hey, if you're not happy with the work that you just created,
8:33
Ejaaz: don't give it to me to give you feedback and say like, hey, improve this.
8:36
Ejaaz: Just use another agent to do this. And this beautiful.
8:40
Ejaaz: Kind of synonymous loop can be applied to anything, not just a game.
8:43
Ejaaz: It could be applied to your Excel tasks that you're trying to do at work.
8:46
Ejaaz: It could be applied to a blueprint that you're trying to create your architecture
8:50
Ejaaz: from. It could be applied to anything.
8:51
Ejaaz: And I think this is a really valuable lesson to figure out with Opus 5 specifically,
8:56
Ejaaz: is that you can spin up multiple agents and create something really, really awesome.
9:00
Ejaaz: But Josh, I think kind of moving on from the visual aesthetics of what Claude
9:04
Ejaaz: Opus 5 can do, it made some pretty major breakthroughs in some other things,
9:09
Ejaaz: including something called Arc AGI 3, which is basically the precipice of any AI model test.
9:15
Josh: Yes, Arc AGI 3 is like the flagship test. And when it comes to this flagship
9:20
Josh: test, Opus has done exceptionally well.
9:23
Josh: The previous high score was 7.8, scored by GPT 5.6 Sol Max, the highest end
9:30
Josh: GPT model that's currently available. Opus 5 got a 30%.
9:33
Josh: I mean, that's like an unbelievably high amount of rating on this benchmark
9:38
Josh: and we have to ask the question why what did it do and
9:42
Josh: arc actually came out and explained themselves and they said during our analysis
9:45
Josh: of opus 5 we observed a new capability previously unseen from frontier models
9:50
Josh: opus 5 used advanced logical reasoning to turn arc agi 3 layouts into algebraic notation
9:56
Josh: on action 23 it describes the seen as four center equal two axis five center
10:01
Josh: basically what it's doing here is instead of taking the visuals you just like
10:05
Josh: you mentioned opus can't see
10:07
Josh: it takes the visuals converts them into math and then thinks on that math that it's
10:12
Josh: visualizing and as a result it's able to achieve significantly better scores
10:16
Josh: on this visual reasoning
10:17
Josh: logical reasoning i guess is how they describe it versus other models so i find
10:21
Josh: this really interesting if you look at the example on screen that they were
10:24
Josh: showing if we could just go back to just the general demo it shows these shapes that are shifting
10:29
Josh: and a lot of times models have difficulty
10:32
Josh: Applying logic to make these work. Opus, because they converted it all to math,
10:36
Josh: it was able to achieve such a high score. It was really impressive.
10:38
Josh: This is what, four or five times that of the next leading model?
10:42
Josh: And cost, I know for a fact, this cost a fraction of what it costs for 5.6 Sol max.
10:47
Josh: So really impressive outputs from this thing.
10:49
Ejaaz: If we compare this directly with Fable, it can do most of the things that Fable
10:54
Ejaaz: is capable of, but at a fraction of the cost, specifically 50% less cost.
10:59
Ejaaz: Now, it can't do some things that Fable is really good at, such as extreme reasoning
11:05
Ejaaz: or coding at the highest level.
11:07
Ejaaz: But as a workhorse model, it is exceptionally intelligent.
11:11
Ejaaz: And it's now my default versus Opus 4.8. But let's look at some of the examples
11:16
Ejaaz: as to like cost wise and what this shapes up in other kinds of benchmarks.
11:20
Ejaaz: So with Fable 5, it's kind of on average better than Fable 5 when it comes to computer use.
11:26
Ejaaz: The price ends. So you're paying around, I think it's $5 per million tokens
11:30
Ejaaz: input and $25 per million tokens output. So $30 overall, if you want to kind
11:35
Ejaaz: of use this for usage credits, that is exactly half the cost of Fable,
11:38
Ejaaz: but you're getting so much more.
11:40
Ejaaz: And then the other thing I'll say is my biggest critique for Fable,
11:44
Ejaaz: Josh, is that whenever I'm trying to use it for pretty much most things,
11:49
Ejaaz: it sometimes gets triggered or rather frequently gets triggered through its safety flag.
11:53
Ejaaz: So it gets relegated to Opus 4.8.
11:56
Ejaaz: With Opus 5, I haven't come across that at all.
12:00
Ejaaz: When I ask it to do something, even if it's bio-related or cybersecurity-related,
12:03
Ejaaz: like if I'm saying, hey, can you check my computer for bugs and make sure that it's all good?
12:07
Ejaaz: It just does the thing. And that's the other thing about Opus 5.
12:11
Ejaaz: It is almost as intelligent as Fable 5 when it comes to biology and cybersecurity-type
12:17
Ejaaz: things, but it's really hard to exploit.
12:21
Ejaaz: If you try to prompt inject it, it is an extremely aligned model,
12:25
Ejaaz: which just won't do the thing. It's smart enough to understand when you're asking
12:29
Ejaaz: an innocent question about research
12:30
Ejaaz: or whatever that might be without triggering any safeguards. It is,
12:34
Ejaaz: arguably the perfect version of fable five and it's going to be my go-to model going forwards.
12:39
Josh: Yeah i love this model and there's interesting ways in which you can use it
12:43
Josh: and it feels like it's the perfect
12:45
Josh: completion to this family of five class models from anthropic i mean when you think about it
12:50
Josh: anthropic released four frontier models in eight weeks we're getting now one
12:53
Josh: every other week we have mythos five fable five sonnet five now so we have the
12:58
Josh: this top three that's kind of sitting here this rounds out the set and this is the perfect fallback
13:02
Josh: from fable and it feels like that, where when,
13:05
Josh: as I've been testing it, as I've been using this, I found the most success with
13:09
Josh: really hard problems using Fable.
13:11
Josh: Fable will think for a very long time. And the longer I think,
13:13
Josh: the better the outputs generally are.
13:16
Josh: I find that Fable is very ambitious. It will go off. It will spin out its own subagents on its own.
13:21
Josh: It will think very critically. It will develop plans and then check against
13:24
Josh: those plans and do so in a way that I can trust.
13:27
Josh: And I know that the longer it's running, the better my result will be.
13:30
Josh: Opus, I will say, is a little bit different. It's a great fallback in the sense
13:34
Josh: that if Fable gets tripped up with security or bioweapon, I'm so glad that it
13:38
Josh: now lands on Opus 5 instead of 4.8. It's much more capable.
13:41
Josh: But I think it's different. As you mentioned that it's a workhorse model,
13:44
Josh: it feels like a workhorse model. It's great for shorter duration tasks.
13:48
Josh: And I would, generally speaking, if you're using Fable, if you're using Opus,
13:52
Josh: I would delegate it to those smaller tasks. In fact, during testing,
13:56
Josh: one of my favorite modes to use it on is medium instead of high or extra high.
14:00
Josh: Because what I find is the longer Opus thinks the kind of more tripped up it
14:04
Josh: can get it feels like the little brother that was trained by fable so it knows
14:08
Josh: to think like fable but it doesn't quite understand the reasoning behind why fable does everything
14:14
Josh: and that unlock does make the outputs feel a little messy sometimes over really
14:18
Josh: long duration tasks so if you're someone who's using this
14:21
Josh: Try medium and see how medium effort performs. It's not necessarily a worse
14:25
Josh: output. It just thinks for a little bit less longer.
14:27
Josh: And if it has less time to kind of think and turn into these thought spirals,
14:31
Josh: you might wind up getting slightly better outputs. That's how I've been using it.
14:34
Josh: For the people who are wondering how to use this, I believe the free plan has
14:37
Josh: access to it, but also the $20 a month plan gives you quite a bit of access
14:41
Josh: to Opus 5, which seems very worthwhile for the amount of output tokens that
14:45
Josh: you can get from this thing.
14:46
Josh: And if you did want fable that does require i think the hundred dollar max plan
14:49
Josh: but also i know they're giving out a hundred dollars in usage tokens to test
14:53
Josh: this out to a lot of accounts so there is the option for everyone to kind of get involved
14:56
Josh: and try this out but for twenty dollars a month man this is a really compelling
15:00
Josh: model to do some like serious damage on any workloads that you might have
15:04
Ejaaz: It's my new go-to model and if if you're wondering hmm should i switch to gpt 5.6 sol
15:10
Ejaaz: which model should be my everyday model you should basically use this model
15:14
Ejaaz: that that would be my kind of like bit of advice here because it's really good
15:18
Ejaaz: at just general reasoning and answering most general questions that people have.
15:21
Ejaaz: Now, what it's not exceptional as, I think it's worth pointing out,
15:25
Ejaaz: is if you want the best-in-class model for coding or to reference different
15:30
Ejaaz: types of health or, in some cases, some kind of cybersecurity e-bals,
15:35
Ejaaz: you are still better off using.
15:37
Ejaaz: Fable or if you are one of the classified partners from Anthropic, Mythos 5.
15:41
Ejaaz: But as a general model, it is a fantastic thing to get access to.
15:45
Ejaaz: The other thing I'll say is, I think this is the first ever model launch from
15:50
Ejaaz: Anthropoc or any Frontier Lab, where they didn't advertise.
15:54
Ejaaz: How intelligent it is. You mentioned ARC AGI 3 earlier, Josh.
15:58
Ejaaz: Now, if you've heard of the AGI index or the test, there was AGI 1 and AGI 2.
16:03
Ejaaz: But the issue was all these models kept getting smarter and just beating these tests to 100%.
16:08
Ejaaz: They had to start creating more difficult, hard-to-solve benchmarks.
16:12
Ejaaz: And ARC AGI 3 was the most recent one. And as you mentioned,
16:15
Ejaaz: the most recent model, GPT 5.6 SOL, scored 7%. This one scored 30%,
16:19
Ejaaz: the highest of any model higher than Fable, but this wasn't the headline advertisement from Anthropic.
16:26
Ejaaz: And I think a reason behind that might potentially be, you know,
16:29
Ejaaz: they don't really want to advertise how intelligent these models are because
16:32
Ejaaz: of fear of regulation or whatever that might be.
16:34
Ejaaz: But the point is, if you don't have access, if you don't have one of those memberships
16:37
Ejaaz: that Josh just mentioned to get access to Fable 5, fear not,
16:41
Ejaaz: because Opus 5 itself is actually a fantastic model in itself.
16:44
Ejaaz: And I encourage you to go out and try this. Now, the other part that I will
16:48
Ejaaz: say is really powerful, what makes this model like really valuable to use is
16:52
Ejaaz: the fact that it can natively spin out multiple AI agents to go and do your task.
16:58
Ejaaz: And this cannot be understated because usually when people interact with an
17:02
Ejaaz: AI model, they just have a one-on-one conversation with one instance of Claude.
17:05
Ejaaz: But imagine if you could have a one-on-one conversation with Claude and it spins
17:10
Ejaaz: out a hundred different employee workers to go and do your job and it works
17:13
Ejaaz: in parallel simultaneously. AI agents is without doubt the future of how you execute work.
17:19
Ejaaz: And it's incredibly important to focus on their security.
17:21
Josh: And speaking of agents and just all the sub-agents that Opus can spin up,
17:25
Josh: we have a sponsor that is here to talk about agents and they are Ledger.
17:29
Josh: If you are building with AI agents, you're worried about security.
17:31
Josh: An agent with unchecked access is a brilliant but probabilistic teammate with
17:36
Josh: access to important secrets.
17:37
Josh: Ledger Agent Stack fixes that with four open source tools, DMK Skills and CLIs
17:41
Josh: for Ledger Wallet, Ledger Enterprise and Ledger Enterprise Multisig that lets
17:45
Josh: your agents work with crypto while you stay in control. You can tell your agent, rebalance my wallet.
17:50
Josh: ETH to 40%, fastest swaps. It does the math, drafts the swap, and stops.
17:55
Josh: The transaction lands on your ledger signer and nothing moves until you approve it.
17:58
Josh: Agents Propose, Humans Approve, Ledger Signers Enforce. And this works with
18:03
Josh: Cloud Code, Codex, Cursor, it's open source, ready today on developers.ledger.com,
18:08
Josh: which you can find in the link in our description below. Thank you so much to
18:11
Josh: Ledger for sponsoring this episode.
18:13
Josh: And now, speaking of all these safeguards, these approvals, we have to talk
18:16
Josh: about where Opus 5 sits on the security spectrum. EJS, I know for a fact that
18:21
Josh: if I ask it, how to build me a nuclear warhead, it's not going to answer.
18:24
Josh: But there's a line that is closer to there than Fable 5 is. So where is that
18:29
Josh: line in terms of what you can use this model for?
18:31
Ejaaz: Well, rather than give you the line first, I want to give you a stat,
18:36
Ejaaz: which should relieve a lot of people listening to this that have been using Fable.
18:40
Ejaaz: 85% of the time, you have fewer stops, so fewer relegations to older models.
18:47
Ejaaz: And what that means is, if you ask it a question, let's say it's like,
18:50
Ejaaz: you know, can you tell me that the mitochondria is the powerhouse of the cell?
18:53
Ejaaz: It won't flag you for trying to be an aggressor.
18:56
Ejaaz: It instead just answers your question 85% of the time.
19:01
Ejaaz: And this is massive because it's almost, from my experience personally,
19:04
Ejaaz: every other attempt that I use Fable 5 for, it gets triggered unless I'm explicitly
19:09
Ejaaz: not talking about any kind of AI models. And when you're doing a show where
19:12
Ejaaz: you talk about AI models, it's kind of a hard thing to avoid.
19:15
Ejaaz: So the level up that's happened with Opus 5 here is they saw the feedback with Fable 5.
19:20
Ejaaz: They saw that the safety guards were getting triggered. And for context,
19:23
Ejaaz: for anyone that's listening, if you're wondering why these safeguards are in
19:25
Ejaaz: place, it's because the US government got involved with Anthropic's release
19:29
Ejaaz: of Fable 5 and said, hey, like this thing is too powerful. You need to add more
19:32
Ejaaz: safeguards to it. You can't release your Mythos model.
19:35
Ejaaz: We need to have a safe version so that people don't exploit it and use it for malicious purposes.
19:39
Ejaaz: So Anthropic, pause, put out this Fable 5 model with a lot of safety guards,
19:43
Ejaaz: and they've been iterating ever since. Opus 5 is the embellishment of what a
19:47
Ejaaz: good version of Mythos 5 looks like to me personally.
19:51
Ejaaz: And it's because when it comes to the biggest risk indicators,
19:55
Ejaaz: which is when anyone's asking about cybersecurity, they could potentially want
19:59
Ejaaz: to exploit another app or platform using that.
20:02
Ejaaz: Or if they're evaluating biological types of things and they don't want to make
20:06
Ejaaz: sure they create a bioweapon, they can succinctly have a conversation with Opus
20:11
Ejaaz: 5 and you can't prompt inject Opus 5 to give you any kind of a malicious answer.
20:16
Ejaaz: And this is the main thing. when I was looking at the system card,
20:18
Ejaaz: the way, the blueprint of how they created this model, there is a heavy index,
20:23
Ejaaz: a heavy framework around Opus 5, which prevents you from trying to get a malicious answer from it.
20:29
Ejaaz: Now, can you use it to look at your computing operating system and figure out
20:33
Ejaaz: if there are any bugs to fix,
20:35
Ejaaz: Yes. But can you ask the Opus model to then create a potential defense to that exploit?
20:42
Ejaaz: The answer is no. You need to kind of like go through your own way to build
20:45
Ejaaz: it yourself, either by hand or to use it as an assistant.
20:47
Ejaaz: But you can't directly ask it to create a fix for it. And that,
20:50
Ejaaz: I think, is the most important nuance between the cybersecurity evals,
20:54
Ejaaz: the biosecurity evals between Opus 5 and Fable 5.
20:57
Josh: I mean, to frame this pretty practically, I've been using this for a while and
21:00
Josh: I've been testing all the edge cases and you basically don't notice it.
21:03
Josh: I'd say this is about the same safeguard level as standard models in the sense
21:06
Josh: that you're not going to run into it unless you are intentionally trying to run into it
21:10
Josh: you really got to ask some kind of edgy fringe questions in order to hit those
21:13
Josh: safeguards so i would say in terms of what you're actually going to realize
21:18
Josh: you are going to get blocked significantly less than you would with fable to
21:20
Josh: the point where you don't really notice it
21:23
Josh: and i think this is kind of where i've landed as a more power user who is
21:27
Josh: i just kind of token hungry just for the fun of it and i love using fable as
21:30
Josh: the default but i really appreciate the fact that I can have Opus 5 as that
21:34
Josh: fallback that can confidently answer questions that Fable cannot.
21:38
Josh: For example, I was trying to figure out stuff about there's this like new breakthrough
21:42
Josh: with like genomic sequencing and it'll like diverse the aging of your skin.
21:45
Josh: I was like, oh, this is sick. Let me read about it.
21:47
Josh: Fable was on the case. It couldn't figure it out. Opus took over.
21:50
Josh: It can handle that stuff fine. So in terms of safeguards, this will feel just
21:53
Josh: like an unfiltered model. This will give you the outputs that you want.
21:56
Josh: It's a lot less kind of on a hair trigger when it comes to these safeguards.
22:01
Josh: And that makes it a really fun model to use. So it's amazing.
22:04
Josh: It is accessible to everyone.
22:05
Josh: I will say that the one thing that you could caveat it with is the long duration thinking.
22:10
Josh: Sometimes it gets a little bit confused. So you may want to keep it on a slightly
22:13
Josh: lower setting for a lot of the work that you're doing.
22:15
Josh: But outside of that, it's freaking awesome. I mean, this is a great upgrade.
22:18
Josh: And now I feel good. Like when I opened up my Cloud app, I got Sonnet,
22:21
Josh: which I'm not sure I really use Sonnet because I'm not burning through like
22:25
Josh: a trillion tokens where I need it.
22:27
Josh: But opus 5 and fable 5 are this unbelievable tag team that just work together
22:31
Josh: so well there is also one one other nuance um fable speak which is like this
22:35
Josh: kind of funny thing where
22:36
Josh: it has a very unique way of talking it's very verbose that has been transferred
22:40
Josh: to opus 5 as well i've noticed so opus 5 likes to talk a lot it likes to speak
22:45
Josh: the way that fable does so that's an interesting little quirk um but outside
22:48
Josh: of that i've been using the model i love the model opus 5 is great
22:51
Josh: i think my ask to everyone who's listening is like hey are you using it what
22:54
Josh: are you using it for best use cases, anything like that. Like this has been cool.
22:58
Ejaaz: If you are on any kind of a membership on Claude or Anthropic specifically, this is the model to use.
23:04
Ejaaz: Now I will say that although it is 50% cheaper than Fable, this thing thinks
23:09
Ejaaz: quite a lot. It is a token guzzler. It uses a lot of tokens versus Fable 5.
23:15
Ejaaz: But you have a higher allowance when you're using this particular model.
23:18
Ejaaz: So if you're using Fable 5 and you don't want to pay access credits,
23:22
Ejaaz: you should definitely be using this model.
23:24
Ejaaz: The other thing I'll say is, I think this is very much Anthropik's response
23:28
Ejaaz: to some of the Chinese open source model debate that's been happening over the last week or so.
23:34
Ejaaz: Kimi K3 was launched by Moonshot Labs. It's a fantastic model,
23:37
Ejaaz: almost frontier capability, but at a fraction of the cost.
23:40
Ejaaz: And actually, as we're recording this episode, their model weight should become
23:44
Ejaaz: open to day which means that you can kind of run it on hardware at home right
23:48
Ejaaz: now they're good and bad things about this good is.
23:52
Ejaaz: You know you can run it privately on your data it's arguably cheaper the bad
23:55
Ejaaz: side of the thing is it has no safeguards around it which means that you could
23:59
Ejaaz: potentially use it for cyber security threats or bio threats
24:02
Ejaaz: moving forwards now the great thing about a model like opus 5 is it has those
24:07
Ejaaz: safeguards in place so you know that anyone that's using it can't really maliciously
24:10
Ejaaz: use it for any kind of mal intent and i think that there are pros and cons at both sides of things.
24:15
Ejaaz: I know that a large number of companies, prominent companies like NVIDIA,
24:18
Ejaaz: Microsoft signed an open source waiver this weekend. Anthropic was the one that
24:23
Ejaaz: hadn't signed it, but they have their reasons for this.
24:25
Ejaaz: And I'm excited for people to get their hands on Opus 5 and really tell us,
24:28
Ejaaz: you know, what's your experience about this? Are you enjoying it?
24:31
Ejaaz: Are you using it for any particular use cases that we haven't mentioned on this show?
24:35
Ejaaz: It's a fantastic model. So get your hands on it and let us know in the comments
24:38
Ejaaz: what your feedback is. But I think that is the end of the episode,
24:41
Ejaaz: the recap of Opus 5 today. I encourage everyone to use it. If you're listening
24:45
Ejaaz: to us or watching this on YouTube and you aren't subscribed to us, what are you doing?
24:48
Ejaaz: Leave us a comment, give us a thumbs up. It helps us out massively.
24:51
Ejaaz: If you're listening to this on Spotify or Apple Music, we are there.
24:54
Ejaaz: Leave us a comment. We read everything, by the way, and your feedback is wonderful.
24:57
Ejaaz: It influences what episodes we put out later on in the week and in the future.
25:01
Ejaaz: We are also looking for sponsors. Josh and I have spun Limitless out on our
25:04
Ejaaz: own and we are running this kind of like off our own budget. So if there's.
25:08
Josh: Anyone out there- Thank you to Ledger for stepping up.
25:10
Ejaaz: Yes, exactly. And thank you to Ledger for stepping up. and if there's anyone
25:14
Ejaaz: else that would like to sponsor and advertise your product or service,
25:17
Ejaaz: please let us know. We would love to hear from you. But I think aside from that,
25:20
Ejaaz: That is it.
25:21
Josh: That's everything. Thank you all so much for watching. We're coming at you live from...
25:25
Josh: Ejaz and I are bouncing all over the place this summer. So we're just like kind
25:28
Josh: of dropping in wherever we are in the world.
25:29
Josh: Currently, I'm like up north of San Francisco. Ejaz is going to the West Coast.
25:33
Josh: So we're doing this everywhere, but we're not missing the episode.
25:35
Josh: So if anything does, you know, if the video goes in and out a little bit,
25:38
Josh: we're doing this from remote, but we're getting the job done.
25:40
Josh: And we appreciate all of you listening and being here with us for the entire
25:44
Josh: journey. We will not let you down. And we appreciate all the support.
25:46
Josh: So thank you for watching as always. We have another...
25:49
Josh: We have a couple of really fun episodes coming up this week,
25:50
Josh: so stay tuned for that. Let us know in the comments section all the things that
25:53
Josh: we need to know in general, and we will see you guys in the next one. Thanks so much.
25:56
Ejaaz: See you guys.