Latest / In the Loop AI / THIS WEEK IN AI: Pope Approves Anthropic, Comparing AI Plans, Starcloud SpaceX Partnership

Transcript

THIS WEEK IN AI: Pope Approves Anthropic, Comparing AI Plans, Starcloud SpaceX Partnership · In the Loop AI

0:00
Josh: Everyone needs to stop forgetting about Meta, the social media company turned
0:04
Josh: AI company, because this week they have released quite a few new plans and options
0:08
Josh: that I think are going to be interesting to someone.
0:11
Josh: If you're watching this, chances are you will be affected by these new subscriptions.
0:14
Josh: What we're going to do is not only share the new meta subscriptions,
0:17
Josh: but also compare them to all the existing subscriptions that you can go and
0:21
Josh: pay for, whether it be for social media platforms like X or through AI like
0:25
Josh: Gemini, Claude, Anthropic.
0:27
Josh: There is now a full spread of all the different types of subscriptions you could
0:31
Josh: have to different models.
0:32
Josh: And we're going to walk through them systematically, starting with Meta,
0:35
Josh: who started on the social front this week, offering $4 plans for Instagram,
0:40
Josh: Facebook, and WhatsApp.
0:42
Ejaaz: Yeah, exactly. So Meta is getting into the subscription game,
0:45
Ejaaz: which they're probably following in the lead of like Anthropik and OpenAIR.
0:48
Ejaaz: They're seeing these annual recurring revenue numbers just kind of skyrocket.
0:51
Ejaaz: And they're like, I want some of that as well.
0:53
Ejaaz: Also, So like a bit of important context here, Meta's now spent around $30 billion
0:58
Ejaaz: between AI CapEx GPUs and like acquiring talent and trying to build their own
1:03
Ejaaz: AI products. But I mean, Josh, I think you share the same opinion.
1:06
Ejaaz: Which of the meta AI products have you actually used. This is their kind of
1:10
Ejaaz: like foray to try and get people to use it. So it comes in the form of a few
1:14
Ejaaz: different subscriptions.
1:15
Ejaaz: Now, the consumer ones look like Instagram Plus for not $4, but $3.99 a month.
1:21
Ejaaz: You've got Facebook Plus at $3.99 a month, and you've got WhatsApp Plus at $2.99 a month.
1:25
Ejaaz: Now, if you're wondering like why anyone would be using these kind of like individual
1:29
Ejaaz: apps and individual subscriptions, it's because Instagram kind of caters towards
1:33
Ejaaz: a content creator-focused type of platform.
1:36
Ejaaz: Facebook Plus, a lot of enterprises and business run their business on there via ads.
1:41
Ejaaz: The ad business for Meta alone is absolutely huge.
1:44
Ejaaz: And they're actually growing at a faster rate than Google did at the equivalent
1:47
Ejaaz: time that they existed in their lifecycle right now.
1:49
Ejaaz: And then you've got WhatsApp as well, which handles a lot of customer support
1:53
Ejaaz: for a lot of these enterprises. So you can think about one particular user,
1:56
Ejaaz: not just subscribing to one of these subscriptions, but many.
1:59
Ejaaz: And then collectively, there's this new enterprise plan. I think it's on the
2:03
Ejaaz: MetaOne AI subscription plan, which we'll get into in a second,
2:07
Ejaaz: which is kind of like this collective suite of access.
2:09
Ejaaz: Now, the thing that jumped out to me the most here is
2:12
Ejaaz: honestly, how cheap the accessibility is here.
2:15
Ejaaz: If you are a content creator, the chances are it's a very competitive game and
2:19
Ejaaz: you're not really earning as much money as like some of these like Fortune 500 companies.
2:23
Ejaaz: Now, if you want to get access to some, maybe not frontier intelligence,
2:27
Ejaaz: But a model that's good enough to do the thing that you're hyper-focused on,
2:31
Ejaaz: such as advertising or creating content or using the right SEO-timed words,
2:35
Ejaaz: meta is in the attention game.
2:37
Ejaaz: That is what all their models have been trained on, using their own data.
2:39
Ejaaz: They've been in this game for over a decade and they know how to capture people's attention.
2:43
Ejaaz: That's very useful for the particular type of users that they're going after.
2:47
Ejaaz: But I wonder how this compares directly to some of the other major frontier
2:50
Ejaaz: labs like Google or Anthropic.
2:52
Josh: Yeah, a helpful way of looking at Metis plans is there's three different types of them.
2:56
Josh: One is the one that we just mentioned for $4 a month, you get access to Facebook
3:00
Josh: and Instagram plus, which gives
3:02
Josh: access to extra features like profile customization, super reactions,
3:06
Josh: story insights, among other things this is still very new we're still figuring
3:09
Josh: out what exactly this includes second to this sitting on top of it are their
3:13
Josh: standard plus premium and max plans those are if you are an actual creator on
3:18
Josh: the platform that gives you the verified badge if you see people with blue check marks
3:22
Josh: not on a platform like X, they are paying for these plans.
3:25
Josh: That includes lots of things noteworthy. It doesn't include the skipping of
3:30
Josh: ads. No matter what, you cannot escape ads, it appears.
3:33
Josh: And then third to this, we have Meta's AI plan. And Meta's AI plan is the one
3:36
Josh: that we're going to sit on.
3:37
Josh: We're going to spend some time on and we're going to start comparing to the
3:39
Josh: other companies because it's starting pretty cheap.
3:42
Josh: Meta's AI subscription service starts at $8 a month.
3:46
Josh: What does that get you? We're not entirely sure just yet. It's
3:49
Josh: called meta one plus and it will give you access to presumably the latest
3:53
Josh: in meta's ai tools what those ai tools look like
3:56
Josh: we're not entirely sure this is all very new and novel but the second thing
4:00
Josh: is the 20 a month plan and that 20 a month plan is seeming to be the one that
4:04
Josh: wants to compete head to head with the other companies like claude and chat
4:09
Josh: gpt and gemini so how does it compare is what we're going to get into here i.
4:13
Ejaaz: Think a good place to start is uh looking at their creative plans and pricing
4:17
Ejaaz: that they have for their existing subscriptions.
4:20
Ejaaz: Now, this isn't the latest subscriptions that we just spoke about,
4:23
Ejaaz: but it kind of implies what the other kind of new subscription types will include.
4:28
Ejaaz: Now, I've noticed that across Plus, Premium, and Max, you have this thing being
4:32
Ejaaz: brandished right in front of you, which is protect your brand.
4:35
Ejaaz: I think a lot of Meta's users who are business users who care about the kind
4:38
Ejaaz: of social image and branding online across all their different platforms...
4:42
Ejaaz: Get a lot of impersonators so this is something that like i
4:45
Ejaaz: think ai is going to result in a proliferation of it's
4:48
Ejaaz: just kind of like fake content people that copy your content copyright issues etc
4:51
Ejaaz: this kind of like collectively protects you against
4:54
Ejaaz: that i'm wondering how that actually works but it seems like there's some
4:57
Ejaaz: form of like agent interaction uh whether they're human or digital and there's
5:01
Ejaaz: also maximized discovery i think this is really where meta kind of like shines
5:05
Ejaaz: compared to like the google ai models the anthropic ai models and open air themselves
5:10
Ejaaz: um Meta really has been focused on the search engine optimization role of things.
5:16
Ejaaz: Now, in a world of AI, you can think that the search engine kind of like evolves quite a bit.
5:21
Ejaaz: Google's been speaking about this quite a bit, which is the AI model itself
5:25
Ejaaz: or the AI agent ends up becoming the customer.
5:27
Ejaaz: And you need to cater to an AI agent versus a human.
5:29
Ejaaz: That SEO role looks very different. And Meta is trying to pioneer what that model looks like.
5:34
Ejaaz: The first model that they released a few months ago, I believe, is MewSpark.
5:38
Ejaaz: And it excels at this type of work. Whereas compared to like regular LLM work, it kind of sucks.
5:43
Ejaaz: Now, I want to look at like a direct comparison between Google,
5:47
Ejaaz: Meta and the other likes. And we have this really cool artifact here.
5:50
Josh: Yes. So thank you, Claude, for creating this artifact. We are clearly paying
5:53
Josh: for the Claude plan, or at least I am to say the least.
5:56
Josh: But this is a really helpful table that compares the different types of offerings
5:59
Josh: from all these companies. We have Meta, Google, Claude, ChatGPT and X slash Grok.
6:04
Josh: The entry level pricing is something that is used by almost all companies except
6:08
Josh: for Claude at $8 per month.
6:10
Josh: You can get access to meta one google's ai plan chat gpt
6:13
Josh: and x slash grok on x that gives you the blue check mark plus access to grok
6:17
Josh: chat gpt gives you the go plan so you get ads inside of your offerings that
6:22
Josh: when you type into your llm claude noteworthy doesn't do ads so they have no
6:26
Josh: plan at this level google plus is ai supported and then the same with meta one
6:31
Josh: now standard is the 20 a month.
6:33
Josh: X is noteworthy and the fact that they don't have one they
6:36
Josh: actually started $30 but among the others this is
6:39
Josh: the point where i think a lot of people will pay the most attention because people
6:42
Josh: are willing to pay $20 a month and in most cases it's very much worth it if
6:46
Josh: you are using a chat gpt plan if you are using a cloud plan that's free you're
6:50
Josh: currently on an old cheap model that doesn't really work nearly as well as the
6:55
Josh: frontier models are and if you want to judge the progress of ai models,
6:59
Josh: you really need to be on this $20 a month plan because it at least gives you
7:03
Josh: access to the Frontier models as they're being released.
7:06
Josh: To Opus 4.7, to ChatGPT 5.5, and Gemini and soon Meta's new offerings.
7:11
Josh: So if we're comparing these, the standard level, I'm not sure who would really
7:16
Josh: want to pay $20 for Meta's offering when you can go and get ChatGPT premium,
7:21
Josh: Claude premium, Google premium even at the same $20 price point. That's a bit difficult.
7:25
Josh: Then these go up and really just the the higher tiers
7:28
Josh: of this are more tokens so there's not a huge difference in
7:31
Josh: terms of feature set it's mostly just the ability to use it
7:34
Josh: more productively so as a starting point the 20 month plans are all very competitive
7:38
Josh: except for the fact that two of these are frontier models and the others are
7:43
Josh: not so if you're looking for frontier level intelligence top intelligence per
7:46
Josh: token it's really cloud and chat gpt google has a lot of fun tools if that's
7:51
Josh: where you like to spend your time.
7:52
Josh: But aside from that, Meta and Grok's offerings leave a lot to be desired.
7:56
Ejaaz: I'm kind of split on this, okay? So like on the one half, I agree with you.
8:00
Ejaaz: On the other half, I'm like, I do think Meta is going after a different type of market.
8:04
Ejaaz: Like when I look at like a $7.99 a month plan, I instantly think of OpenAI's ChatGPT Go plan.
8:10
Ejaaz: Do you remember they launched that specifically just in India?
8:12
Ejaaz: And I think it was like five bucks a month.
8:15
Ejaaz: That resulted in adding, I think it was like tens of millions of new users over
8:19
Ejaaz: the next couple of weeks since they launched it, right?
8:21
Ejaaz: And I think that Metz is trying to do a similar thing here. They're going for
8:23
Ejaaz: a very specific archetype of individual, which is like,
8:26
Ejaaz: You're out in the content game. They understand that, like, you may not have
8:29
Ejaaz: an abundance of money yet, but they want to kind of, like, lead you on that exponential.
8:32
Ejaaz: And they know that they have search and discovery kind of, like,
8:35
Ejaaz: nailed to the bone. And so they want to kind of, like, offer that or be the
8:38
Ejaaz: first major AI provider for that.
8:40
Ejaaz: Now, on the other hand of things, like, I can imagine as I see all these,
8:44
Ejaaz: like, new models released from Anthropic and OpenAI, the next model release
8:47
Ejaaz: means that the previous models released is, like, even cheaper.
8:50
Ejaaz: In fact, the newer model releases end up being cheaper than the old model releases,
8:53
Ejaaz: which is kind of antithetical because you like get a smarter model which
8:56
Ejaaz: is kind of cheaper than the previous model so it's like bang for your
8:59
Ejaaz: buck i also think that with more gpus and
9:02
Ejaaz: semiconductors as that all scales you end up just getting uh
9:05
Ejaaz: more tokens per watt i believe it's performance per watt it's just going to
9:09
Ejaaz: end up kind of driving lower so i don't know about the staying power of meta
9:14
Ejaaz: subscriptions here their models do need to improve pretty rapidly and this kind
9:17
Ejaaz: of leads me on to like an overarching theme that i have on meta which is this
9:21
Ejaaz: is actually like good news for Meta.
9:23
Ejaaz: And this is desperately needed by Meta because they've blown around $30 billion
9:27
Ejaaz: over the last 12 months, acquiring talent and investing in this AI thing with
9:32
Ejaaz: nothing to show for it. They released a MuseSpark model.
9:35
Ejaaz: It was kind of like whatever people have forgotten about it.
9:36
Ejaaz: People aren't really kind of using it aside from people who actually use Meta's different platforms.
9:40
Ejaaz: But this could technically or theoretically on paper result in kind of like a goldmine for them.
9:45
Ejaaz: If you imagine like they can convert like 100 million users for like their $20
9:49
Ejaaz: dollar a month plan, that is reasonable income coming in.
9:52
Ejaaz: But I've also seen comments from Zuck earlier this week, which talks about if
9:55
Ejaaz: this doesn't play out, this subscription thing,
9:58
Ejaaz: if their AI models aren't used by a lot of people, they're willing to go down
10:01
Ejaaz: the NeoCloud route, which is what Elon Musk has done recently,
10:04
Ejaaz: and partner with major AI labs to rent out their compute and data classes,
10:08
Ejaaz: which they've invested billions of dollars in.
10:09
Ejaaz: So I think Zuck is being pressured quite a lot by shareholders at this point.
10:14
Ejaaz: He's being forced to kind of show some kind of revenue, he fired or laid off
10:18
Ejaaz: 8,000 plus people over the last seven days.
10:20
Ejaaz: And that was like one of like, I think five cuts in the last couple of months.
10:24
Ejaaz: So Meta's like backed into a corner right now and they need to kind of like
10:28
Ejaaz: show the goods or they need to pivot pretty aggressively.
10:31
Josh: You mentioned the word neocloud, which is funny because I have some neocloud news.
10:34
Josh: This is, again, if you're listening to this, this is a Friday roundup.
10:36
Josh: We're covering all the news. So we're going to speed run this.
10:38
Josh: It's a very busy morning at Limitless HQ.
10:40
Josh: So we're going to try to get through these as quick as we can,
10:42
Josh: starting with the StarCloud news, which I found to be really interesting.
10:46
Josh: Elon and SpaceX have a new partner. They are working with, it's funny enough,
10:50
Josh: Gavin Baker here is in the photo. We just recorded an episode with him yesterday about his portfolio.
10:55
Josh: Clearly he is bullish on space. And the announcement is that StarCloud
10:58
Josh: is building star cloud three that is going to
11:00
Josh: be launching on the new spacex starship this is
11:03
Josh: starship version three this is the test flight that they just had last week that
11:06
Josh: we covered and the idea is that that three-ton
11:09
Josh: spacecraft is designed to fit inside of the pes dispenser inside of starship
11:14
Josh: the ceo philip johnson claims that star cloud three would be the first orbital
11:18
Josh: data center that's actually cost competitive with terrestrial at about five
11:22
Josh: cents per kilowatt if commercial costs land near $500 per kilogram.
11:28
Josh: So again, this is fully contingent on Starship being able to get it up into orbit.
11:31
Josh: The idea is that commercial access should open up around 2028-2029 given the
11:38
Josh: amount of time that it's going to take to get Starship online and then start launching these,
11:43
Josh: aircraft in space but it's a really cool announcement where we have an actual,
11:47
Josh: client for SpaceX that is going to be launching serious payloads into space
11:50
Josh: and fully pursuing the AI data centers in orbit plan.
11:54
Josh: It's going to take a little while but this is where we build the foundation
11:57
Josh: this is where companies are starting to plan these launches for the next couple of years.
12:02
Ejaaz: We have another SpaceX AI alliance member Josh you know over the last couple
12:06
Ejaaz: of weeks you've got Elon partnering with Anthropic twice you've got them partnering
12:11
Ejaaz: pretty aggressively with cursor,
12:12
Ejaaz: which is going to result in a $60 billion acquisition 30 days after the IPO,
12:16
Ejaaz: which is expected to be sometime next month.
12:19
Ejaaz: And now you've got this partnership with StarCloud. Now, what I love about this
12:22
Ejaaz: is if you think of SpaceX owning the highway or the toll booth into space,
12:27
Ejaaz: they're going to need the goods to be putting out into space, right?
12:30
Ejaaz: So they've got Anthropik to kind of tentatively agree to say like,
12:33
Ejaaz: yeah, we're going to actually use some of your space AI data centers,
12:36
Ejaaz: but also you need the goods to get out there.
12:39
Ejaaz: Now, Elon Musk is investing a ton of money over the next five years into something
12:42
Ejaaz: called the TerraFab, which will end up becoming the largest chip fab in America
12:47
Ejaaz: if it plays out how he plans it to.
12:49
Ejaaz: And there's an interesting statistic from that goal, which is 70% of the chips
12:53
Ejaaz: that are produced are going to be radiation hardened and sent out into space.
12:56
Ejaaz: So he's very much like committing his entire focus and goal into this.
13:00
Ejaaz: That's why the SpaceX IPO is valued at expectedly a $2 trillion valuation.
13:06
Ejaaz: But he needs the goods to be able to do that now I'm looking at this tweet up here
13:10
Ejaaz: Saying that StarCloud recently, or is in the process of raising $200 million
13:13
Ejaaz: at a $2 billion valuation, that is cheap.
13:16
Ejaaz: If StarCloud can actually put out radiation-hardened GPUs, which, by the way, they have.
13:21
Ejaaz: There's currently, I think it's like a stack of like H100s that are currently
13:25
Ejaaz: orbiting space that are training and inferencing on it, as Gavin mentioned in the previous tweet.
13:28
Ejaaz: What's stopping Elon from acquiring them and now having the actual goods to
13:32
Ejaaz: be able to do that? Obviously, he's still relying on NVIDIA.
13:34
Ejaaz: He has a good partnership with them. But I just I'm extremely bullish about
13:37
Ejaaz: this, and I like that he's choosing to kind of partner with different firms
13:41
Ejaaz: versus trying to build everything himself.
13:43
Ejaaz: But moving on, there is another company that is raising a pretty decent valuation
13:47
Ejaaz: right now, Cognition, which is the, I believe they started Devin,
13:51
Ejaaz: yeah, two years ago, which was the OG AI agent.
13:55
Ejaaz: They raised over a billion dollars at $26 billion valuation.
13:59
Ejaaz: Now, the coolest part about these guys is everyone thought that they were completely
14:03
Ejaaz: cooked after Anthropic and OpenClaw kind of like launched their like autonomous agents.
14:09
Ejaaz: But what I liked about Cognition is they have a lot of staying power.
14:12
Ejaaz: And we've spoken about on previous episodes that there's this company called Cursor.
14:16
Ejaaz: And everyone blamed them for being like kind of like, or criticized them for
14:19
Ejaaz: being like a rapper, a thin rapper around an AI model.
14:21
Ejaaz: But what ended up being the case was they had this sort of agent harness,
14:25
Ejaaz: this agent mode, which kind of like orchestrates how models work.
14:28
Ejaaz: Uses the right models, uses the right prompts, and it's an incredibly valuable
14:33
Ejaaz: mode when it comes to training a model.
14:35
Ejaaz: Cognition is probably second in the ranking of having or owning this particular type of mode.
14:41
Ejaaz: And Josh, I know you wrote an entire essay about this recently.
14:44
Josh: I'm a big fan of harnesses. I really like harnesses. If you haven't subscribed
14:48
Josh: to a newsletter, I would highly recommend.
14:49
Josh: I released an entire essay about harnesses earlier this week.
14:52
Josh: It's basically all of the peripherals required to turn the brain of an LLM into
14:57
Josh: a full body, into something fully capable of doing agentic tasks that are long-running.
15:01
Josh: One thing I found funny from this announcement was the idea that Cognition uses
15:04
Josh: now 90% of their own code.
15:07
Josh: And if they're valued at a $26 billion valuation, clearly there's a lot of value
15:10
Josh: there. So this is a continuation of the wrapper wars.
15:13
Josh: We now kind of have very clear winners on who's winning the LLM wars,
15:17
Josh: the large language models that sit on the frontier, how you package them,
15:21
Josh: how you add a harness for them will be the differentiator.
15:23
Josh: We saw Cursor with a $60 billion deal with SpaceX. Now we have Cognition raising another big round.
15:28
Josh: The current war, that new frontier, is being fought on the wrapper level with the harnesses.
15:34
Josh: Cognition's the new eccentric in that. I'm pretty bullish, pretty excited on Cognition.
15:38
Ejaaz: Josh, on your bingo card for this week, did you expect the Pope to sign a partnership
15:42
Ejaaz: with the Major Frontier AI Lab?
15:44
Josh: A partnership? Tell me about this partnership. What do you mean partnership?
15:48
Ejaaz: Okay, so listen, we don't typically get into the swing of reviewing biblical
15:53
Ejaaz: architecture here on this show, but we are going to today.
15:56
Josh: Who got the mandate of heaven?
15:58
Ejaaz: Yeah, yeah, exactly. So Pope Leo kind of blew the newsroom wide open this week,
16:04
Ejaaz: releasing his Magnifica Humanitas of His Holiness, Pope Leo,
16:09
Ejaaz: on safeguarding the human person in the time of artificial intelligence.
16:12
Ejaaz: Now, my initial reaction to this is,
16:15
Ejaaz: I actually kind of love it because kind of like the religious pious leaders
16:19
Ejaaz: don't tend to kind of like technology a lot or get involved with it much.
16:23
Ejaaz: The fact that he's written, I believe this is like a 50 page plus script on
16:28
Ejaaz: his kind of like thoughts of how this plays out,
16:30
Ejaaz: how AI is going to evolve, how it kind of equates to like ethical use and religious
16:37
Ejaaz: scripts. He gets into a whole thing.
16:39
Ejaaz: Like there's this thing I was reading earlier on where it's like between two
16:41
Ejaaz: biblical images and talks about the construction of the Tower of Babel and these
16:45
Ejaaz: two scenes within the Bible.
16:46
Ejaaz: And the main takeaway that I got from this entire thing is he's trying to be
16:50
Ejaaz: extremely thoughtful and careful about how these models are developed because
16:54
Ejaaz: he knows about the power of influence that these models will have on kids,
16:58
Ejaaz: young adults, and everyone across every single age in the future.
17:03
Ejaaz: And so he knows that it's gonna guide a lot of education and learning about
17:06
Ejaaz: religious scripts and all these kinds of things. And so he wants to make sure
17:09
Ejaaz: that it's developed in the right way.
17:10
Ejaaz: Now, one of the biggest call to actions that he has out of this entire document
17:13
Ejaaz: and the preceding hearing or talk or discussion that they had after this was,
17:18
Ejaaz: He wants there to be governance instilled into this.
17:21
Ejaaz: Now, the governance, he suggests, comes from two particular parties,
17:24
Ejaaz: one being the national government that runs the entire country,
17:27
Ejaaz: respective country where it's operating in,
17:29
Ejaaz: the model it's operating in, as well as the church or religious groups involved
17:34
Ejaaz: so that they can kind of mold the personality and ethics of this.
17:37
Ejaaz: Now, when I kind of like speak about this, I think of one particular AI lab,
17:41
Ejaaz: which is Anthropic, who have been speaking to religious leaders independently
17:45
Ejaaz: of their own accord over the last, I believe, couple of months.
17:48
Ejaaz: And the Pope actually calls that Anthropic saying like, we would like to walk
17:52
Ejaaz: together. So I thought that was a pretty interesting update.
17:55
Josh: That's fun. It's nice when there's more awareness being raised around the idea.
18:00
Josh: It's like this will bring a lot of new eyes to the world of AI.
18:03
Josh: It will start a conversation that is going to need to happen.
18:06
Josh: It is going to bring awareness in a way that is constructive,
18:09
Josh: not damaging, hopefully. And I think it's just, it's a fun...
18:13
Josh: Cool thing and it's a testament to how large ai has
18:16
Josh: gotten where this is now sitting at a global scale and it is top of
18:19
Josh: mind and i like the idea that all of these people are going to be working together
18:23
Josh: to figure out the best way to navigate forward as a society and that that's
18:27
Josh: pretty exciting so glad i'm glad the pope's getting on board and uh hopefully
18:31
Josh: a lot of people who you know respect and trust him will take ai a little bit
18:35
Josh: more seriously and start considering this conversation and.
18:37
Ejaaz: Then next on the docket we have two more topics we're going to breeze through
18:40
Josh: It All right, bang them out.
18:41
Ejaaz: Josh, something that you and I complain about a lot is when a new model releases,
18:46
Ejaaz: you have these fancy benchmarks, right?
18:48
Ejaaz: But we don't know what that means. And often these benchmarks are maxed out
18:52
Ejaaz: or gamified and doesn't actually tell us or portray what the model can actually do.
18:57
Ejaaz: So let's say a model is super good at coding and then a software engineer actually
19:00
Ejaaz: uses it for their specific work. It's actually not good enough.
19:05
Ejaaz: Now, what a bunch of people ended up doing was they developed this new benchmark
19:08
Ejaaz: called DeepSWE, and it focuses on agentic AI engineering specifically.
19:16
Ejaaz: So when you spin up multiple instances of Claude or ChatGPT,
19:20
Ejaaz: for instance, to do a bunch of your coding work in parallel, how good is it actually?
19:24
Ejaaz: Now, the typical benchmark, and this is, by the way, like the holy Bible, to use a pun here,
19:29
Ejaaz: of the holy grail of benchmarks, which is software engineering verified,
19:34
Ejaaz: which is what every AI lab kind of like shows off when they release a new model.
19:40
Ejaaz: Currently, Claude and ChatGPT
19:43
Ejaaz: at the top of this, but it's tested against pre-approved solutions and answers
19:50
Ejaaz: against a particular problem. So the model already knows the answer.
19:53
Ejaaz: And so it's not really authentic or genuine.
19:55
Ejaaz: What this benchmark ends up doing is providing not only new problems of which
19:59
Ejaaz: the models have no idea what the answer is, they're not given the answers beforehand,
20:02
Ejaaz: but also real long horizon tasks that go for between 7 to 12 hours.
20:06
Ejaaz: And so they're put to the test. And we have a new ranking over here.
20:10
Ejaaz: GPT 5.5 and GPT 5.4, their predecessor model is number one and two of this new benchmark.
20:16
Ejaaz: You've got Claude Opus 4.7 just behind on 54% and Sonnet 4.6 as well.
20:21
Ejaaz: Gemini 3.5 Flash, even though it is their latest model, comes out at a pretty poor 28%.
20:26
Josh: Yikes. I do like the idea of more benchmarks. I think benchmarks are fun. They are important.
20:31
Josh: They are oftentimes games. So if we can come up with ways in which we could
20:34
Josh: organically test these,
20:35
Josh: if you ask me the way that i choose to
20:38
Josh: do my benchmarks it's purely vibes based i'll get
20:41
Josh: in i'll do the day-to-day task you could very much feel a model
20:44
Josh: if you use it every day you could feel where it's better feel where it's worse i find
20:47
Josh: that they're oftentimes spiky in the sense that they're very very
20:50
Josh: good at specific things and oftentimes a lot weaker than others and that
20:53
Josh: kind of spike complex changes over time as these
20:56
Josh: new models get released but i'm glad we have a more formal way
20:59
Josh: of doing this greater than vibes and greater than the
21:02
Josh: things that have been gamed recently now last thing on the agenda is
21:05
Josh: snowflake stock um the headline i'm seeing is snowflake stock jumps 30 percent
21:11
Josh: in a day this was after the earnings report that just came out this week and
21:16
Josh: for those that don't know snowflake is a cloud compute providing platform and
21:20
Josh: clearly they've done something right because uh wow this is a lot of revenue in one day.
21:24
Ejaaz: Yeah, so to give a bit of context here,
21:27
Ejaaz: there's something, there's a phenomenon happening called the SaaS-pocalypse,
21:32
Ejaaz: software-as-a-service apocalypse, where basically as Claude and ChatGPT end
21:36
Ejaaz: up upgrading their own models,
21:38
Ejaaz: they become able to replace a bunch of subscription software tools and services
21:44
Ejaaz: because they could just vibe code it or rebuild it from scratch.
21:47
Ejaaz: And this has led to a massive decline. I'm talking tens to hundreds of billions
21:51
Ejaaz: of dollars lost in the stock market for all these different SaaS products.
21:54
Ejaaz: I remember a time where Anthropic would release a plugin update for,
21:57
Ejaaz: let's say, a security review tool, and all the cybersecurity stocks were absolutely done, right?
22:02
Ejaaz: And so the point here is the trend has been kind of not only broken, but inverted.
22:07
Ejaaz: Snowflake, which is a major SaaS provider for a lot of enterprise companies,
22:11
Ejaaz: not only had their stock jump up 35%, but it's because they defined a moat that
22:15
Ejaaz: sat on top of Claude and ChatGPT.
22:17
Ejaaz: Now, in previous episodes, we've spoken about Anthropic and OpenAI signing what's
22:21
Ejaaz: known as a joint venture with some major private equity firms.
22:24
Ejaaz: And the intention of this is to place their own engineers, they're known as
22:28
Ejaaz: forward-deployed engineers, into Fortune 100 companies so that they can kind
22:33
Ejaaz: of like design and architect-specific workflows, so AI-agentic workflows,
22:37
Ejaaz: using Core or using ChatGPT.
22:39
Ejaaz: Now, the only reason why they would be doing that is because their own generalized
22:42
Ejaaz: LLM isn't good enough. And so you need someone to kind of like guide this kind of orchestration.
22:47
Ejaaz: Snowflake is basically arguing in their latest quarterly earnings that they already have that data.
22:51
Ejaaz: They already have that design and they're doing it themselves using a mixture of Claude and ChatGPT.
22:56
Ejaaz: And that's why the stock jumped up massively. It just shows that not only are
23:00
Ejaaz: SaaS companies here to stay, but they also might be having or creating a defensible moat.
23:04
Josh: I feel like we need a dartboard that we can start throwing darts at to pick
23:07
Josh: the companies that are going to go up 30% because every week it seems like there
23:10
Josh: is 10 new ones that are going absolutely nuclear.
23:12
Josh: But that wraps up the episode today. Thank you all so much for watching.
23:15
Josh: If you have made it through all of our previous episodes this week,
23:18
Josh: you are now fully caught up.
23:19
Josh: You can go touch grass, enjoy your weekend, have an amazing time.
23:21
Josh: And we will be back as always with a new episode starting bright and early next week.
23:25
Josh: Thank you guys so much for watching. Have an amazing weekend and we'll see you next time.