Latest / In the Loop AI / OpenAI Just Gave Away Their Secret Formula... For Free?

Transcript

OpenAI Just Gave Away Their Secret Formula... For Free? · In the Loop AI

0:03
Josh: The unthinkable has just happened open ai
0:06
Josh: has released an open source model open ai
0:09
Josh: has been closed ai since the time that i knew them
0:12
Josh: they have been named themselves open ai they were not
0:15
Josh: open source they have finally released an open source model and surprise surprise
0:18
Josh: it's actually really great and i think the downstream implications of an open
0:22
Josh: source model from a company like this that is this good are really it's a really
0:26
Josh: big deal i think this really matters a lot just yesterday they announced the release of GPT-OSS.
0:32
Josh: There are two models. There's a 120 billion parameter model and there's a 20
0:36
Josh: billion parameter model. We're going to get into benchmarks.
0:38
Josh: We're going to get into how good they are.
0:40
Josh: But the idea is that OpenAI has actually released an open source model.
0:44
Josh: And this can compare to the Chinese models because we recently had DeepSeek and we've had Kimi.
0:49
Josh: And those would be very good. But this is the first really solid American-based open source model.
0:54
Josh: So Ijaz, I know you've been kind of digging in the weeds about how this works.
0:57
Josh: Can you explain us exactly why this is a big deal why this happened what's going on here
1:00
Ejaaz: Yeah it's it's pretty huge so so here
1:03
Ejaaz: are the hot highlights um as you mentioned there's two
1:06
Ejaaz: models that came out the 20 billion parameter model which is actually small
1:09
Ejaaz: enough to run on your mobile phone right now and they have a 120 billion parameter
1:14
Ejaaz: model which is big but still small enough to run on a high performance laptop
1:19
Ejaaz: so if you guys have a macbook out there jump in go for it um it's fully customizable.
1:25
Ejaaz: So remember, open source means
1:26
Ejaaz: that you can literally have access to the design of the entire model.
1:30
Ejaaz: It's like OpenAI giving away their secret recipe to how their frontier models
1:34
Ejaaz: work. And you can kind of like recreate it at home.
1:37
Ejaaz: This means that you can customize it to any kind of use case that you want,
1:41
Ejaaz: give it access to all your personal hard drives, tools, data,
1:44
Ejaaz: and it can do wonderful stuff.
1:46
Ejaaz: But Josh, here's the amazing part.
1:48
Ejaaz: On paper, these models are as good as GPT-4 mini models, which is,
1:54
Ejaaz: it's pretty impressive, right?
1:56
Ejaaz: In practice and i've been playing around with it for the last few hours they're
2:00
Ejaaz: as good in my opinion and actually quicker than
2:03
Ejaaz: gpt-03 which is their frontier model and i
2:06
Ejaaz: mean this across like everything so
2:09
Ejaaz: reasoning um it spits out answers super quickly and i can see its reasoning
2:14
Ejaaz: it happens in like a couple of seconds and i'm so used to waiting like 30 seconds
2:18
Ejaaz: to a couple minutes on gpt-03 josh so it's pretty impressive and an insane unlock
2:23
Ejaaz: on coding it's as good and on creativity as well.
2:28
Ejaaz: So I'm my mind's pretty blown at all of this, right? Josh, what do you what do you think?
2:32
Josh: Yeah, so here's why it's impressive to me is because a lot of the times I don't
2:35
Josh: really care to use the outer bands of what a model is capable of.
2:38
Josh: Like I am not doing deep PhD level research. I'm not solving these math Olympiad questions.
2:43
Josh: I'm just trying to ask it a few normal questions and get some answers.
2:46
Josh: And what these models do is an excellent job at serving that need.
2:49
Josh: They're not going to go out and solve the world's hardest problems,
2:51
Josh: but neither do I. I don't want to solve those problems.
2:54
Josh: I just kind of want the information that I want, whether it be just a normal
2:57
Josh: Google type search or whether it be asking it some miscellaneous question about
3:01
Josh: some work that I'm doing.
3:03
Josh: It's really good at answering that. So I think initial impressions,
3:07
Josh: because they did allow you to test it publicly through their website,
3:10
Josh: it's just really good at the things that I want.
3:12
Josh: So the fact that I can run one of these models on a local device on my iPhone,
3:16
Josh: well, it feels like we're reaching this place that AI is starting to become
3:19
Josh: really interesting because for so long we've had compute handled fully on the
3:23
Josh: cloud and now this is the first time where
3:26
Josh: Compute can really happen on your computer. It could happen on your laptop.
3:28
Josh: I could download the model and I could actually store the model,
3:31
Josh: the 120 billion parameter model on a 56 gigabyte USB drive.
3:35
Josh: So you can take the collective knowledge of the world and put it on a tiny little USB drive.
3:40
Josh: And granted, it needs a bit of a bigger machine to actually run those parameters,
3:43
Josh: but you can install all the weights. It's 56 gigabytes.
3:46
Josh: It's this incredibly powerful package. And it probably, I don't know if this
3:51
Josh: is true, but it's probably the most condensed knowledge base in the history of humanity.
3:55
Josh: They've really managed to take a tremendous amount of tokens,
3:59
Josh: smush them into this little parameter set, and then publish it for people to
4:02
Josh: use. So for me, I'm really excited.
4:04
Josh: I like having my own mini portable models. I am excited to download,
4:08
Josh: try it out, run it on my MacBook.
4:09
Josh: I'm not sure I could run the 120 billion parameter model, but at least the 20B
4:12
Josh: and give it a shot and see how it works.
4:14
Ejaaz: You need to get the latest MacBook, Josh. I know, I got to upgrade. We can test that out.
4:19
Ejaaz: What I also love about it is it's fully private, right? So you can give it access
4:24
Ejaaz: to your personal hard drive, your Apple Notes, whatever you store on your computer, basically.
4:31
Ejaaz: And you can basically instruct the model to use those different tools.
4:36
Ejaaz: So one review that I keep seeing from a number of people who have been testing
4:39
Ejaaz: it so far is that it's incredibly great and intuitive at tool use.
4:44
Ejaaz: And the reason why this is such a big deal is a lot of the Frontier models right
4:48
Ejaaz: now, when they allow you to give access to different tools, they're kind of clunky.
4:53
Ejaaz: The model doesn't actually know when to use a specific tool and when not to.
4:56
Ejaaz: But these models are super intuitive, which is great. The privacy thing is also
5:00
Ejaaz: a big thing because you kind of
5:02
Ejaaz: don't want to be giving all your personal information away to Sam Altman.
5:05
Ejaaz: But you want a highly personalized model.
5:08
Ejaaz: And I think if I was to condense this entire model release in a single sentence,
5:13
Ejaaz: Joss, I think I would say it is the epitome of privacy and personalization in an AI model so far.
5:20
Ejaaz: It is that good. it is swift it is cheap and I'm going to replace it completely
5:26
Ejaaz: with all my GPT-4.0 queries as you said earlier like,
5:30
Ejaaz: Who needs to use the basic models anymore when you have access to this?
5:34
Josh: Yeah. So it's funny you say that you're going to swap it because I don't think I'm going to swap it.
5:38
Josh: I still am not sure I personally have a use case right now because I love the
5:41
Josh: context. I want the memory.
5:42
Josh: I like having it all server side where it kind of knows everything about me.
5:46
Josh: I guess in the case that I wanted to really make it a more intimate model experience
5:50
Josh: where you want to sync it up with like journal entries or your camera roll or
5:55
Josh: whatever, whatever interesting like personal things, this would be a really cool use case.
5:59
Josh: I think for the people who are curious why this matters to them,
6:03
Josh: well, we could talk a little briefly about like the second order effects of
6:07
Josh: having open source models as powerful, because what that allows you to do is
6:10
Josh: to serve queries from a local machine.
6:13
Josh: So if you are using an app or let's say you're an app developer and you're building
6:17
Josh: an application and your app is serving millions of requests because it's a GPT wrapper.
6:22
Josh: Well, what you could do now is instead of paying API calls to the OpenAI server,
6:26
Josh: you can actually just run your own local server, use this model,
6:29
Josh: and then serve all that data for the cost of the electricity.
6:31
Josh: And that's a really big unlock for the amount of compute that's going to be
6:34
Josh: available for not only developers, but for the cost of the users in a lot of these applications.
6:38
Josh: So for the applications that aren't doing this crazy moon math and that are
6:42
Josh: just kind of serving basic queries all day long, this like really significantly drops the cost.
6:47
Josh: It increases the privacy, like you mentioned. And there's a ton of really important
6:51
Josh: upsides to open source models that we just haven't seen up until now.
6:54
Josh: And I'm very excited to see come forward.
6:58
Ejaaz: Well, Josh, the thing with most of these open source models,
7:01
Ejaaz: we spoke about actually two major Chinese open source models that were released last week.
7:06
Ejaaz: It's not accessible to everyone. Like you and me aren't necessarily going to
7:10
Ejaaz: go to Hugging Face, a completely separate website, download these models,
7:14
Ejaaz: run the command line interface.
7:16
Ejaaz: Most of the listeners on the show doesn't even know what that means.
7:18
Ejaaz: I don't even know if I know what that means, right?
7:20
Ejaaz: But here you have a lovely created website where you could just kind of log
7:25
Ejaaz: on and play around with these open source models. And that's exactly what I've been doing.
7:28
Ejaaz: I actually have a few kind of demo queries that I ran yesterday, Josh.
7:34
Josh: Yeah, walk us through, let's see.
7:35
Ejaaz: Okay, so there's an incredibly complex test, which a lot of these AI models,
7:41
Ejaaz: which cost hundreds of billions of dollars to train, can't quite answer.
7:45
Ejaaz: And that is how many R's, the letter R's are there in the word strawberry? Most say two.
7:52
Josh: The bar's on the floor,
7:53
Ejaaz: Huh? Yeah, if we were to go with most models, they say two. They're convinced that they are only two.
7:58
Ejaaz: And I ran that test today, rather yesterday, with these open source models,
8:03
Ejaaz: and it correctly guessed three, Josh. So we're one for one right now.
8:07
Josh: We're on our way.
8:08
Ejaaz: But then I was like, okay, we live in New York City. I love this place.
8:12
Ejaaz: I'm feeling a little poetic today. Can you write me a sonnet?
8:16
Ejaaz: And my goal with this wasn't to test whether it could just write a poem.
8:19
Ejaaz: It was to test how quickly it could figure it out.
8:22
Ejaaz: And as you see it thought for a couple of seconds on this so it literally spat
8:26
Ejaaz: this out in two seconds um and it was structured really well you know it kind
8:30
Ejaaz: of flowed would i be you know reciting this out loud to the public no but you
8:34
Ejaaz: know i was pretty impressed.
8:35
Ejaaz: And then, Josh, I was thinking, you know, what's so unique about open source models?
8:40
Ejaaz: You just went through a really good list of why open source models work.
8:44
Ejaaz: But I was curious as to why these specific open source models were better than
8:48
Ejaaz: other open source models or maybe even other centralized models.
8:51
Ejaaz: So I wrote a query. I decided to ask it. I was like, you know,
8:54
Ejaaz: tell me some things that you could do that are the larger centralized models.
8:57
Ejaaz: And I spat out a really good list. I'm not going to go through all of them,
9:00
Ejaaz: but, you know, some of the things that we've highlighted so far, you can fine tune it.
9:03
Ejaaz: It's privacy. See, I really like this point that it made, Josh,
9:06
Ejaaz: that it just shows that AI is probably getting smarter than us,
9:08
Ejaaz: which is you can custom inject your own data into these models.
9:13
Ejaaz: Now, without kind of digging deeper into this, when you use a centralized model,
9:18
Ejaaz: it's already pre-trained on a bunch of data that companies like Anthropic and
9:23
Ejaaz: Google have already fed it.
9:25
Ejaaz: And so it's kind of formed its own personality, right?
9:29
Ejaaz: So you can't change the model's personality on a centralized model.
9:32
Ejaaz: But with an open model you have full reign to do whatever you want and so if
9:37
Ejaaz: you were feeling kind of uh adventurous you could use your own data and make
9:41
Ejaaz: it super personal and customizable so i thought that was really cool and fun
9:44
Ejaaz: demo josh have you been playing around with this.
9:46
Josh: Yeah it's um it's it's smart it's fun it's smart i wouldn't say it's anything
9:50
Josh: novel the like query results that i get are you know on par with everything
9:54
Josh: else i don't notice the difference which is good because it means they're performing
9:57
Josh: very well it's not like i feel like i'm getting degraded performance because
10:01
Josh: I'm using a smaller model.
10:02
Josh: But it's just like it's nothing too different, I would say.
10:06
Josh: The differences, I mean, again, all this boils down to the differences of it
10:09
Josh: being open source versus being
10:10
Ejaaz: Run on the server. Well, let me challenge you that, right? OK,
10:13
Ejaaz: so you're saying it's good but nothing novel.
10:16
Ejaaz: Would you say it's as good as GPT-4.0,
10:20
Ejaaz: minus the memory let's just put memory aside for a second would you use it if
10:24
Ejaaz: it had memory capability.
10:26
Josh: Actually no probably not um i still wouldn't
10:29
Josh: because i love my desktop application too much i
10:32
Josh: love my mobile app too much and i like that the conversations are
10:35
Josh: shared in the cloud um so i can use them on my phone i could
10:38
Josh: start on my laptop and go back and forth so even in
10:41
Josh: that case i'm probably still not a user um because
10:44
Josh: the convenience factor but there are there are a
10:47
Josh: lot of people and a lot of industries that would be and this is actually something probably
10:50
Josh: worth surfacing is the new industries that are now able to
10:52
Josh: benefit from this because a lot of industries have
10:55
Josh: a tough time using these AI models because
10:58
Josh: of the data privacy concerns particularly I mean if you think about a
11:01
Josh: healthcare industry people who are dealing with patients data it's
11:04
Josh: very challenging for them to fork it over to open AI and just trust that they're
11:07
Josh: going to keep it safe so what this does is it actually allows companies that
11:11
Josh: are in like the healthcare industry the finance industry who's dealing with
11:13
Josh: very like high touch personal finance the legal industry who's dealing with
11:16
Josh: a lot of legality government and defense a lot of these industries that were
11:20
Josh: not previously able to use these popular AI models,
11:24
Josh: well, now they have a pretty good model that they could run locally on their machines.
11:27
Josh: And that doesn't have any possibility of actually leaking out their customer
11:31
Josh: data, leaking out financials or healthcare data or, or like any sort of legal documents.
11:35
Josh: And, and that feels like a super powerful unlock. So for them,
11:38
Josh: it feels like a no brainer, obviously get the 120 B model running on a local
11:43
Josh: machine inside of your office, and you can load it up with all this context.
11:46
Josh: And that seems to be who this would be most impacting, right?
11:51
Ejaaz: But still to that point, I wonder how many of these companies can be bothered
11:56
Ejaaz: to do that themselves and run their own internal kind of like infrastructure.
12:01
Ejaaz: I'm thinking about OpenAI, who cracked, I think, $10 billion in annual recurring
12:06
Ejaaz: revenue this week, which is like a major milestone.
12:09
Ejaaz: And a good chunk of that, I think 33% of that is for enterprise customers.
12:14
Ejaaz: And to your point, like these enterprise customers don't wanna be giving open
12:18
Ejaaz: AI their entire data. You know, they can be used to train other AI models.
12:21
Ejaaz: So their fix or solution right now is they use kind of like private cloud instances,
12:28
Ejaaz: that I think are supplied by Microsoft by their Azure cloud service or something like that.
12:32
Ejaaz: And I wonder if they chose that,
12:35
Ejaaz: One, because there wasn't any open source models available or because they kind
12:40
Ejaaz: of just want to offload that to Microsoft to deal with.
12:42
Ejaaz: My gut tells me they're going to want to go with the latter,
12:46
Ejaaz: which is like, you know, just give it to some kind of cloud provider to deal with themselves.
12:50
Ejaaz: And they just trust Microsoft because it's a big brand name.
12:52
Ejaaz: But yeah, I don't really know how they'll materialize. I still think,
12:55
Ejaaz: and maybe this is because of my experience in crypto, Josh, that the open source
13:00
Ejaaz: models are still for like people that are at the fringe that are really experimenting
13:04
Ejaaz: with these things. but maybe don't have billions of dollars.
13:07
Josh: Yeah, that could be right. It'll be interesting to see how it plays out on all
13:10
Josh: scale of businesses because I mean, as a, like I think of a lot of indie devs
13:15
Josh: that I follow on Twitter and I see them all the time
13:17
Josh: just running local servers and they just, if they had this local model that
13:20
Josh: they could run on their machine and it takes the cost per query down from like
13:24
Josh: a penny to zero, that's like a big zero to one change.
13:28
Josh: So he does this model special because there are also a number of breakthroughs
13:32
Josh: that occurred in order to make this possible,
13:34
Josh: in order to condense this knowledge to be so tight so here's this
13:37
Josh: tweet from the professor talking about the cool tech tweaks in
13:40
Josh: this new model and what open ai was able to achieve some of
13:43
Josh: these i believe are novel some of these are seen before um if
13:46
Josh: you look at point two mixture of experts we're familiar with mixture of experts
13:49
Josh: we've seen other companies use that like kimmy and deep
13:52
Josh: seek basically instead of one brain doing everything the ai
13:54
Josh: has this team of experts that are kind of like mini brains
13:57
Josh: and specialize in different tasks it picks the right expert for
14:00
Josh: the job and it makes it faster so like instead of
14:03
Josh: having the entire 120 million parameter model search for one question maybe
14:08
Josh: you just take a couple million of those parameters that are really good at solving
14:11
Josh: math problems and they use it and that that's what brings compute down the first
14:15
Josh: point is this thing called the sliding window attention so if you imagine an
14:18
Josh: ai is like reading a really long book
14:21
Josh: It can only focus on a few pages at a time this trick
14:24
Josh: kind of lets it slide its focus window along the text so
14:27
Josh: when you think of a context window generally it's fixed right where you can see
14:30
Josh: a fixed set of data this sliding window
14:33
Josh: attention allows you to kind of move that context back and forth a
14:36
Josh: little bit so it takes what would have normally been
14:39
Josh: a narrow context window and extends it out a little bit to
14:42
Josh: the side so you get a little bit more context which is great for a
14:44
Josh: smaller model again you really want to consider that all of these are
14:47
Josh: are optimized for this microscopic scale that
14:50
Josh: can literally run on your phone and then the third point is this
14:52
Josh: thing called rope with yarn which sounds like a cat toy but this
14:56
Josh: is how the ai keeps track of the order of words so like the position
14:59
Josh: of the words in a sentence um so rope
15:02
Josh: you could imagine it like like the twisty math way to do
15:05
Josh: it and yarn makes it stretch further for really long stuff
15:07
Josh: so we have the context window that is
15:10
Josh: sliding we have this rope with yarn that allows you
15:13
Josh: to just kind of like stretch the words a little bit further and
15:16
Josh: then we have attention sinks which is the last one which is
15:19
Josh: there's a problem when ai is dealing with these endless chats that
15:22
Josh: lets it it kind of sinks in or ignores the boring old
15:25
Josh: info so it can pay attention to the new stuff so basically what it
15:28
Josh: is is if you're having a long chat with it and it determines hey this stuff
15:32
Josh: is kind of boring i don't need to remember it it'll actually just throw it away
15:35
Josh: and it'll increase that context window a little bit so again hyper optimizing
15:39
Josh: for for the small context window that it has and those are kind of the key four
15:42
Josh: breakthroughs that made this special again i'm not sure any of them are particularly novel,
15:47
Josh: But when combined together, that's what allows you to get these 04 mini results
15:52
Josh: or even 03 results on the larger model on something that can run locally on your laptop.
15:56
Josh: So it's a pretty interesting set of breakthroughs. I think a lot of times OpenAI,
16:01
Josh: we talk about them because of their feature breakthroughs, not really their
16:04
Josh: technical breakthroughs.
16:05
Josh: I think a lot of times the technical breakthroughs are reserved for like the
16:08
Josh: Kimi models or the DeepSeq models
16:09
Josh: where they really kind of break open the barrier of what's possible.
16:12
Josh: But I don't want to discredit OpenAI because these are pretty interesting things
16:16
Josh: that they've managed to combine together into this like one cohesive,
16:19
Josh: tiny little model, and then just gave it away.
16:22
Ejaaz: Yeah. I mean, they actually have a history of front-running open source frontier breakthroughs.
16:29
Ejaaz: If you remember when DeepSeek got deployed, Josh, one of their primary training
16:34
Ejaaz: methods was reinforcement learning, which was pioneered by an open AI researcher,
16:38
Ejaaz: which who probably like now works at Meta.
16:40
Ejaaz: Yeah, and I was I was I was looking at the feature that you mentioned just not
16:46
Ejaaz: the feature, but the breakthrough sliding window attention, and you mentioned
16:48
Ejaaz: that it can basically toggle reasoning.
16:51
Ejaaz: And I was pleasantly surprised to just notice that on the actual interface of
16:54
Ejaaz: the models here, Josh, can you see over here?
16:57
Ejaaz: You can toggle between reasoning levels of high, medium and low.
17:01
Ejaaz: So depending on what your prompt or query is, if it is kind of like a low level
17:04
Ejaaz: query where you're like hey just record this shopping or grocery list you know
17:09
Ejaaz: that's probably like a medium or a low query so oh it's pretty cool to to see
17:12
Ejaaz: that surface to the user like see it actively being used.
17:15
Josh: Yeah, no, super cool. I think I like the fine tuning of it.
17:19
Josh: And again, allowing you to kind of choose your intelligence levels,
17:22
Josh: because I imagine a lot of average people just don't, a lot of average queries
17:25
Josh: just don't need that much compute.
17:27
Josh: So if you can toggle it for the low reasoning level and get your answers,
17:30
Josh: that that's amazing. Super fast, super cheap.
17:32
Ejaaz: Did you see that trending tweet earlier this week, Josh, which basically said
17:37
Ejaaz: that the majority of ChatGPT users have never used a different model than ChatGPT 4.0?
17:42
Josh: I haven't seen it, but that makes sense.
17:43
Ejaaz: Yeah i i feel like the bulk of people i was chatting to
17:46
Ejaaz: my sister yesterday and she was kind of
17:49
Ejaaz: like using it for some research project at work and the
17:52
Ejaaz: screenshot she sent me over was foro and i was like hey you know like
17:55
Ejaaz: you could just run this on like a model that's like
17:58
Ejaaz: five times better than this right uh we'll come
18:01
Ejaaz: up with a much more creative set of ideas so just made me think that
18:04
Ejaaz: like i don't know how many people like care that they are like
18:06
Ejaaz: these brand new novel models and maybe um you know
18:09
Ejaaz: this kind of like basic model is good enough for everyone i don't know
18:12
Ejaaz: but um but moving on josh um there
18:15
Ejaaz: was a big question that popped into my head as
18:18
Ejaaz: soon as these models released which was are they as good
18:21
Ejaaz: as the chinese open source models right i wanted
18:24
Ejaaz: to get some opinions from people and and the reason
18:26
Ejaaz: why this matters i'm just give the listeners some context
18:29
Ejaaz: is china has been the number one
18:32
Ejaaz: nation to put out the best open source
18:35
Ejaaz: models over the last 12 months it started with deep seek
18:38
Ejaaz: and then alibaba's quen models got involved
18:42
Ejaaz: and then recently we had kimmy k2 and i think
18:44
Ejaaz: there was another ai lab out of china which came out so they
18:47
Ejaaz: have outside of america the highest density.
18:51
Ejaaz: Of the top ai researchers they all come out of this one university
18:54
Ejaaz: zinghua i believe they kind of like partially work
18:57
Ejaaz: or train in the u.s as well so they've got this like kind of hybrid ai
19:00
Ejaaz: mentality of how to build these models and they come up with a lot of these
19:03
Ejaaz: frontier breakthroughs um kimmy k2 for context had uh one trillion parameters
19:09
Ejaaz: in their model right comparing this to like 120 billion and 20 billion parameters
19:14
Ejaaz: models from open air i was curious like does this beat them to the punch some people josh.
19:20
Ejaaz: Don't think so okay this guy jason lee
19:23
Ejaaz: he asks uh is the gpt oss stronger
19:27
Ejaaz: than quen or kimmy or chinese open models and then
19:30
Ejaaz: he later kind of quote tweets that tweet and says answer the model is complete
19:35
Ejaaz: junk it's a hallucination machine overfit to reasoning benchmarks and has absolutely
19:40
Ejaaz: zero recall ability so a few things he's mentioning here is one it hallucinates
19:45
Ejaaz: a lot so it kind of makes up jargon terms,
19:48
Ejaaz: ideas, or parameters that didn't really exist before.
19:51
Ejaaz: Number two, he's saying that OpenAI designed this model purely so that it will
19:55
Ejaaz: do well on the exams, which are the benchmarks that rate how these models compare to each other.
20:01
Ejaaz: So they're saying that OpenAI optimized the model to kind of like do really
20:06
Ejaaz: well at those tests, but actually fail at everything else, which is what people want to use it for.
20:10
Ejaaz: And the final point that he makes is that it has zero recall ability,
20:13
Ejaaz: which is something you mentioned earlier, Josh, which says it doesn't have memory
20:16
Ejaaz: or context so you can have a conversation and then open up another conversation
20:20
Ejaaz: and it's completely forgotten about the context that it has for you from that
20:23
Ejaaz: initial conversation okay.
20:25
Josh: So not not the best not to be unfair to open ai but it feels like they delayed
20:30
Josh: this model a good bit of times oh yeah and they wanted it to look good and it
20:34
Josh: intuitively makes sense to me that they would be kind of optimizing for benchmarks
20:38
Josh: with this one um but nonetheless it's still impressive i'm seeing this big wall
20:41
Josh: of text now what is what is this what is this post here
20:44
Ejaaz: Well it's this post from uh one of these accounts i follow and they have an
20:48
Ejaaz: interesting section here which says comparison to other open weights oh sick.
20:52
Josh: Yeah what is this
20:53
Ejaaz: So he goes while the larger gpt oss
20:56
Ejaaz: 120 billion parameter model does not come
20:59
Ejaaz: in above deep seek r1 so he's saying that deep seek r1
21:02
Ejaaz: just beats it out the park it is notable that
21:05
Ejaaz: it is significantly smaller in both total and active
21:08
Ejaaz: parameters than both of those models deep seek
21:11
Ejaaz: r1 has 671 billion total parameters and
21:14
Ejaaz: 37 billion active parameters and is released natively right which makes it 10x
21:20
Ejaaz: larger than gpt's 120 billion parameter models but what he's saying is even
21:24
Ejaaz: though gpt's model is smaller and doesn't perform as well as deep seek it's
21:29
Ejaaz: still mightily impressive for its size.
21:32
Josh: Okay that's cool because that gets back to the point we made earlier in the
21:35
Josh: show that this is probably the most densely condensed
21:41
Josh: however you want to say it like base of
21:43
Josh: knowledge in the world they've used a lot of efficiency gains
21:46
Josh: to squeeze the most out of it so in this small model
21:49
Josh: it is i guess if we're optimizing maybe we
21:52
Josh: can make up a metric here on the show which is like um output per
21:56
Josh: per parameter or something like that like based on the total parameter
21:59
Josh: count of this model it gives you the best value per
22:03
Josh: token and that seems to be where this falls
22:06
Josh: in line where it's not going to blow any other open source model out of the
22:08
Josh: water but in terms of its size the fact that we can
22:11
Josh: take a phone and literally run one of these models on a phone and
22:14
Josh: you could go anywhere in the world with no service and have access to these models running
22:17
Josh: on a laptop or whatever mobile device that that's super
22:20
Josh: powerful and that's not something that is easy to do with the other open source
22:24
Josh: models so perhaps that's the advantage that open ai has it's just the density
22:28
Josh: of intelligence and the efficiency of these parameters that they've given to
22:32
Josh: us versus just being this like home run open source model that is going for the frontier,
22:37
Josh: it's just a little bit of a different approach.
22:40
Ejaaz: Yeah, we need like a small but mighty ranking on this show, Josh,
22:44
Ejaaz: that we can kind of like run every week when these companies release a new model.
22:48
Ejaaz: No, but it got me thinking, if we zoomed out of that question,
22:52
Ejaaz: right, because we're talking about small models versus large models,
22:56
Ejaaz: parameters and how effectively they use versus other models that are bigger.
23:01
Ejaaz: What really matters in this, Josh? In my opinion, it's user experience and how
23:06
Ejaaz: useful these models are to my daily life, right?
23:09
Ejaaz: At the end of the day, I kind of don't really care what size that model is unless
23:14
Ejaaz: it's useful for me, right? It could be small, it could be personal, it could be private.
23:17
Ejaaz: It depends on, I guess, the use case at the time. And I have a feeling that
23:22
Ejaaz: the trend of how technology typically goes, you kind of want a really high-performant
23:29
Ejaaz: small model, eventually.
23:31
Ejaaz: Right? I try and think about like us using computers for the first time,
23:35
Ejaaz: you know, back in our dinosaur age.
23:37
Ejaaz: And then, you know, it all being condensed on a tiny metal slab that we now
23:42
Ejaaz: use every day. And we can pretty much work from remotely from wherever.
23:45
Ejaaz: And I feel like this is where models are going to go. They're going to become
23:47
Ejaaz: more private. They're going to become more personal.
23:49
Ejaaz: Maybe it'll be a combination of, you know, it running locally on your device
23:53
Ejaaz: versus cloud inference and trusting certain providers.
23:57
Ejaaz: I don't know how it's going to fall out, but I think Like it's not a zero to
24:01
Ejaaz: one. It's not a black or white situation.
24:03
Ejaaz: I don't think everyone's just going to go with large centralized models that
24:06
Ejaaz: they can inference from the cloud.
24:07
Ejaaz: I think it'll be a mixture of both. And how that materializes,
24:09
Ejaaz: I don't know, but it's an interesting one to ponder.
24:12
Josh: Yeah, I think this is funny. This is going to sound very ironic,
24:15
Josh: but Apple was the person that got this most right.
24:19
Ejaaz: Sorry, who's Apple again?
24:21
Josh: Yeah, right. I mean, it sounds ridiculous to say this. And granted,
24:24
Josh: they did not execute on this at all.
24:25
Josh: But in theory, I think they nailed the approach initially,
24:28
Josh: which was you run local compute where all of
24:31
Josh: your stuff is so my iphone is the device i never
24:34
Josh: leave without it is everything about me it is all of my messages my
24:37
Josh: contacts all the contacts you could ever want from me and then the idea was
24:41
Josh: they would give you a local model that is integrated and embedded into that
24:44
Josh: operating system and then if there's anything that requires more compute well
24:47
Josh: then they'll send the query off into the cloud but most of it will get done
24:50
Josh: on your local device because most of it isn't that complicated and i think as
24:54
Josh: a user when i ask myself what i want from AI.
24:57
Josh: Well, I just want it to be my ultimate assistant. I just want it to be there
25:00
Josh: to make my life better. And so much of that is the context.
25:03
Josh: And Apple going with that model would have been incredible.
25:08
Josh: It would have been so great. It would have had the lightweight model that runs
25:10
Josh: locally, it has all the context of your life, and then it offloads to the cloud.
25:13
Josh: I still think this model is probably the correct one for optimizing the user
25:17
Josh: experience. But unfortunately, Apple just has not done that.
25:20
Josh: So it's up for grabs. I mean, again, Sam Altman's been posting a lot this week,
25:24
Josh: we do have to tease what's coming because this is probably going to be a huge
25:27
Josh: week. There's a high probability we get GPT-5.
25:30
Josh: And then they've also been talking about their hardware device a little bit. And they're saying how
25:34
Josh: It's like it's genuinely going to change the world. And I believe the reason
25:37
Josh: why is because they're taking this Apple approach where they're building the
25:39
Josh: operating system, they're gathering the context, and then they're just they're
25:42
Josh: able to serve it now locally on device.
25:44
Josh: They're able to go to the cloud when they need more compute.
25:47
Josh: And it's going to create this really cool, I think, duality of AI where you
25:51
Josh: have your your super private local one, and then you have the big brain one,
25:55
Josh: the big brother that's off in the cloud that does all the hard computing for you.
25:58
Ejaaz: Well, one thing is clear. There are going to be hundreds of models and it's
26:01
Ejaaz: going to benefit the user, you and I, for so many multiple...
26:05
Ejaaz: It's the big company's problems to figure out how these models work together
26:09
Ejaaz: and which ones get queried. I don't care.
26:12
Ejaaz: Just give me the good stuff and I'm going to be happy.
26:14
Ejaaz: Folks, OpenAI has been cooking. This was the first open source models they've
26:20
Ejaaz: released in six years, Josh.
26:22
Ejaaz: The last one was 2019 GPT-2, which seems like the stone age and it was only like four years ago.
26:29
Ejaaz: Thank you so much for listening. We are pumped to be talking about GPT-5,
26:34
Ejaaz: which we hope to be released in maybe 24 hours.
26:38
Ejaaz: Hopefully this week, fingers crossed. I don't know, we might be back on this
26:40
Ejaaz: camera pretty soon. Stay tuned.
26:42
Ejaaz: Please like, subscribe, and watch out for all the updates. We're going to release
26:45
Ejaaz: a bunch of clips as well if you want to kind of like get to the juicy bits as well.
26:49
Ejaaz: Share this with your friends and give us feedback. If you want to hear about
26:52
Ejaaz: different things, things that we haven't covered yet or things that we've spoken
26:55
Ejaaz: about, but you want to get more clarity on or guests that you want to join the show, let us know.
26:59
Ejaaz: We're going full force on this and we'll see you on the next one.
27:02
Josh: Sounds good. See you guys soon. Peace.
27:04
Music: Music