Latest / Newcomer Pod / Fear-Mongering & Forecasting: Assessing AI's Predictions About AI
Transcript
- 0:01Welcome to the Cerebral Valley Podcast.
- 0:03I'm Eric Newcomer. We're calling this fear
- 0:06mongering and forecasting. We're looking to the future.
- 0:09We're making predictions, or really I think we're going to be
- 0:12relying on AI and chat bots to make our predictions and then we
- 0:17will shit all over them. I'm here with Max Child.
- 0:20Hey, Eric. And James Wilserman.
- 0:22Hey, hey guys. Ready to make some hot take
- 0:26predictions here? We're gonna use both Open AI,
- 0:29Chaji, BT and Anthropic, so we'll also get sort of the meta
- 0:34sense of who's which bot is making better predictions.
- 0:38But yeah, excited what? What is AI if not constantly
- 0:42looking to the future and charting out what could be given
- 0:46the rate of growth we've seen so far?
- 0:48So it's very fitting that that in our last episode before this
- 0:52Ruebel Valley AI Summit on November 20th in San Francisco,
- 0:56we are going to fiendishly fantasize about what could be to
- 1:02come. This episode is presented by
- 1:05Breck's The Financial Stack that founders and VCs can truly bank
- 1:09on. Imagine what your founders could
- 1:11do with their runway if they had a banking solution that had no
- 1:14minimums, no transaction fees, and 20 times the standard FDIC
- 1:19protection. Plus, they could earn an
- 1:21industry leading yield while maintaining access to funds
- 1:25whenever needed. Brex simplifies financial
- 1:27services for startups so they can focus on building.
- 1:31Connect your portfolio to the financial stack that one in
- 1:33three US venture backed startups already use.
- 1:37Check out brex.com/banking Dash solutions.
- 1:43Without further ado, James. So for this episode I asked both
- 1:47ChatGPT and Claude the same question to come up with a list
- 1:53of predictions about AI for the year 2025.
- 1:56I also asked the predictions to be hyper specific and
- 2:00falsifiable so that by the end of 2025 we could easily
- 2:04determine whether the predictions were correct or
- 2:06incorrect. The predictions I said could be
- 2:08spicy, medium, mild. However, however hot Thai spicy.
- 2:14I didn't get any pepper emojis out on the on the output.
- 2:17Maybe I should ask for that. But the, the, the predictions I
- 2:22was, I was pretty happy with how they came out and I asked, you
- 2:26know, relevant to the AI industry and then I also asked
- 2:30to get a probability estimate for each prediction.
- 2:32So then you guys are going to be able to provide me kind of the
- 2:36over undertake on each prediction.
- 2:38So we have our own prediction market basically.
- 2:41Exactly. We basically created a bunch of
- 2:43possible prediction markets around AI and you guys have to
- 2:48decide whether you're short or long each prediction.
- 2:52Nice, how's that? I love it.
- 2:53Is this is this GPT 01 or is this?
- 2:56Yeah, this is the ChatGPT 01 preview and Claude Sonnet 3.5.
- 3:04OK, so the the most thinking oriented AI models.
- 3:07We're paying the big bucks here. We're premium subscribers for.
- 3:12We're we're bankrupt now they're running, but no, these these are
- 3:19the best AI can do in a consumer consumer product.
- 3:22It it feels if I was pretty impressed, like I was kind of
- 3:25skeptical actually, that this would work that well.
- 3:27You know, I think AI has a tendency to provide sort of
- 3:32generic takes in a lot of ways, but if you prompt it correctly,
- 3:37I was able to get these really specific, really specific
- 3:40predictions that. They could keep this is for 12
- 3:43months from now. Like what's the date?
- 3:45You have the year 25. Yeah, by the end of 2025,
- 3:48December 31st. OK, cool.
- 3:51All right, let's do it. All right, we'll start with O1
- 3:55preview from Open AI in Chacha BT.
- 3:58So here's the first prediction So.
- 4:05We have not heard these before. To be clear, Max and I are
- 4:08coming in blind. Unless.
- 4:10You guys, you guys are Co founders so maybe you're in
- 4:12cahoots. Nope, I got nothing running
- 4:14blind. All right, so first prediction,
- 4:17Open AI will release ChatGPT or GBT 5 exceeding 10 trillion
- 4:24parameters. What do you guys think?
- 4:28How many parameters is 4? Again, one trillion.
- 4:31The bots are already smarter than me.
- 4:32What is that? You're like what's a parameter
- 4:34isn't. Isn't it like 400 billion?
- 4:36Can you can we get the number of the current it's like 4 or 500
- 4:40billion, right? 1.5 trillion.
- 4:42OK, let's add the. Elevator .76 trillion Is the
- 4:45rumours OK? 1.
- 4:47One to two, OK. So this would be, you know.
- 4:50What's it predicting to rate how many?
- 4:51A 10X like like A7X basically or whatever.
- 4:55And we think the current is 1.7 something.
- 4:57Yeah. Also the the probability for
- 5:00this prediction is 50% so. Oh, over smash the over on that.
- 5:05Wait, wait, wait, wait. Sorry, it's is it making a claim
- 5:08about the name of it? Because I feel like I, I think I
- 5:11was just listening to Dario on Lex Friedman and like, I feel
- 5:15like half of his, like Hemming and hawing about what they're
- 5:18gonna do is like not committing to the name of various models
- 5:22because you know, they're doing all these like half updates and
- 5:24stuff. So how much is this a prediction
- 5:26about what the model is going to be called?
- 5:28I think we can decide there, but I would say both have to come
- 5:33true. The GPT 5 naming and the 10
- 5:36trillion. I think, you know, if I'm
- 5:38reading the tea leaves of what open what 01 was thinking and
- 5:41reasoning about here, maybe they're saying, hey, 10 trillion
- 5:44is a big step up to the point where it would have to almost be
- 5:47called, you know, GBT. Five, I mean, I don't know, this
- 5:50is kind of pedantic, but I would smash the over on a model that
- 5:54it has 10 trillion parameters, but I don't know if they're
- 5:56gonna call GBT 5 exactly. I feel like Altman has been you.
- 6:00Committed to over already, I think the.
- 6:02Name. OK, fine, I'll commit.
- 6:03Why are you guys so skeptical that it's gonna be called GBT 5?
- 6:06I'm just chaotic on the names. The names are bad at the moment.
- 6:10Like a circle that we get you introduced as like open AI chat
- 6:15GBTO one like they they need to like I I do I mean anthropic has
- 6:19more colorful names, but they're getting chaotic sure, but so I'm
- 6:23just take the under on the basis of the name in terms of the
- 6:26parameters yeah, I'm probably bullish.
- 6:32I mean, thankfully it's a compound prediction, but yeah,
- 6:36the under. OK, All right, Well, I will
- 6:37stick with my over, but yeah, I mean, I think they're basically
- 6:40running into like the iPhone 4S5 S 6S problem where Apple used to
- 6:45give the like in between phones, like, you know, an S instead of
- 6:48a new number. And then I think the marketing
- 6:50team realized like people want to see the number go up.
- 6:53Like we don't see, we don't we don't buy new iPhones unless the
- 6:56number goes from 14 to 15. So I think, I think This is why
- 7:00GBT is trying to move off these numbers because I think they're
- 7:02like, nobody wants the new model unless we give it a higher
- 7:05number, right? And they're like, they don't
- 7:06even care if we introduce like 4 O and voice mode and you know,
- 7:10O1 and all this stuff. So I just, I feel like Altman is
- 7:12severely foreshadowing that they're gonna drop these
- 7:15numbers. I do think you're predicting.
- 7:17Yourself. No, I'm taking the over. 50% is
- 7:20a good price. I just think I understand where
- 7:23they're coming from with the names.
- 7:24James, are you making predictions?
- 7:26You'll give us a quick your quick take.
- 7:29Yeah, I'll just react to your guys takes.
- 7:31I think that yeah, you guys are way too focused on the naming
- 7:34conventions. I think the reason they are
- 7:38calling things like, you know, 40 and 01 preview is to try to
- 7:43like, you know, downplay the model a little bit before the
- 7:47next evolution of the model. Like I think they will stick
- 7:50with the GPT 456 naming convention, but they will need
- 7:53it to be kind of step change, you know, improvement on the
- 7:58model. So I do think A10 trillion
- 8:00parameter model will get that name.
- 8:01But there's an open question. I mean, this is gonna be
- 8:03probably coming up throughout, just like how monumental is A10
- 8:07trillion parameter model change, right?
- 8:10Like if it's true if they have consumed basically all the data
- 8:14there is to consume and set that data has its limits.
- 8:19That's fair. Then if there's not a step
- 8:20change in the model capabilities, maybe they don't
- 8:23give the new name. OK, all right, let's move on.
- 8:25Good, good first discussion there.
- 8:27Are you over or under though? I never got that part.
- 8:30I'm over, OK? I mean if you want me to make
- 8:33make my own predictions. You've got to come in.
- 8:34I just. Say I just put.
- 8:35I'll say where you are. I'll take the over.
- 8:37All right, so next prediction from Chachi BT we have by the
- 8:45end of 2025, at least three countries will have implemented
- 8:50regulations at the national level specifically governing the
- 8:55use of AI to diagnose healthcare disease.
- 9:00And what's the price on that? 70% probability.
- 9:05Short. Way short.
- 9:09I just think it's really, it takes a long time to write
- 9:11legislation. I mean, you're saying like
- 9:13national level entities are going to write complex
- 9:16legislation about healthcare and AI in the next 14 months.
- 9:20Yeah, yeah, it's. Interesting, though I'll grab.
- 9:22Go ahead. I'll let Eric answer, but you're
- 9:24just I'm. Taking the over, I mean, I don't
- 9:27want to just do the opposite of Max every time here, but I do
- 9:31That's good. There are a lot of countries.
- 9:32There are a lot of countries we've actually seen aggressive
- 9:36AI regulation already. There's clearly a lot of
- 9:39appetite. And I don't think this
- 9:40prediction requires some sort of comprehensive, they just need
- 9:43some regulation touching sort of health, health diagnostics,
- 9:48right? Yeah, I, I guess.
- 9:50I guess the question is like, is some random EU regulator issuing
- 9:54a decree count as 35? Countries.
- 9:56Yeah, exactly. I think it should like I think
- 9:58it. Should.
- 9:59Well, yeah, of course you think it should.
- 10:01Like, yeah, but I'm just saying like, you know, I'm this.
- 10:04Episode isn't gonna be about the nature of predictions more than
- 10:07this underlying. Game.
- 10:11You know, for every good game, we've realized the rules are
- 10:15really what's most important to discuss, not the actual OK the
- 10:19gameplay. All right, well, I'm still I'm
- 10:21still grabbing the under here. But yeah, that just seems like a
- 10:25lot of work for three different nation states to execute and
- 10:27then in the 12 month time frame would be mine.
- 10:29Are we getting more point like or so like does he get more
- 10:33return on his investment for predicting something that was at
- 10:3570? Yeah, that's a good idea.
- 10:37I think that we could use something like that.
- 10:39If you take the over on a low probability prediction and then
- 10:43at the end of next year you're proven correct, I think we'll
- 10:46give you give you some more points.
- 10:48I think we need. ROI like let's let's let's save
- 10:51the exact one. Right, exactly.
- 10:54Could we get into another rule here?
- 10:55Yeah. James, what are you picking?
- 11:01OK, I am taking the over there and I think, you know, we're
- 11:05just going to see this come up a lot in the next year is like
- 11:08people using AI for their jobs, maybe in areas where they're not
- 11:11supposed to. And I just think that'll be a
- 11:13key kind of political discussion.
- 11:15Point is, hey, why is why is my doctor, you know, using ChatGPT,
- 11:20right? And is that a should that be
- 11:22legal? So I wouldn't be surprised by
- 11:23three countries. Taking I want them to be to be
- 11:25clear or like as an assistant. I think it would be good.
- 11:28I'm not saying that's a bad thing, but maybe it should be a
- 11:30little bit regulated, right? There should be something.
- 11:33I might just, I might just be biased based on living in the
- 11:35United States where we have just epic legislative gridlock.
- 11:38But yeah, I mean they. Passed AI regulations in
- 11:42California. Well, that's true.
- 11:45I mean, you're. Not the big one.
- 11:46But you're saying the state, the state government, Yeah.
- 11:48Yeah, which is like one of. The.
- 11:50Well, they didn't pass it because the.
- 11:51They passed other ones, they passed it all there other.
- 11:54Things OK, this prediction specifically says national,
- 11:57national. Yes, wouldn't count.
- 11:59Yeah, all right. But Europe?
- 12:01'S like anyway, sorry. Moving on prediction, by
- 12:05December 2025, Tesla will release fully self driving cars
- 12:11that are legally approved for unsupervised operation on a on
- 12:16public roads in at least one US state.
- 12:21What's the price? We are saying 40% probabilities.
- 12:28Interesting. I'll go over actually I one of
- 12:33my predictions coming in was around Tesla AV capabilities.
- 12:38I just think that Waymo's proved it's possible.
- 12:42Elon's got more chips than anyone.
- 12:43They've got more video data than anyone.
- 12:45They're obsessed with this problem.
- 12:47He thinks it's the entire future of the company.
- 12:48And then coming down to the ticky tacky state approval bit,
- 12:54you know, he's got his boy in the White House and he's got
- 12:56friends in every Republican government in America.
- 12:59So like fucking Wyoming could approve this shit and he win the
- 13:02bat. So I think that I'll take the
- 13:05over at 40%. So it it's to be driving like
- 13:10without a driver behind the wheel.
- 13:13Yeah, that's I think unsupervised, meaning you don't.
- 13:16Have to I still, I still think there's a very likely scenario
- 13:18where it's like in Texas or Wyoming or whatever you, you
- 13:24know, you can sort of allow it to drive, but people are
- 13:27effectively behind the wheel. Like I'm not very bullish on
- 13:31Tesla's like technology, I don't think.
- 13:33Are you talking about like remote assistance, like remote
- 13:35drivers taking over or something or like I think?
- 13:39That would be that would count or not I feel like.
- 13:42That would count like that's what Waymo does.
- 13:43Right, Waymo has remote takeover if there's.
- 13:46No takeover. Remote takeover, yeah.
- 13:48But it's a little. It's like a remote.
- 13:51How is what they have right now not already passing that?
- 13:53Like people are allowed to use autopilot.
- 13:56You're talking about Tesla. Tesla you.
- 13:58From a legal perspective, you have to have a driver in the car
- 14:03behind the wheel, ready to take over at a moment's notice.
- 14:05Basically it also has to be supervised, like you know that
- 14:08you have to be looking at the road or disengage.
- 14:11Like they have a camera on your eyes that show that checks if
- 14:15you're watching the road or not and it gets mad at you if you're
- 14:17like on your phone. I've tried this.
- 14:21Oh, wow, yeah, yeah, it it literally throws a warning up on
- 14:25the screen saying I like, I can tell you're looking at your
- 14:27phone. And I'm like, well, what's the
- 14:29point of self driving if I can't look at my phone?
- 14:32Like that's the entire use case. I think I'm I'm taking the over
- 14:39just because of all this sort of seeming cases where this would
- 14:43count as approved. I don't think they're close to
- 14:46Waymo. I'm not bullish on Tesla's
- 14:49approach, but I think there are lots of ways to sort of fudge
- 14:53this with humans still sort of behind the wheel.
- 14:57All right, for bonus points, which is the 1st state.
- 15:00That will allow this Texas. Texas would be the obvious
- 15:04choice, but it's also really big and there's a lot of complicated
- 15:07roads and cities and stuff there, so I don't.
- 15:10Know, but does it have to be the whole state for the rules or?
- 15:13Yeah, Yeah. It's a state government, right?
- 15:15Yeah. Wait the whole state.
- 15:17It says one state. Will approve it.
- 15:19That's interesting. You mean like Waymo is not in
- 15:21the whole state, it's in San Francisco, right?
- 15:23Or I think. I think it would just have to be
- 15:26not the whole state, but you know, somewhere within somewhere
- 15:29in the state. Yeah, like the whole state.
- 15:31I would change my whole. Yeah, yeah.
- 15:34I think it has certain. Things I guess, like I wasn't
- 15:36even thinking you'd be able to own a Tesla but only use it for
- 15:40full self driving if you're in one city.
- 15:43Yeah, that's interesting. All right, Texas.
- 15:46All right, We'll see how that plays out.
- 15:49James. Yeah, my take taking the over as
- 15:53well. I think you guys are, you guys
- 15:54are spot on. And yeah, I think I will
- 15:58actually go a little contrarian and take California as the first
- 16:02thing. Wow, OK, wow all.
- 16:04Right guys, let's move on to the next prediction we have and
- 16:08another prediction from ChatGPT. By the end of 2025, AI generated
- 16:14content will constitute at least 50% of online news articles
- 16:20published by a major news outlet, probability 30%.
- 16:27What's major? Again, I think this is a key
- 16:28question here. Like how major?
- 16:30Mitigated after the fact. What would be capital M major?
- 16:38I mean like I'm. Watching the I'm taking the
- 16:40over, I think Conde Nast is clearly major.
- 16:44I think Forbes is probably major.
- 16:46And I think if you're just talking articles, you know,
- 16:50we've seen a lot of media just be willing to produce a lot of
- 16:53articles to, you know, get get gobbled up by SEO.
- 16:58And so it's not necessarily the majority of their homepage, but
- 17:02the majority of the articles. One outlet.
- 17:05I say yes. I think, yeah, I think it's 30.
- 17:08I'd grab the over. I just think 3 to one odds on
- 17:11that or two to one odds sounds pretty good.
- 17:13I think to your point, like yeah, the SB Nation network I
- 17:17could see or Vox or whatever, like just somebody is gonna give
- 17:21in to the SEO, the SEO temptation to have a crap load
- 17:26of articles generated by AI that people are going to link to.
- 17:29So will. They talk about that publicly.
- 17:31Do you think like will they? No, I'm sure there will be some
- 17:33journalist that gets leaked to in a bar where where Eric lives
- 17:36in Brooklyn and then then somebody will write an.
- 17:39Angry story about I don't think Axios would necessarily do it,
- 17:42but there's definitely a breed of sort of business first media
- 17:45publication that would brag about this to seem cutting edge.
- 17:50I bet someone will write a cranky article before somebody
- 17:53brags brags about it. But before.
- 17:55Yeah, yeah. We'll see.
- 17:57AI itself break the news that. Not until it hooks up to the
- 18:05Bitcoin network and starts getting paid for it.
- 18:09I'll take the under on 30% I'm sure have to litigate what
- 18:15content a major outlet, but I'm skeptical that it'll hit 50% of
- 18:21articles like I mean, at least in the next year.
- 18:25I think that I think that it'll maybe start, you know, kind of
- 18:27long form like some some indie Blogger or something.
- 18:30We'll be generating hundreds of thousands of articles or
- 18:32something with AI next year, but I'm skeptical that a major
- 18:35outlet will be doing that at that scale next year, but we'll
- 18:39see. I just like.
- 18:40Media outlets. I just go with the, yeah, the
- 18:43Logan Roy attitude here, which is that money wins.
- 18:46And there's a lot of private equity firms that own these now
- 18:49and. If they're not doing this,
- 18:51they'll they'll yeah, management will get fired.
- 18:53This is an amazing way to cost cut and it's also an amazing way
- 18:55to get links from Google search. So I just think it's going to be
- 18:58very tempting for a lot of these companies.
- 19:02All right, next prediction. By December 31st, 2025, the
- 19:08first AI generated feature length film will be released in
- 19:13theaters, with the script and visuals both entirely crafted by
- 19:18AI. 25% probability under. I'm crushing the under on that.
- 19:25I just think that you have to do a deal with theaters like get
- 19:29distribution and Hollywood's going to lose their effing minds
- 19:32if somebody tries to do this in the next year.
- 19:34So I just think the the Gears of entertainment don't turn that
- 19:38quickly. I'm not saying somebody won't
- 19:40generate a feature film by the end of 25 that plausibly could
- 19:43be in a theater, although I I would actually debate that as
- 19:45well. But I think that the odds that
- 19:47it's like actually released in theaters.
- 19:48I'm I'm just super under on that.
- 19:51They barely, they barely let Netflix into the Oscars.
- 19:54And your mistake is misunderstanding the
- 19:57entertainment industry. Even though you're you grew up,
- 20:01you grew up in LA area, but. The you can pay to play in
- 20:06theaters any day. Like there are all these sort of
- 20:09documents, there are all these films that sort of pay to get
- 20:12theatre distribution so that they account for stuff and you
- 20:15can absolutely do that. I'm taking the over.
- 20:17There are tons, tons of companies with an interest to
- 20:20have this achieved and somebody will just buy the theatre plays
- 20:24to make it so you. Have the like, buy the US
- 20:26Constitution angle here, which is somebody's just gonna fucking
- 20:29do it because it's gonna sound cool basically, and pay to play
- 20:32the theaters. OK?
- 20:33I mean, there's already, there's already a.
- 20:35Film that Sarah, my wife saw that you know, there are some
- 20:40boundaries like it was shot by humans, but then it reorders
- 20:44based on AI, you know, so that was already in theaters.
- 20:47So I think. This is a good, this is a good
- 20:49robust debate. I like that.
- 20:50You're like, you guys believe in the forces of change and I
- 20:53believe in the entrenched hating institutions.
- 20:56I haven't taken my. I haven't taken my.
- 20:59Well, I was referencing the the legislation discussion from
- 21:02earlier, but yeah, this is a meta point about how quickly the
- 21:04Gears of power turn. And you guys are like really
- 21:07quick. And I'm like, I don't know.
- 21:09So, all right, James, what do you think?
- 21:13I am also taking the under on this one.
- 21:15Yes, and not for any remote reason around theater
- 21:19distribution, but just purely from model capability.
- 21:22Like the specification of this prediction is that the script
- 21:26and visuals have to entirely be created by AII think that we are
- 21:30not there on visuals, we're barely there on scripting.
- 21:34I don't think we're there on scripting either, by the way.
- 21:36I haven't. Seen the script as you started
- 21:39the episode with is that there's a lot of control in models over
- 21:42prompting. And so it's like if they prompt,
- 21:45they could prompt a whole script into it and then output a script
- 21:49and say it was AI. And that I just think there's
- 21:52such a strong incentive for the result to be it was AI that like
- 21:56humans can put a lot of thought into it and get what they want.
- 21:59I don't think it'll be hard to do a script.
- 22:01I mean, it won't be a good script, but I do think it'll be
- 22:04hard to create a full feature length.
- 22:07Does it say it says feature length?
- 22:08I don't know that Probably 120 minutes or something of visuals
- 22:12like a that's extremely expensive to create, and also B,
- 22:17it's like very hard to get any sort of consistency of the
- 22:20characters or anything. I'm just skeptical of the
- 22:22visual. Aspect.
- 22:23I think our guys at Runway will personally make sure this
- 22:26happens. Yeah, probably the guys.
- 22:28It's probably the way Eric wins is at the runway.
- 22:31Our friends at Runway just bribe.
- 22:33Have a theater chain into doing right?
- 22:34Yeah. They listen to this episode,
- 22:36they listen to this episode and they're like.
- 22:39They're like Eric's got some skin in the game and we're gonna
- 22:42help him out here. We do this, Eric is gonna brag
- 22:45about us on the next. Year's yeah.
- 22:48I Episodes I. Don't know they haven't even
- 22:50released Sora you know yet or. It's coming soon.
- 22:54Somebody was just one of the competitors.
- 22:55I think Runway maybe was saying we think Sora is about to come,
- 22:58come all right. All right, good discussion.
- 23:00Let's switch over to Claude and see how anthropics predictions
- 23:05do and how you guys respond to the probabilities.
- 23:09And then we can have a little discussion at the end of who's
- 23:12better at predicting, but meta discussion but starting with.
- 23:17We're already having a discussion about our opinions on
- 23:20their predictions, and then we're going to have a higher
- 23:22level of discussion about our opinions of the relative
- 23:25predictive power of the two models.
- 23:27And then we're going to have AI watch this represent and react
- 23:31to our predictions. It's all good.
- 23:33It's all good. Well, I guess on a certain
- 23:35level, you know, think about AI is all about intelligence.
- 23:38So it's like, how do we introspect our introspection
- 23:40even more anyway? Sure.
- 23:42Yeah, somebody. Someday we'll be teaching a
- 23:46class in university. About this episode.
- 23:48All. Right, too self indulgent.
- 23:51All right, all right. Starting with Claude's first
- 23:55prediction, by December 31st, 2025, at least three major
- 24:00smartphone manufacturers among Apple, Samsung, Google and
- 24:05Xiaomi will release phones with dedicated AI Co processors
- 24:10capable of running large language models with at least 7
- 24:14billion parameters entirely on device probability 75%.
- 24:21And the parameters ones are where I'm like.
- 24:23I don't even know how many parameters they can run on
- 24:25device right now. Can't they run 7 billion locally
- 24:27already? Is that wrong?
- 24:30I think that the Apple intelligence chip can run 7
- 24:33billion, right? I thought.
- 24:34It's at least three to four, you know. 3 billion. 3 billion.
- 24:38Yeah. OK, three to four.
- 24:38All right, I got it. I nailed.
- 24:40It Apple's current. Just to reiterate there, Apple's
- 24:44current on device LLM uses 100% local LLM with three billion
- 24:49parameters and then can also connect to external servers for
- 24:54larger parameter models. And what's the price on this?
- 24:58Over 75% or probability 75%? OK, well I'm gonna grab the
- 25:05under on that cuz I think dedicated Co processor is really
- 25:10what's holding me back here cuz that requires like a separate
- 25:13chip essentially to go into the hardware.
- 25:16I'm a avid follower of Apple rumors and I have not gotten any
- 25:19sense that they're interested in adding chips like separate chips
- 25:22to their system to do this kind of thing.
- 25:24And so you'd basically have to get Xiaomi, Google and Samsung
- 25:27to all do this separately chip in the next year.
- 25:30And there's not really clear sentiment that that's necessary
- 25:34to run these models. So yeah, I'll take, I'll take
- 25:38the under at 75 for sure. You sold me under.
- 25:44Hardware Hardware timelines are really long, and it that that's
- 25:47only a slightly larger model than they can already run
- 25:49locally, so it's just not clear. To me, the parameters make
- 25:52sense, I And it's also not all these come on device for Google.
- 25:58Like, is that really in their mission?
- 26:00Like there's so much like, we'll send everything to the cloud.
- 26:02Like, are they really? Yeah, Apple's the one selling
- 26:05like everything will happen on device.
- 26:07I think I'd like grab the over at like 10% on this, but that's
- 26:1175 I think under big time. Yeah.
- 26:16I mean, I guess I'm hearing what you're saying on the dedicated
- 26:20aspect, but like I feel like you could quibble with that
- 26:24semantically of like as long as it's a GPU at all that is
- 26:30dedicated to running AI tasks. Like that's what the.
- 26:34GPU. They already have a GPU, right?
- 26:36I mean, yeah. OK, so they have essentially a
- 26:39GPU and then there's a set of cores, right?
- 26:48I guess it's. Just what is the neural engine I
- 26:49guess is what? I'm yeah, it's like are we
- 26:51counting the current neural engine on there for like 8 years
- 26:54already as as a yes to this because.
- 26:56Then it's like. Sure.
- 26:58Like I guess I mean like, but I don't think that's the spirit of
- 27:01the prediction, right, OK. Like the the thing they've had
- 27:05on the phones for the last eight years.
- 27:06Is I don't. Know.
- 27:08So far, Cha Chi BT is beating Anthropic because I feel like
- 27:10there's. A quality of prediction.
- 27:12I don't love this. This.
- 27:13Whatever, Yeah, Anyway. James, Just pick, pick and we'll
- 27:16litigate everything afterwards. Yeah, I'll take the over and
- 27:20I'll try to litigate my take that the neural ending.
- 27:23All right, Great, great. All right.
- 27:25Prediction #2 Anthropic will release a model that achieves a
- 27:30score above 90% on the Unified Benchmark for AI Reasoning, or
- 27:35Bar Evaluation Suite, which will become the first model to
- 27:38outperform the human baseline. Human expert baseline of 87%,
- 27:44probability 65%. Where are they at right now?
- 27:48Unified. What was the the metric?
- 27:50It was unified. Benchmark for unified, benchmark
- 27:53for AI reasoning. We're all going to frantically
- 27:57Google for that, yeah? It's funny that we have
- 28:00anthropic predicting its own intelligence.
- 28:03Capabilities I know. I mean, a key part of this is
- 28:07the claim that it's the first also, right?
- 28:12It says first, right? Yeah, it'll be the first model.
- 28:16For this, for the record, is this benchmark real?
- 28:19I'm I can't find this benchmark. Is this is this benchmark?
- 28:23Being my God, that would be hallucinated.
- 28:25I think it's my God. I can't find this.
- 28:28Unified benchmark for AI reasoning.
- 28:34It's like I was like God. I was like, I've never heard of
- 28:37this benchmark like. I love it.
- 28:40It's crazy. It's hallucinate a benchmark.
- 28:44I mean, can you find it? Like it says something about our
- 28:50deference that at first I'd. Wear like I did not defer.
- 28:54I didn't. No, no, no, I assume.
- 28:58That it's. Do do not loot me into this.
- 29:00Definitely. I feel like I I immediately
- 29:03googled it and I was like, what I've never heard of this.
- 29:05I've heard of like MMLU. I've heard of like, you know, a
- 29:08couple other evals, but I've never heard of the bar like or
- 29:11whatever. Maybe this is like internal
- 29:14philanthropic. Maybe this is like like, we'll
- 29:18all be proven idiots when this becomes a benchmark, right?
- 29:22And then? They leaked this.
- 29:24Yeah, OK. I guess I will.
- 29:27I'll take the under given I'm not sure this benchmark.
- 29:29Is real I guess if it doesn't exist we can create it and you
- 29:33know y'all rule it and you can make it true.
- 29:38We'll have to do some more Googling.
- 29:39Later OK, but I don't think this benchmark exists for my own
- 29:43recent. Just very quick for the listener
- 29:47to get to the substance. I know we're focused on
- 29:49technicalities. Do you think someone will
- 29:53surpass sort of human intelligence overall by the end
- 29:58of next year? And do you think it will be
- 30:00anthropic first? Just quick 10.
- 30:02In some kind of evaluation, you're saying basically like,
- 30:05yeah, yeah, for sure. I, I sort of think we're like
- 30:09already there. But I guess it depends what
- 30:11invented benchmark. Exactly.
- 30:13That's the take I was going to get, which is we have sort of
- 30:15quietly passed humans on a lot of things already.
- 30:19And we're just this, you know, since people, you know, smartest
- 30:25in every domain still remains smarter than it.
- 30:27Like we've sort of under undersold how exciting it is for
- 30:31it to be smarter than the average human on on a lot of
- 30:34stuff. Totally.
- 30:36Next prediction from Claude. At least 5 Fortune 500 companies
- 30:40will replace more than 25% of their middle management
- 30:43positions with AI systems for task delegation and performance
- 30:48monitoring, publicly acknowledging this transition in
- 30:51their annual reports. Probability 25%. 25% chance that
- 30:5825% of middle management positions get axed.
- 31:03What if they become ICS? If they become individual
- 31:06contributors where they don't get fired, is this met?
- 31:09I think that this would be met because I think that we're
- 31:12saying past delegation and performance monitoring, meaning
- 31:16it's most most of the management responsibilities.
- 31:19I'll, I'll, I'll take that over. And my reasoning is essentially
- 31:23just that there's a lot of companies, there's in fact 500
- 31:26companies in the Fortune 500 and.
- 31:29A lot. There's some.
- 31:30Number. Yeah, yeah.
- 31:31There's a pretty high number, somewhere around 500 of them.
- 31:34And and plenty of those companies are not growing or
- 31:39shrinking and need to cut talent in some way or another to manage
- 31:45the business. And this will be the greatest
- 31:48freaking like cover stories. Yeah, exactly.
- 31:51Smoke and mirrors. Like, you know, look over here,
- 31:55like the left hand's doing some AI doing layoffs because they're
- 31:58right. Yes, right, right.
- 32:00This is going to be an amazing way to drop in your earnings
- 32:03report that like we had to layoff a bunch of people, you
- 32:06know, but it's cool because it's, it's about the future dog.
- 32:08We're adopting AI. So I think that, yeah, I just
- 32:12totally buy that, you know, 5 out of 500, we'll need to do
- 32:17that in the next year. And it's just an amazing cover
- 32:21story for doing layoffs. What's the percentage?
- 32:23Eric James. They need to replace more than
- 32:2825% of their management positions and the probability is
- 32:3225% that this occurs. Wow, so it's a much better
- 32:35return for me to say. Yes, yeah, you get a, you get a
- 32:384X return on the Yes. Yeah, over.
- 32:43I mean, yeah, I, I think the, the covers you guys, I'm, I'm
- 32:46not, I'm not as excited about AI in middle management
- 32:50necessarily, but I do think there are too many middle
- 32:54manager like we have this whole separate trend of like Facebook
- 32:57and companies saying let's just like be flatter organizations
- 33:01overall. And I feel like we're in that
- 33:02trend. And I do agree that the AI trend
- 33:05is sort of going to get mixed in with that, but I'm not convinced
- 33:10that AI is going to come for the middle managers first.
- 33:14I think it's going to supplement sort of the coders and
- 33:17frontline. You know, I to me, the
- 33:19deployment model is more like people are actually doing the
- 33:22work, figuring out how AI makes them better and sort of adding
- 33:26it to their just making themselves look better to their
- 33:29bosses. And I think that's how my model
- 33:32for how AI is getting deployed in workplaces, which is just
- 33:35sort of like people hacking shit together and delivering results.
- 33:40But I'm still taking the over. Yeah.
- 33:43I, I don't know, you kind of convinced me to take the under
- 33:45there. I think that I think that AI
- 33:50will have a much bigger impact on sort of IC roles.
- 33:54And you know, we've seen this already and some of the customer
- 33:57service oriented companies sort of announcing and bragging that
- 34:02they can hire less representatives.
- 34:05So I think we're more likely to see that first before companies
- 34:08start taking this AI approach on management, but could be wrong.
- 34:1425% is a big number at a Fortune 5.
- 34:19Exactly like are are they gonna? I I, I think I'm gonna I've my
- 34:24whole argument is the under. I feel like I should just like.
- 34:27No, you're not allowed to switch.
- 34:28You you really you? You made me not switch on like
- 34:312. On the fence, I could see the
- 34:33advantage you. Know you're on the fence, you
- 34:34know, No, I'm committed or I'm over.
- 34:37Yeah. Committed.
- 34:38It's funny that I made the whole case for the I.
- 34:40Know, I know. Thank you.
- 34:42Thank you for. Your all right, all right.
- 34:45Once you say the words, you're. Locked.
- 34:47Yeah, once you say the words, you're locked, Bear.
- 34:49It locked. All right, let's go to the next
- 34:53prediction from Claude. This prediction is that the deep
- 34:59mind or let's say Google broadly, will demonstrate an AI
- 35:02system capable of discovering at least one novel pharmaceutical
- 35:06compound that also passes phase one clinical trials, reducing
- 35:11that typical discovery to trial timeline by around 50%.
- 35:16We will just specifically say that it has to have discovered a
- 35:20compound and passed phase one trials within the next year,
- 35:23probability 20%. I'm gonna take the Max world
- 35:27view, which is like a lot of these are predictions about
- 35:30society. And I definitely don't think
- 35:33drug approvals are going to sort of just get much faster because
- 35:37AI companies say they should. So I'm going to take the under.
- 35:42Who's Running the FDAI guess is another.
- 35:44Yeah, exactly. Oh yeah, that's.
- 35:45Trump I'm only conflicted because the Trump administration
- 35:48is about to take over. But they don't like Google,
- 35:50Google or they're not going to like Fast Track Google.
- 35:52That's the last company. That's the company they want to
- 35:53hit the most. Yeah, they don't like the FD.
- 35:56Or sorry, Yeah, they don't like Google.
- 35:59They also like pretend they didn't do Operation Warp Speed
- 36:01even though that was like the coolest thing ever.
- 36:04I don't really know how long, like Phase 1 specifically.
- 36:07I know I don't know a lot. About.
- 36:08I mean, I know the entire half. The time, right?
- 36:11Yeah, so. I know the entire.
- 36:12I don't know. I think that's just like kind of
- 36:15an additive less necessary component of the prediction
- 36:19reducing the typical discovery. OK.
- 36:21So well it just says bypassing phase one clinical trial within
- 36:24a year, it is reducing the. Typical would be very fast.
- 36:27Would be a 50% improvement in speed I guess.
- 36:30I know the entire time is like passage.
- 36:32It's like 10. 10 to 15 years. And so it's on the order of
- 36:36years for sure for phase one. Gosh, yeah, I think I just got
- 36:41to stick with my, I don't believe that the the
- 36:44institutions change as fast as the technology here and take the
- 36:47under even though 20% with the new administration is is
- 36:50appealing, that's just feels like a high bar to pass phase
- 36:54one trials. It does seem like phase one
- 36:56involves actual experiments on yeah, which AI cannot speed up.
- 37:01I just read Dario's essay and like one of the main points that
- 37:06he makes in his bowl case on AI is just a lot of the limits will
- 37:10be human systems and things where you have to run real world
- 37:13experiments and even if it has good ideas, the experiments
- 37:17could take a while anyway. All right, would you say?
- 37:21I'm taking the under too. I think that it just seems too
- 37:24fast to occur. Would you guys take the over on
- 37:28Google entering phase one trials or something with something?
- 37:33Yeah, I think there'll be a discovery or some.
- 37:36Yeah, yeah. Registering for them or
- 37:38something? Over 20%, I would take the
- 37:39around 20%. Obviously Demis at DeepMind is
- 37:43quite passionate about biology, is a major application for AI.
- 37:47So yeah, I mean, I think that would be a cool thing for them
- 37:49to demo, say, hey, look, we made a drug.
- 37:52Yeah. All right.
- 37:54Let's do one more from Claude. All right, prediction.
- 37:59The first international treaty specifically governing AI
- 38:02development and deployment would be ratified by 15 countries or
- 38:06more, including three of the following countries, US, China,
- 38:12EU, UK. What's the percentage on this?
- 38:16Is 5% like? Yeah.
- 38:18What is this like a 1%? They're legally binding
- 38:21commitments on things like model evaluation and a standard.
- 38:26What is the percentage they gave?
- 38:28Yeah, 50%. Oh my God, this is that's the
- 38:31Claude's. That's the I feel like that we
- 38:33needed much better. That's like that's the easiest.
- 38:35Under Under. Claude's like, under Claude's
- 38:39like dreaming of this fantasy safety world or it's something
- 38:42where where every country has come together.
- 38:45China, or like I didn't get the memo.
- 38:48You don't need the US or China because you only need three of
- 38:51these entities. So you could have EUUK and Japan
- 38:56all entering some sort of treaty.
- 38:58OK, interesting. EUI would believe I mean UK
- 39:02though is sort of I mean I definitely am picking the.
- 39:05Under I just feel like the yeah, the UK is a mess economically
- 39:09right now. Like our, you know, is the
- 39:11Labour Party really going to be like, yeah, guys, we're not
- 39:14fixing the economy, but we signed this dumb AI treaty that
- 39:17none of you care about. Like, I, I don't know, I, I
- 39:20think they got bigger fish to fry there.
- 39:23Yeah, under, under A50 for sure. Yeah, I agree with you guys
- 39:27taking the under on that. Yeah, I think this is Claude's
- 39:32own fever dream of hoping for. A better place, a better world,
- 39:37a better world. AI where Claude itself is highly
- 39:39regulated, Yes. Yeah, OK, great.
- 39:45Great work everyone. That was fun.
- 39:48How do you guys think ChatGPT did compared to Claude?
- 39:52What do you like about the different predictions?
- 39:56Well, it seemed like Claude made-up some stuff.
- 39:58I did think it had a little bit of a safety ISM trend line.
- 40:04So I don't know. They're all blur to me but it
- 40:08felt like the first half. Were.
- 40:11More interesting in the second. It seemed like GBT's predictions
- 40:15were more entertaining as well as more somewhat realistic.
- 40:19I guess. So I think I would give GBT the
- 40:22clear victory in this round of entertaining and reasonably
- 40:26accurate predictions. Maybe you prompted?
- 40:28For O1 for reasoning models. Yeah, you prompted exactly the
- 40:32same stuff. Or.
- 40:33Exact same prompt. Yeah, yeah.
- 40:35Right. Well, that's a that makes me
- 40:37more excited for the full version of O1 coming out.
- 40:40Yeah, that would that's probably one of our one of our
- 40:42predictions. I think that's coming out this
- 40:43year, right, is the rumor. I mean, the zoom out thing that
- 40:47I took from this is just how much human systems interplay
- 40:53with the things we want AI to deliver.
- 40:56And even if we are bullish on rapid development of AI, there
- 41:02are a lot of things where we're like, well, the human systems
- 41:05will still effectively limit what it can do.
- 41:08That's self driving drug development, sort of everything
- 41:11under the sun. And so I think that's
- 41:14interesting to come out of this. How much?
- 41:18Yeah, this this sort of societal piece is still sort of a strong
- 41:23clever on what AI can accomplish.
- 41:25I feel like the way we get to that treaty is that something
- 41:29horrible goes on, right? Yeah, exactly right.
- 41:32Like, yeah. Something cool.
- 41:34Yeah. We underestimated Black Swan.
- 41:36We have to narrowly escape from some horrible thing to get a
- 41:40binding treaty between a bunch of countries, I think.
- 41:43Oh. Yeah, then we'll look really
- 41:44dumb. Yeah.
- 41:44It's like they, you know, AI does figure out how to like.
- 41:48Although we just had a paper. Clips or whatever to send a
- 41:50nuclear. Bomb we just had a pandemic and
- 41:53nothing came out of that like hey.
- 41:55Move on. It's not like.
- 41:58Some tragedies that bother us more than others.
- 42:00Yeah, but the US, China and the EU didn't get together and be
- 42:03like, hey, maybe we should like control pandemics a little more
- 42:05carefully. Like, you know, let's let's
- 42:06regulate that a little bit. It's like, Nah, we we got
- 42:09through it. Don't worry about it.
- 42:11Let's just get on the record on yeah, I mean the core just
- 42:14because it is a prediction episode and like we, we got into
- 42:17sort of the technicals like do you think the models will be
- 42:22significantly smarter next year, like by the end of next year in
- 42:26that they will we'll see the same basic rate of progress,
- 42:30meaning these are like PhD level.
- 42:34Thinkers I think by the next year, I think next year the
- 42:39overall take from this discussion like you said is the
- 42:41difference between things that can be facilitated by software,
- 42:44things that can be require hardware and then things that
- 42:47require institutions and government and so on right and I
- 42:51think next year I think it's going to be a crazy year for
- 42:54leaps forward and software. I think that GBT 5 is going to
- 42:58come I think 01 or O2 or whatever the GBT 5 level version
- 43:01of the thinking model is going to come.
- 43:03I think these computer use API's, as I said in a previous
- 43:06episode, or just an incredibly powerful interface to the
- 43:08Internet and the software world for these models.
- 43:11And so I think it's going to be like a crazy, crazy software a
- 43:17year. I'm I think it'll be the
- 43:18beginning of the impact on hardware and institutions.
- 43:23But it feels like the old Intel like tick tock strategy where
- 43:26you have one year where you have huge progress and then one year
- 43:29of refinement. And that's like the tick and the
- 43:31talk or whatever. I think 24 was kind of a talk
- 43:34year where we didn't have really dramatic progress in the models,
- 43:37but we started to see the existing models eat away at all
- 43:40these, you know, tools and software that we have.
- 43:44And now next year, I think we have a big, big tick here.
- 43:46And that'll be the foundation for like, you know, societal
- 43:50transformation for years and decades to come.
- 43:52So I'm I'm very long model progress in 25.
- 43:56I think that next year we'll we'll be talking about this
- 44:01trend of AI getting slower and slower.
- 44:04So we started out. Just going slower and adventure
- 44:09define. Slower.
- 44:10All right. No, no, no, I don't know.
- 44:12I'm not talking about reasoning capability.
- 44:14I'm talking about inference time essentially.
- 44:16And basically what you, what we've seen already is like with
- 44:21O1, right? And now it takes, you know, 5 to
- 44:2430 seconds to return an answer, but it can be better.
- 44:27What we're hearing is that, you know, internally, at least in
- 44:30open AI, they feel like that's a whole new scaling paradigm to
- 44:33get better and better reasoning is, is inference time compute.
- 44:37And what I think, you know, we'll see is like, like Max
- 44:41said, a lot of these computer use models are opening up
- 44:45opportunities for kind of agentic systems.
- 44:48And we'll start to see, you know, people like ourselves
- 44:51using these models that take hours to return US information,
- 44:55right? So Eric could go talk to an
- 44:57agent and just be like, hey, could you go research this thing
- 45:00for an upcoming article I'm writing?
- 45:02Or, you know, Max and I could have one of these agents go off
- 45:07and kind of research some game genre that we're looking to
- 45:10build, right? But it will just take longer and
- 45:13longer to return. And, and, you know, I think
- 45:15that'll be a really interesting trend.
- 45:17And I think that, you know, maybe that's the prediction for
- 45:21our, our own internal usage of how things are, how we're going
- 45:24to be changing how we work with these age, these AI systems.
- 45:27But I also think more specific prediction for like industry or
- 45:32or enterprise would be that we're going to see a lot of use
- 45:36cases in, in sort of the finance space for these agents, right?
- 45:39Like there will be new forms of hedge funds or investment groups
- 45:44that are like using these agents to actually trade.
- 45:47And, you know, maybe we'll see that primarily in crypto, but I
- 45:49could also see that occurring in sort of traditional assets.
- 45:52But I do think we're going to see a lot of like adoption of
- 45:55these agentic models to like trade their own portfolios on
- 45:59behalf of their, you know, human kind of owners or overseers, I
- 46:04guess. I mean, this isn't a prediction.
- 46:06And I think it sort of I mean, I think one thing we are seeing is
- 46:10that intelligence can be out there and most a lot of people
- 46:15don't know how to use it, right. So I think we're going to see
- 46:18obviously this uptick in applications taking existing,
- 46:22you know, the intelligence that has been proven with open AI,
- 46:25Anthropic and others and making it much more straightforward for
- 46:28people to use it in particular use cases.
- 46:30And I think there's going to be a ton of money there.
- 46:33So that's applications, which we've talked a lot about.
- 46:36And then I think the other piece that you're sort of touching on
- 46:38is that they're going to be these sort of specialists that
- 46:41appreciate how intelligent these models are.
- 46:44And they're going to figure out either how to use it and make
- 46:46money on it directly or become service providers for people who
- 46:51haven't really figured it out and do things better and say,
- 46:54all right, just pay me. I'll solve this problem.
- 46:56Secretly I'm using AI or not so secretly I'm using AI so I can
- 46:59do it cheaper than you. And I think that's how we're
- 47:01going to see these two models. There's sort of two ways people
- 47:05are going to implement AI. And I think so we'll see more AI
- 47:08applications and AI service companies.
- 47:11And, you know, the AI applications will almost
- 47:13certainly lag the actual power, the thinking power of AI.
- 47:18So it'll be an interesting sort of push and pull there.
- 47:22For sure. Cool.
- 47:25Yeah, Super fun. Yeah.
- 47:28Well, I feel, I don't know, James, you'll try and come up
- 47:32with the score of those predictions.
- 47:34I don't know. I think, I think to me they were
- 47:38more like what the draft I am invested, ego invested in the
- 47:42outcome. I feel like these predictions
- 47:44were more thought experiment than necessarily to me, the sort
- 47:48of perfect proxy for what we think, how accurately we see
- 47:53what's coming in AI, but certainly enjoyed it.
- 47:56That was a fun, fun concept. Fun episode.
- 47:58Yeah. Yeah, Greg.
- 48:01Well, like really really demonstrated that AI was better
- 48:03at creating the grist for the content.
- 48:06Than we are. Yeah, exactly.
- 48:07I mean, we were supposed to come up with predictions as like,
- 48:10yeah. I got a couple.
- 48:12Ideas. I mean it shows that sometimes
- 48:14like just having throwing shit at the wall to react to you is
- 48:16better than like overthinking my perfect prediction.
- 48:20And getting and getting your hot picks live, right?
- 48:23Getting. Yeah, exactly.
- 48:24Having to think through it. Seeing the thinking, that's
- 48:27something that, you know, AI is just starting to learn how to
- 48:30do. You know, humans still went on
- 48:33the showing our thought process. Anyway, we will be back after
- 48:38Cerebral Valley and we will be publishing all the talks.
- 48:43We're going to be throwing you our favorite clips and then we
- 48:46will be back with Max and James after the conference, reacting
- 48:51to everything that happened on stage.
- 48:52So come back and see us, newcomer.
- 48:56All right. Thanks so much.
- 48:58Thank you.