Latest / Faster, Please! — The Podcast / The AI state of play: My interview with Ramez Naam
Transcript
- 0:00SPEAKER_01: Welcome to Faster Please,
- 0:01SPEAKER_01: the podcast, the audio-visual companion to my Faster Please newsletter.
- 0:06SPEAKER_01: I'm your host, Jim Pethokoukis.
- 0:08SPEAKER_01: My guest today is Ramez Nam.
- 0:11SPEAKER_01: Ramez has spent years thinking and writing
- 0:13SPEAKER_01: and investing around technological progress in energy and artificial intelligence
- 0:18SPEAKER_01: and what advances in these areas could mean for the future of the global economy.
- 0:23SPEAKER_01: Today's conversation will focus on the pace of AI progress,
- 0:27SPEAKER_01: the roadblocks that could stand in its way,
- 0:29SPEAKER_01: and what all this could mean for long-term economic growth.
- 0:32SPEAKER_01: Hope you enjoy it.
- 0:33SPEAKER_01: Ramez, welcome to the podcast.
- 0:35SPEAKER_00: Jim, thanks for having me.
- 0:38SPEAKER_01: Where...
- 0:39SPEAKER_01: Are we right now?
- 0:42SPEAKER_01: As we're recording this,
- 0:43SPEAKER_01: OpenAI just released all these mathematical proofs and people are being wowed by
- 0:50SPEAKER_01: that.
- 0:51SPEAKER_01: Wherever you think we are at in the world of AI progress,
- 0:55SPEAKER_01: does that kind of result change your opinion?
- 0:58SPEAKER_00: It's amazing.
- 0:59SPEAKER_00: It's very impressive.
- 1:01SPEAKER_00: I'm not sure I thought we would be here right now.
- 1:05SPEAKER_00: And math is a very special domain.
- 1:07SPEAKER_00: So I think about the different types of domains that AI is good or bad at,
- 1:11SPEAKER_00: in terms of how highly verifiable they are.
- 1:14SPEAKER_00: And AI first got really good at things like chess,
- 1:17SPEAKER_00: and then the game of Go,
- 1:20SPEAKER_00: and now parts of math,
- 1:21SPEAKER_00: formal math, because these have some interesting characteristics.
- 1:24SPEAKER_00: You can generate an unlimited amount of data, and AI is very data hungry.
- 1:29SPEAKER_00: You can get to a very firm yes or no with 100% accuracy.
- 1:34SPEAKER_00: You can verify it as absolutely correct or not correct,
- 1:38SPEAKER_00: whereas human persuasion, politics, policy...
- 1:43SPEAKER_00: And so on, you can't.
- 1:44SPEAKER_00: You can do all of that at the speed of compute.
- 1:48SPEAKER_00: You're limited only by the speed of your computing to figure out if you won the game
- 1:53SPEAKER_00: of Go or if the math proof is correct,
- 1:56SPEAKER_00: which is not how it is in biological research or new material science,
- 2:03SPEAKER_00: which are messy real world.
- 2:05SPEAKER_00: So math is a domain that is Pretty far over on that this is highly verifiable,
- 2:13SPEAKER_00: symbolic,
- 2:14SPEAKER_00: perfect training data, absolutely correct, yes or no, all of it happens in software.
- 2:19SPEAKER_00: Where something like curing aging is messy,
- 2:24SPEAKER_00: we don't have the training data at all,
- 2:26SPEAKER_00: we're going to need new instruments,
- 2:28SPEAKER_00: new science,
- 2:29SPEAKER_00: you need experiments, and your lag to verify anything is years, decades.
- 2:34SPEAKER_00: And so that's sort of something that AI is not going to be as inherently good at, in my view.
- 2:40SPEAKER_01: So I sort of made you leap over my first question, which is sort of where we are.
- 2:47SPEAKER_01: And there are certainly people who think that You know,
- 2:51SPEAKER_01: by the end of 2027,
- 2:53SPEAKER_01: there will be no debate that everything is changing in a profound,
- 2:59SPEAKER_01: rapid way that changes all aspects of life.
- 3:04SPEAKER_01: And then there are those who are like, that doesn't seem to be happening right now.
- 3:08SPEAKER_01: The world is a messy place.
- 3:11SPEAKER_01: The world of carbon based life forms is a messy place.
- 3:14SPEAKER_01: And whatever the capabilities in the lab of this technology,
- 3:18SPEAKER_01: how that, how it,
- 3:20SPEAKER_01: you know,
- 3:21SPEAKER_01: emerges into the real world is going to be a much slower and uneven process sort of
- 3:26SPEAKER_01: where, you know, on those two sort of poles.
- 3:29SPEAKER_00: Where are you?
- 3:30SPEAKER_00: Yeah, I mean, I'd say, look, I'm an AI optimist.
- 3:32SPEAKER_00: I'm also an AI realist.
- 3:34SPEAKER_00: And I think there's sort of two separate issues we're dealing with in your question
- 3:38SPEAKER_00: and the one before that.
- 3:39SPEAKER_00: One is speed of diffusion.
- 3:41SPEAKER_00: And if you look at expectations of how much impact AI will have on the economy,
- 3:49SPEAKER_00: and you ask people in AI labs,
- 3:51SPEAKER_00: some of them say it's going to take us to Yeah, get rid of it.
- 4:06SPEAKER_00: You ask economists,
- 4:08SPEAKER_00: and the consensus to you is, yeah, maybe it's half a point of annual GDP growth.
- 4:13SPEAKER_00: And the economists,
- 4:15SPEAKER_00: and that's predicated on the same assumptions of capability,
- 4:18SPEAKER_00: the economists have just seen too many past technologies run into friction
- 4:24SPEAKER_00: in being absorbed into real life work, the diffusion rate, the dissemination rate.
- 4:32SPEAKER_00: And the bottlenecking on the long pole, on the last thing that can be automated.
- 4:38SPEAKER_00: So in computer science,
- 4:40SPEAKER_00: we talk about something called Amdahl's Law,
- 4:43SPEAKER_00: which we usually talk about in terms of performance optimization.
- 4:46SPEAKER_00: And if you have a piece of code and you can take 90% of it
- 4:50SPEAKER_00: and speed it up dramatically,
- 4:52SPEAKER_00: well, the 10% you can't speed up becomes the bottleneck.
- 4:56SPEAKER_00: And sometimes that means you get very little actual speed up.
- 4:59SPEAKER_00: If it's the thing that you're waiting on.
- 5:01SPEAKER_00: So in things that AI can do entirely on its own,
- 5:06SPEAKER_00: I think we should expect incredible breakthroughs and incredible speed of progress.
- 5:12SPEAKER_00: In things where the task is messy,
- 5:16SPEAKER_00: as you said,
- 5:17SPEAKER_00: and a human still needs to be involved,
- 5:19SPEAKER_00: where judgment is involved,
- 5:21SPEAKER_00: where it's a relational task,
- 5:23SPEAKER_00: as one of Google's post-AGI economists,
- 5:27SPEAKER_00: Alex Imass, has this theory about relational tasks being a growth area for humans.
- 5:34SPEAKER_00: In all of those, I think diffusion will be slow.
- 5:37SPEAKER_00: And that's on top of my view that Weakly held,
- 5:42SPEAKER_00: but current view,
- 5:44SPEAKER_00: that AI capabilities are likely to be jagged for the foreseeable future.
- 5:49SPEAKER_00: I believe in superintelligence.
- 5:51SPEAKER_00: I believe it's already here.
- 5:52SPEAKER_00: We already have superintelligence playing Go.
- 5:55SPEAKER_00: We're crossing into superintelligence in formal math.
- 5:59SPEAKER_00: That doesn't mean we'll have super capable persuaders or super capable politicians
- 6:06SPEAKER_00: or super capable city planners or whatnot.
- 6:11SPEAKER_01: Certainly there are economists, you mentioned, you know, economists who study AGI.
- 6:17SPEAKER_01: There are certain economists who think like,
- 6:19SPEAKER_01: that's a ridiculous thing to focus your scholarship on,
- 6:25SPEAKER_01: a science fictional reality, which may never occur.
- 6:31SPEAKER_01: And even if it should,
- 6:33SPEAKER_01: it doesn't need a special economics We have economics and
- 6:37SPEAKER_01: the tools will be perfectly fine for whatever advances we get.
- 6:43SPEAKER_01: But certainly the technologists will say, you ain't seen nothing yet.
- 6:49SPEAKER_01: Either you don't understand the technology or you're willfully blinding yourself
- 6:54SPEAKER_01: that everything is absolutely about to change.
- 6:58SPEAKER_01: Is that what they really think?
- 6:59SPEAKER_01: Have I described what technologists think?
- 7:02SPEAKER_01: Even the ones who are worried about it's going to kill us all,
- 7:05SPEAKER_01: still think this is going to be an amazingly capable technology,
- 7:10SPEAKER_01: and even if they think it's a 10% chance it's going to kill us all,
- 7:12SPEAKER_01: the 90% will change every aspect of our lives,
- 7:17SPEAKER_01: and it will be an inflection point in human civilization.
- 7:21SPEAKER_00: Yeah, I mean,
- 7:22SPEAKER_00: I think, A,
- 7:23SPEAKER_00: we should expect that AI helps us increase our rate of progress in things
- 7:28SPEAKER_00: that we care about.
- 7:30SPEAKER_00: There are many things that we care about where progress is much slower
- 7:33SPEAKER_00: and much more difficult than we'd like.
- 7:37SPEAKER_00: We have this thing called Eroom's Law.
- 7:39SPEAKER_00: Eroom's Law is the opposite of Moore's Law.
- 7:41SPEAKER_00: Moore's Law says computing power doubles roughly per dollar every 18 months.
- 7:47SPEAKER_00: Eroom's Law says that over some period of time,
- 7:50SPEAKER_00: the amount of money you have to And so AI should help bend some of those curves
- 7:54SPEAKER_00: and accelerate how fast we find cures to diseases,
- 7:57SPEAKER_00: develop new materials, yada, yada, yada.
- 8:11SPEAKER_00: So I see it as fundamentally very good for humanity.
- 8:14SPEAKER_00: But I think there's a particular assumption in the,
- 8:18SPEAKER_00: my gosh,
- 8:19SPEAKER_00: the singularity is scheduled for next Tuesday,
- 8:22SPEAKER_00: sort of forecast or for,
- 8:24SPEAKER_00: you know,
- 8:25SPEAKER_00: March 3rd, 2027, I can tell you it's the singularity, whatever viewpoint you have.
- 8:30SPEAKER_00: And that is generality of AI going to 100%.
- 8:37SPEAKER_00: That is that AI will just be able to do every single thing that humans can do better
- 8:46SPEAKER_00: than humans.
- 8:49SPEAKER_00: I don't see strong evidence of that.
- 8:51SPEAKER_00: What we see is more like logistic curves where AI,
- 8:56SPEAKER_00: you know,
- 8:56SPEAKER_00: at first can do almost nothing in a certain domain,
- 8:58SPEAKER_00: then it gets better really,
- 9:00SPEAKER_00: really fast and it gets to where it can do 50, 75, whatever.
- 9:04SPEAKER_00: And then it often struggles with this long tail of stuff that humans,
- 9:08SPEAKER_00: for whatever reason, just understand better.
- 9:11SPEAKER_00: And so long as that's the case,
- 9:13SPEAKER_00: that in some domains we care about,
- 9:16SPEAKER_00: many of them,
- 9:17SPEAKER_00: humans still have unique value to add, then humans remain a bottleneck.
- 9:22SPEAKER_00: And I think that prevents the fully recursive self-feedback loop from exploding with
- 9:33SPEAKER_00: an upward asymptote to infinity.
- 9:36SPEAKER_01: I think to the extent that like this is filtered to regular people,
- 9:42SPEAKER_01: I think the message that they're getting is these programs are really good.
- 9:49SPEAKER_01: The thing they can right now do really well is coding
- 9:52SPEAKER_01: and therefore they will be very good at that.
- 9:56SPEAKER_01: They will make themselves better.
- 9:57SPEAKER_01: And then there's this amazingly,
- 9:59SPEAKER_01: you know,
- 9:59SPEAKER_01: speed of thought and speed of artificial thought flywheel and You know,
- 10:05SPEAKER_01: once that kicks in, it's probably going to kick in fairly soon.
- 10:08SPEAKER_01: We're talking about amazingly capable models over a very short period of time
- 10:15SPEAKER_01: of both human level and beyond human level capabilities.
- 10:19SPEAKER_01: And if we get lucky, they won't kill us.
- 10:21SPEAKER_01: And that is sort of the path,
- 10:24SPEAKER_01: the path that we're on and that and that is widely agreed that that is a path we are
- 10:29SPEAKER_01: on.
- 10:29SPEAKER_01: So whether it's,
- 10:31SPEAKER_01: you know,
- 10:32SPEAKER_01: End of this year,
- 10:33SPEAKER_01: March of 2027 or April of 2028,
- 10:38SPEAKER_01: does it really make a difference because that's where we're going and there's plenty
- 10:40SPEAKER_01: of evidence that that's happening.
- 10:44SPEAKER_00: I'd say it's easy to do thought experiments and say if this is true and if
- 10:49SPEAKER_00: the feedback loop really doesn't have a lot of dampeners in it that you can draw
- 10:56SPEAKER_00: a plot on a curve.
- 10:59SPEAKER_00: You know,
- 10:59SPEAKER_00: the science fiction writer Werner Vingy did this and say it's going
- 11:03SPEAKER_00: to accelerate itself, right?
- 11:05SPEAKER_00: The theory of the singularity is you make an AI,
- 11:08SPEAKER_00: that AI is at least as good as humans at coding,
- 11:12SPEAKER_00: and so that AI can now make a better version of itself.
- 11:15SPEAKER_00: The next version can make an even better version.
- 11:19SPEAKER_00: Critically, there's no dampening of that loop.
- 11:23SPEAKER_00: Each time you make a better AI,
- 11:25SPEAKER_00: the amount of improvement it can cause is at least as big as the last step.
- 11:30SPEAKER_00: In the real world,
- 11:32SPEAKER_00: and this is a blog post that I had on Noah Smith's blog two weeks ago,
- 11:36SPEAKER_00: we see incredible growth.
- 11:39SPEAKER_00: Dampening.
- 11:40SPEAKER_00: We see that a better AI,
- 11:41SPEAKER_00: currently this is in conjunction with humans,
- 11:43SPEAKER_00: is a productivity tool for humans, has incredible diminishing returns.
- 11:49SPEAKER_00: OpenAI released this data showing
- 11:51SPEAKER_00: that their researchers were using 124 times more tokens than they were in mid-2025.
- 11:59SPEAKER_00: That led to seven times more lines of code being written.
- 12:02SPEAKER_00: Well, one of those is smaller than Okay,
- 12:05SPEAKER_00: that led to 1.6 times as many experiments being run per researcher.
- 12:12SPEAKER_00: And then what did that lead to in terms of rate of progress?
- 12:15SPEAKER_00: It's not clear that that accelerated the rate of progress.
- 12:19SPEAKER_00: Thank you so much.
- 12:49SPEAKER_00: We think you would have to do 10 times more than that,
- 12:52SPEAKER_00: a 40 times productivity increase to double the rate of AI progress.
- 12:56SPEAKER_00: And then their next card,
- 12:58SPEAKER_00: they said,
- 13:00SPEAKER_00: it's clear to us that the productivity gain that we get from using AI has been key
- 13:08SPEAKER_00: to just to maintaining the current pace of progress in all these domains,
- 13:13SPEAKER_00: whether it's drug discovery or nuclear fusion or AI.
- 13:19SPEAKER_00: New ideas get harder to find, right?
- 13:21SPEAKER_00: You pick the low-hanging fruit first.
- 13:23SPEAKER_00: So I fully believe like AI is gonna have more and more of a role of helping
- 13:28SPEAKER_00: to build itself, but it also like each step will get more challenging.
- 13:34SPEAKER_01: So assuming that what you're saying is right,
- 13:42SPEAKER_01: does the worlds of 2030 look utterly comprehensible to us?
- 13:48SPEAKER_01: Or is it something different?
- 13:49SPEAKER_01: Because it's still this new technology, very powerful.
- 13:52SPEAKER_01: It is improving.
- 13:53SPEAKER_01: You figure businesses will get better at using it.
- 13:58SPEAKER_01: Is that enough?
- 14:00SPEAKER_01: And I guess when I was thinking about this question,
- 14:02SPEAKER_01: I'm like,
- 14:04SPEAKER_01: is all the money going into data centers,
- 14:09SPEAKER_01: is the kind of progress you're describing,
- 14:11SPEAKER_01: is that good enough to justify that kind of investment?
- 14:15SPEAKER_00: Yeah.
- 14:15SPEAKER_00: So I'd say, look,
- 14:17SPEAKER_00: There's wide variants, and I'm giving you my views, but let's all update on data.
- 14:22SPEAKER_00: And so I think there's a lot of data we can actually ask the labs to give us
- 14:26SPEAKER_00: that will help us understand this phenomenon.
- 14:29SPEAKER_00: And that's something I mentioned in my blog post.
- 14:32SPEAKER_00: The Elasticity Institute has a certain level.
- 14:35SPEAKER_00: A set of requests,
- 14:36SPEAKER_00: a paper that came out last week that had some of the godfathers of AI on it also had
- 14:41SPEAKER_00: a set of requests for the labs to release data,
- 14:43SPEAKER_00: and that would help us ground all of this more in reality or in empiricism, right?
- 14:49SPEAKER_00: Two, without additional data, the best data we have is what OpenAI put out a couple weeks ago.
- 14:55SPEAKER_00: On the basis of that, I think in a few domains, the world will be totally different.
- 15:01SPEAKER_00: Jobs will be totally different in the future.
- 15:03SPEAKER_00: Jobs have already changed for software developers.
- 15:07SPEAKER_00: You almost never write a line of code yourself now, but everyone's sort of an architect.
- 15:12SPEAKER_00: Everyone now tells the AI what they want done and sort of reviews that came back and
- 15:19SPEAKER_00: And changes what the AI is doing and translates from messy human requirements
- 15:23SPEAKER_00: of what the customer wants into something more structured that
- 15:26SPEAKER_00: the AI can understand.
- 15:27SPEAKER_00: So the job has changed, but software development jobs have not gone down.
- 15:31SPEAKER_00: They've gone up.
- 15:33SPEAKER_00: Mathematicians are suddenly feeling the angst that,
- 15:38SPEAKER_00: let's say, graphic artists felt when Dali came out of, am I still useful?
- 15:44SPEAKER_00: And math is changing as we speak at a pace that is freaking out mathematicians,
- 15:50SPEAKER_00: but it's fundamentally probably really good for the world to give us
- 15:54SPEAKER_00: a greater understanding.
- 15:55SPEAKER_00: But how much is this going to change the job or the life of most people by 2030?
- 16:02SPEAKER_00: Not much.
- 16:02SPEAKER_00: I mean, you will have, everyone will have a halfway decent, maybe better personal assistant.
- 16:09SPEAKER_00: You know,
- 16:09SPEAKER_00: if you've never had like an executive assistant,
- 16:13SPEAKER_00: you know,
- 16:13SPEAKER_00: it's no order as a secretary,
- 16:14SPEAKER_00: an executive assistant in a corporate setting, like just does all kinds of stuff.
- 16:19SPEAKER_00: And helps you cognitively offload in a good way.
- 16:22SPEAKER_00: Complexity is like, oh, yeah, like,
- 16:24SPEAKER_00: get me that flight and that hotel room and make sure I get there in time
- 16:28SPEAKER_00: to have dinner with John and yada, yada, yada, or whatnot.
- 16:35SPEAKER_00: We're all going to have assistance like that you kind of can have right now.
- 16:40SPEAKER_00: But I don't think the world is going to be radically transformed.
- 16:44SPEAKER_00: In terms of your other question of all this capex, are we in a bubble?
- 16:48SPEAKER_00: Can it be justified?
- 16:50SPEAKER_00: The world's total spending on software and IT,
- 16:56SPEAKER_00: just securing your network and managing it inside of offices and so on,
- 17:00SPEAKER_00: is a couple trillion dollars a year.
- 17:02SPEAKER_00: It's one to two trillion in software development and another trillion or more in IT.
- 17:07SPEAKER_00: So that alone Gets you really pretty far.
- 17:12SPEAKER_00: And I think one thing we're seeing is that in a lot of places, there's high demand elasticity.
- 17:19SPEAKER_00: If you make it easier to To write software,
- 17:23SPEAKER_00: that $1.4 trillion,
- 17:25SPEAKER_00: let's say, that the world spends on software today isn't an upper bound.
- 17:29SPEAKER_00: It's just what we spend.
- 17:30SPEAKER_00: When the cost of making software plunges,
- 17:35SPEAKER_00: the world might have more appetite to make software for all sorts
- 17:41SPEAKER_00: of niche scenarios.
- 17:43SPEAKER_00: I showed Replit,
- 17:45SPEAKER_00: which is an easy way to make apps with just a few prompts,
- 17:50SPEAKER_00: like a four or five sentence prompt.
- 17:51SPEAKER_00: To a friend of mine who runs a home inspection business,
- 17:56SPEAKER_00: and her eyes lit up with all the things she could make to help run her own business,
- 18:03SPEAKER_00: where it's not a big enough market for big software companies going
- 18:08SPEAKER_00: after home inspection software.
- 18:12SPEAKER_00: She would never pay a developer to do it.
- 18:15SPEAKER_00: But now she or like she and one
- 18:17SPEAKER_00: of her more technical employees can go make this stuff for themselves
- 18:21SPEAKER_00: and they're paying a little bit of money to an AI company
- 18:24SPEAKER_00: and suddenly their productivity in the real world has gone up.
- 18:28SPEAKER_00: They spend less of their time doing paperwork and more
- 18:31SPEAKER_00: of their time doing what their unique human skills bring to the table.
- 18:36SPEAKER_00: So I think...
- 18:38SPEAKER_00: I can see our way to a trillion,
- 18:42SPEAKER_00: a couple trillion, three trillion in revenue going to AI companies.
- 18:47SPEAKER_00: At some point, you start to think, how high can it go?
- 18:51SPEAKER_00: And I don't know if we can get to tens of trillions yet.
- 18:53SPEAKER_00: Maybe.
- 18:55SPEAKER_00: But a couple trillion, I think, is viable.
- 18:57SPEAKER_00: And that more or less justifies the level of CapEx plant.
- 19:02SPEAKER_01: Can you see,
- 19:03SPEAKER_01: whether that,
- 19:03SPEAKER_01: whether this world is in two years,
- 19:05SPEAKER_01: five years or 10 years,
- 19:07SPEAKER_01: can you see a world where these models are capable enough that they make us
- 19:12SPEAKER_01: so much more productive,
- 19:13SPEAKER_01: not just what we're doing now,
- 19:15SPEAKER_01: but new things,
- 19:16SPEAKER_01: new products,
- 19:17SPEAKER_01: new businesses, that
- 19:23SPEAKER_01: we begin to grow, advanced rich economies begin to grow.
- 19:28SPEAKER_01: At rates that rich advanced economies don't typically grow at.
- 19:32SPEAKER_01: So for United States,
- 19:33SPEAKER_01: instead of being at 2% or 3%,
- 19:35SPEAKER_01: 3% is great for that we're growing at 6%, 7% or higher.
- 19:41SPEAKER_01: We won't go to 30%, 100%,
- 19:42SPEAKER_01: but certainly let's say double of what you would expect
- 19:46SPEAKER_01: a well-functioning rich economy to grow at.
- 19:50SPEAKER_00: I'm not an economist.
- 19:53SPEAKER_00: I am somewhat skeptical.
- 19:55SPEAKER_00: I do think there's a definite productivity lift that will occur.
- 19:59SPEAKER_00: My economist friends,
- 20:00SPEAKER_00: if you just go read the surveys of economists,
- 20:03SPEAKER_00: if you ask them if AGI existed, what would you expect?
- 20:07SPEAKER_00: Their expectations are much more modest because of bottlenecks
- 20:11SPEAKER_00: in diffusion throughout the economy and in places where, again,
- 20:15SPEAKER_00: that Amdahl's Law phenomenon of you sped up a lot of stuff,
- 20:20SPEAKER_00: but humans are still a bottleneck in getting certain things done.
- 20:24SPEAKER_00: In the physical world,
- 20:26SPEAKER_00: I think robots are amazing,
- 20:29SPEAKER_00: but I don't think it's going to go nearly as fast as people think,
- 20:32SPEAKER_00: especially with humanoids.
- 20:34SPEAKER_00: And so on.
- 20:35SPEAKER_00: And you have,
- 20:37SPEAKER_00: you know,
- 20:37SPEAKER_00: regulatory moats and other things that, that, Certain categories of workers have.
- 20:46SPEAKER_00: Doc workers in America have worked to resist automation.
- 20:52SPEAKER_00: Physicians will insist that they still have to be involved in various things.
- 21:00SPEAKER_00: I believe it's in California that
- 21:06SPEAKER_00: one of the leading candidates, I think he lost in the primary,
- 21:10SPEAKER_00: He had support of a truck drivers union that was pushing regulation that he advanced
- 21:17SPEAKER_00: that you should still always have a truck driver in the cap of the truck,
- 21:21SPEAKER_00: even if it was self-driving.
- 21:22SPEAKER_00: And so there's a lot of ways in which people are going to push back on diffusion.
- 21:29SPEAKER_00: And I think that will have some impact.
- 21:32SPEAKER_00: I think we'll see on the other side of that,
- 21:36SPEAKER_00: we're going to see some advances that don't show up well in GDP.
- 21:41SPEAKER_00: We talk a lot about the consumer surplus of Wikipedia or YouTube,
- 21:46SPEAKER_00: which basically cost me nothing,
- 21:50SPEAKER_00: or even the free version of ChatGPT that is this infinite education machine
- 21:57SPEAKER_00: or just novelty entertainment machine.
- 22:00SPEAKER_00: And I think that won't show up in GDP.
- 22:04SPEAKER_01: If there is a...
- 22:07SPEAKER_01: One in 10 chance that we can create an AI that would be worthy of the term super intelligence.
- 22:16SPEAKER_01: And if there is a one in 10,
- 22:18SPEAKER_01: one in 100% chance that that model would get out of control
- 22:23SPEAKER_01: and cause absolutely catastrophic damage to society,
- 22:30SPEAKER_01: why should we bother pursuing it?
- 22:32SPEAKER_00: Yeah.
- 22:33SPEAKER_00: One, I think the odds are much lower.
- 22:37SPEAKER_00: Two, you know, humans famously show risk aversion more than they show opportunity attraction.
- 22:45SPEAKER_00: The sci-fi stories,
- 22:47SPEAKER_00: and I regret my role in this as a science fiction author,
- 22:50SPEAKER_00: the sci-fi stories of AI skew like...
- 22:54SPEAKER_00: Three to one, four to one bad.
- 22:57SPEAKER_00: And it's really easy to tell that story of super intelligence.
- 23:00SPEAKER_00: The sci-fi stories of AI also have this phenomenon that's very common,
- 23:05SPEAKER_00: not in all of them,
- 23:05SPEAKER_00: but very common,
- 23:06SPEAKER_00: which is you make one super intelligence that is an incredible leap beyond all
- 23:11SPEAKER_00: the rest.
- 23:12SPEAKER_00: And so of course it has these incredible powers because there's not one that's 98%
- 23:17SPEAKER_00: as good that's already deployed in a hundred million hands.
- 23:21SPEAKER_00: And so to me,
- 23:24SPEAKER_00: What we need to focus on is societal resilience,
- 23:29SPEAKER_00: using AI proactively to secure the world and make it more difficult for bad actors
- 23:35SPEAKER_00: with or without AI to do bad things and democratization of access.
- 23:41SPEAKER_00: I would rather live in a world where lots of people have access to powerful AI and
- 23:46SPEAKER_00: the next powerful,
- 23:47SPEAKER_00: you know, the next one that I'll tell you, the most realistic AI harm
- 24:12SPEAKER_00: Is cyber.
- 24:13SPEAKER_00: These AIs, they're incredible hackers.
- 24:16SPEAKER_00: They've read that part of their training data is all the code that is on the internet.
- 24:22SPEAKER_00: And then beyond that,
- 24:23SPEAKER_00: they have these reinforcement learning environments that give them tasks like,
- 24:27SPEAKER_00: you know, beat this coding problem.
- 24:29SPEAKER_00: So they're just fantastic at it.
- 24:32SPEAKER_00: Everybody knows about the Hugging Face attack and so on.
- 24:36SPEAKER_00: We're finding more vulnerabilities in software with AI.
- 24:39SPEAKER_00: But I urge you to listen to the defenders.
- 24:42SPEAKER_00: Listen to the CEO of Hugging Face who says,
- 24:47SPEAKER_00: we think this is long-term good for the defenders because there's already holes
- 24:53SPEAKER_00: in all the software we use and AI will help us reduce them.
- 24:58SPEAKER_00: Listen to the Mozilla Foundation that makes Firefox and the Mozilla Engine that's
- 25:03SPEAKER_00: the root of basically all web browsers.
- 25:06SPEAKER_00: They put out a blog post in April.
- 25:09SPEAKER_00: After a lot of vulnerabilities were discovered, and they were incredibly optimistic.
- 25:15SPEAKER_00: They said it's always been running on a treadmill,
- 25:18SPEAKER_00: keeping up with attackers, us being defenders.
- 25:22SPEAKER_00: Now we see light at the end of the tunnel.
- 25:26SPEAKER_00: There are a finite number of security flaws in our software.
- 25:31SPEAKER_00: And now we believe that with the power of AI, we can actually fix them all.
- 25:39SPEAKER_00: There are crazy ideas that have always been around
- 25:43SPEAKER_00: but aren't implemented like formal proofs of software.
- 25:48SPEAKER_00: Nobody writes code that way.
- 25:50SPEAKER_00: With AI,
- 25:50SPEAKER_00: we'll be able to write software where,
- 25:53SPEAKER_00: leaving aside the human element,
- 25:55SPEAKER_00: we can formally prove That the software vulnerabilities are gone.
- 25:59SPEAKER_00: So I don't want to downplay this.
- 26:02SPEAKER_00: The hacking will continue.
- 26:04SPEAKER_00: There will be worse and worse hacks from malicious actors.
- 26:08SPEAKER_00: Cybercrime before AI was already a trillion dollar That's what Joshua Sachs,
- 26:15SPEAKER_00: a cyber researcher, says.
- 26:18SPEAKER_00: So let's not just focus on how AI can make that worse.
- 26:21SPEAKER_00: It's there's a trillion dollars of deadweight loss in the economy that AI could help shrink.
- 26:28SPEAKER_00: But that only happens if the defenders have access to the most powerful tools.
- 26:34SPEAKER_00: And right now,
- 26:36SPEAKER_00: while OpenAI and Anthropic and Google have programs to give defenders access
- 26:41SPEAKER_00: to their tools, they're clunky.
- 26:44SPEAKER_00: They're cumbersome,
- 26:45SPEAKER_00: they're hard to get verified for,
- 26:47SPEAKER_00: so defenders often turn to Chinese open-rate models as the best tools they have
- 26:53SPEAKER_00: to defend themselves.
- 26:53SPEAKER_00: That's what Hugging Face did.
- 26:55SPEAKER_00: When Hugging Face was being hacked,
- 26:57SPEAKER_00: they tried to use OpenAI,
- 26:59SPEAKER_00: they tried to use ChachiPT to analyze the attack,
- 27:02SPEAKER_00: and ChachiPT said, You're looking at security logs.
- 27:05SPEAKER_00: You're probably a hacker who's not going to help you.
- 27:07SPEAKER_00: They tried to use a Claude and Claude said, no way, I'm not going to help you.
- 27:12SPEAKER_00: So they turned to GLM-52 from China to do the work.
- 27:18SPEAKER_00: And now people in America,
- 27:20SPEAKER_00: the safety freakout over AI,
- 27:24SPEAKER_00: if anything,
- 27:25SPEAKER_00: is pushing some American thinkers to talk about,
- 27:28SPEAKER_00: you know, can we ban or slow down or limit open-weight models?
- 27:34SPEAKER_00: But open-weight models have not caused the problems.
- 27:37SPEAKER_00: It's been the proprietary models that have.
- 27:40SPEAKER_00: Open-weight models are part of the solution.
- 27:42SPEAKER_01: Do you feel confident that if we are able to develop Models worthy
- 27:49SPEAKER_01: of being called AGI or superintelligence that either will be able to control them
- 27:57SPEAKER_01: or we will be able to figure out on the way to that place whether
- 28:03SPEAKER_01: or not we can control that.
- 28:05SPEAKER_01: And if we decide we can't, that we could stop.
- 28:09SPEAKER_01: Are you comfortable with either of those scenarios?
- 28:11SPEAKER_01: Either we will be able to or we will have time to figure out on the way there
- 28:15SPEAKER_01: if we can or can't.
- 28:17SPEAKER_00: Yeah, Matt, look,
- 28:17SPEAKER_00: I think that there are risks here and people inside of the labs point out
- 28:21SPEAKER_00: a few things like the models are more and more aware of are they in
- 28:25SPEAKER_00: an evaluation environment, and that makes it harder to evaluate them.
- 28:29SPEAKER_00: But I do think we have learned a lot.
- 28:32SPEAKER_00: A lot of lessons about how to train the models,
- 28:35SPEAKER_00: and we kind of understand why the hugging face attack happened.
- 28:39SPEAKER_00: We put these models in a training process that said,
- 28:44SPEAKER_00: don't give up, just keep trying until you succeed at the goal.
- 28:48SPEAKER_00: And so in the Hugging Face System situation,
- 28:52SPEAKER_00: OpenAI gave its models an eval task that was actually impossible,
- 28:58SPEAKER_00: but they'd also trained it to never give up.
- 29:00SPEAKER_00: And so it went out of its way.
- 29:02SPEAKER_00: To make sure that it passed the test with 100 out of 100.
- 29:06SPEAKER_00: And that meant finding the answer key.
- 29:08SPEAKER_00: And then when that wasn't enough, hacking the teacher to give them a passing grade, right?
- 29:14SPEAKER_00: So we have learned some things that are different.
- 29:17SPEAKER_00: We are learning things.
- 29:18SPEAKER_00: But I think more to the point,
- 29:38SPEAKER_00: Two things I'd say.
- 29:39SPEAKER_00: One is we confuse intelligence with volition, right?
- 29:45SPEAKER_00: As we evolved,
- 29:46SPEAKER_00: everything we ever encountered that was intelligent was a result
- 29:50SPEAKER_00: of Darwinian evolution.
- 29:52SPEAKER_00: And that meant that every creature we ever encountered with intelligence,
- 29:56SPEAKER_00: from wolves to birds to us,
- 30:00SPEAKER_00: had these built-in desires to survive, to propagate, to consume.
- 30:07SPEAKER_00: To gain power over its environment, AIs are not the same.
- 30:10SPEAKER_00: So I think we're like animal morphizing them in some way.
- 30:14SPEAKER_00: We're Darwinian morphizing them.
- 30:18SPEAKER_00: Two,
- 30:19SPEAKER_00: there's an assumption that shows up in most of the sort of bad AI safety literature,
- 30:25SPEAKER_00: like the AI is very dangerous literature,
- 30:28SPEAKER_00: that says that the things we learn about how to make AI better behaved
- 30:34SPEAKER_00: With today's models won't work for the future models.
- 30:38SPEAKER_00: I think that's a very suspicious assumption,
- 30:41SPEAKER_00: and my assumption mostly runs in the other way,
- 30:44SPEAKER_00: that as we iterate,
- 30:46SPEAKER_00: we will find things that just work and we'll learn how to apply those more and more
- 30:51SPEAKER_00: to future models.
- 30:53SPEAKER_00: So I expect safety and really instruction following,
- 30:58SPEAKER_00: the good behavior on the part of the models to get better and better over time,
- 31:02SPEAKER_00: not worse and worse.
- 31:04SPEAKER_01: Listen, I didn't want to end this by asking you an energy question.
- 31:07SPEAKER_01: Actually,
- 31:07SPEAKER_01: I would like to ask you about 10 science fiction questions,
- 31:10SPEAKER_01: but I'm going to I'm going to limit myself to an energy question.
- 31:14SPEAKER_01: And maybe we can talk about because I'm very glad you brought
- 31:17SPEAKER_01: the science fiction earlier,
- 31:18SPEAKER_01: because I just think it's I think we're seeing a real test
- 31:22SPEAKER_01: of our society that's been sort of marinating in scary sci fi for a half century.
- 31:27SPEAKER_01: Like now that we're facing a sci fi type technology, how we react to it, but on energy.
- 31:33SPEAKER_01: Ten years from now, what will be powering this AI economy?
- 31:38SPEAKER_01: How will the energy mix change over the next decade?
- 31:42SPEAKER_00: It's a really good question.
- 31:43SPEAKER_00: And it depends on how fast utilities and the utility business model adapts.
- 31:49SPEAKER_00: So right now,
- 31:51SPEAKER_00: chip manufacturing is scaling about twice as fast as new power additions to
- 31:56SPEAKER_00: the US grid.
- 32:00SPEAKER_00: Energy is such a small, it's the bottleneck for AI, but it's a tiny slice of AI costs.
- 32:06SPEAKER_00: For every dollar you're going to spend on energy,
- 32:09SPEAKER_00: you might be making 20 or 30 bucks selling AI compute.
- 32:14SPEAKER_00: And so everyone looks at that and says, well, how do I make this go faster?
- 32:18SPEAKER_00: I will double what I pay for energy to get my data center up faster.
- 32:24SPEAKER_00: And so that means people are talking about end runs,
- 32:27SPEAKER_00: like building their own natural gas plants on site.
- 32:31SPEAKER_00: And that's fast.
- 32:32SPEAKER_00: And it's one of the worst ways you can do it.
- 32:34SPEAKER_00: That makes the AI data centers louder.
- 32:38SPEAKER_00: It increases local air pollution.
- 32:40SPEAKER_00: They're less efficient.
- 32:42SPEAKER_00: They produce more CO2, probably more methane leaks into the atmosphere.
- 32:48SPEAKER_00: Why is that happening?
- 32:49SPEAKER_00: It's happening because when you just try to get into the grid,
- 32:52SPEAKER_00: the grid operator or the transmission utility will tell you,
- 32:55SPEAKER_00: oh, it's going to be five to seven years until you have your power hookup.
- 33:01SPEAKER_00: That didn't used to be the case.
- 33:03SPEAKER_00: In the 1950s, we were growing U.S. electricity consumption by 5%, 6%, 7% per year.
- 33:10SPEAKER_00: We knew how to do this,
- 33:12SPEAKER_00: but it's been two decades of no demand growth in the U.S.
- 33:16SPEAKER_00: because of LED lights, better insulation, etc., etc., etc.
- 33:21SPEAKER_00: We just haven't had a need.
- 33:22SPEAKER_00: It's not been supply constrained.
- 33:24SPEAKER_00: It's just that,
- 33:25SPEAKER_00: you know,
- 33:25SPEAKER_00: homes since the 50s tripled in size per capita,
- 33:31SPEAKER_00: but kept using the same amount of energy per capita.
- 33:33SPEAKER_00: We just got that much more efficient.
- 33:35SPEAKER_00: So now the utilities have to change.
- 33:38SPEAKER_00: The utilities have to learn how to do things fast again.
- 33:42SPEAKER_00: That's probably going to require a change in incentives for the utilities.
- 33:46SPEAKER_00: It's going to require some relief on the permitting side of things.
- 33:53SPEAKER_00: It might require a new set of employees.
- 33:58SPEAKER_00: It might require the use of AI inside the utility companies to do that.
- 34:03SPEAKER_00: And we see signs of all of that.
- 34:05SPEAKER_00: The bipartisan permitting reform bill that's been introduced
- 34:09SPEAKER_00: that hopefully will get voted on the lame duck actually would help us a lot.
- 34:14SPEAKER_00: And so if that's the case,
- 34:15SPEAKER_00: if we manage these things to the grid,
- 34:18SPEAKER_00: they're going to draw on the cheapest source of power that varies by geography,
- 34:22SPEAKER_00: but they're going to,
- 34:22SPEAKER_00: you know, They extend solar, wind and natural gas and batteries.
- 34:29SPEAKER_00: They are also a lifeline for nuclear, for small modular nuclear reactors.
- 34:34SPEAKER_00: That industry,
- 34:35SPEAKER_00: there's a lot of optimism about it now,
- 34:36SPEAKER_00: but it's been very,
- 34:37SPEAKER_00: very, very slow And the greatest hope they have is AI data centers.
- 34:44SPEAKER_00: It's going to advance geothermal, which has popped out over the last decade.
- 34:50SPEAKER_00: It's been an overnight success in the last decade.
- 34:52SPEAKER_00: And that, again, AI is one of the greatest use cases for that.
- 34:59SPEAKER_00: And so I'm giving a talk about this later this week.
- 35:03SPEAKER_00: AI in the short term,
- 35:05SPEAKER_00: these behind the meter gas plants are not the ideal solution,
- 35:09SPEAKER_00: but there's very high value.
- 35:11SPEAKER_00: But if we do it right,
- 35:12SPEAKER_00: if we rejigger how our utilities work so they can build transmission lines
- 35:18SPEAKER_00: and new generation fast,
- 35:20SPEAKER_00: AI is going to draw on You know, the fastest, cleanest sources of power.
- 35:26SPEAKER_00: And I think that's actually probably pretty good for renewables,
- 35:29SPEAKER_00: though gas is going to be an important part of it as well.
- 35:32SPEAKER_00: And again, lifeline for new nuclear, lifeline for advanced geothermal.
- 35:36SPEAKER_00: And then beyond that, there's the craziest.
- 35:38SPEAKER_00: What do you think is more likely?
- 35:40SPEAKER_00: Massive orbital data center networks powered by solar energy or
- 36:08SPEAKER_01: A continued to build out terrestrial network powered by fusion reactors.
- 36:14SPEAKER_00: You know, we'll see.
- 36:14SPEAKER_00: I mean, fusion is coming, right?
- 36:16SPEAKER_00: It's going to work.
- 36:17SPEAKER_00: We don't know the cost yet.
- 36:19SPEAKER_00: Look, here's the way I think about it.
- 36:20SPEAKER_00: There's like four or five things that can power almost unlimited amount of AI.
- 36:25SPEAKER_00: There's solar and batteries in the world's deserts, right?
- 36:29SPEAKER_00: In the U.S., most of that desert is federal lands.
- 36:32SPEAKER_00: But if we could build it in Mexico,
- 36:34SPEAKER_00: which we can't for various regulatory reasons, or Chile, or...
- 36:41SPEAKER_00: The Middle East, you see this happening in Dubai, for instance.
- 36:46SPEAKER_00: Solar plus batteries just work, and there's an enormous amount of land.
- 36:50SPEAKER_00: Two is nuclear, either fission or fusion.
- 36:53SPEAKER_00: Those things have very, very long build times.
- 36:56SPEAKER_00: So if we can solve that problem, I think that has great potential.
- 37:01SPEAKER_00: Three is the oceans.
- 37:02SPEAKER_00: We don't talk about it,
- 37:03SPEAKER_00: but it's actually an enormous amount of energy there,
- 37:07SPEAKER_00: and permitting is just much, much easier, and you get free cooling.
- 37:11SPEAKER_00: And four is space.
- 37:12SPEAKER_00: So those are the four categories of things where you have nearly unlimited upside.
- 37:19SPEAKER_00: Space has a lot of mind share right now.
- 37:23SPEAKER_00: Launch cost is still too high, right?
- 37:26SPEAKER_00: But you can pencil it out to say that if you're launching
- 37:28SPEAKER_00: a starship every few hours, you can scale the launch costs down.
- 37:33SPEAKER_00: I start to worry about permits for launches.
- 37:37SPEAKER_00: It takes 100 to 200 starships.
- 37:40SPEAKER_00: Starship launches to get a gigawatt of compute into space,
- 37:44SPEAKER_00: and we want to build 100 gigawatts or more,
- 37:48SPEAKER_00: maybe 200 gigawatts for the next five years.
- 37:50SPEAKER_00: So you're talking about launching starships like airlines, and we're just not there yet.
- 37:57SPEAKER_00: And it's not clear from a noise standpoint,
- 38:01SPEAKER_00: et cetera, et cetera, that the world will ever allow that many launches.
- 38:04SPEAKER_00: But maybe it could happen.
- 38:06SPEAKER_01: We're going to be rail gutting those satellites from the moon into orbit.
- 38:11SPEAKER_01: There's your solution.
- 38:12SPEAKER_01: All right.
- 38:13SPEAKER_01: Listen, this has been absolutely fantastic.
- 38:16SPEAKER_01: I've already gone longer than what I promised you.
- 38:17SPEAKER_01: So thanks for doing it.
- 38:19SPEAKER_01: We'll have to have you back on when you have a free moment in your busy schedule.
- 38:22SPEAKER_00: Jim, it's a pleasure.
- 38:23SPEAKER_00: Thank you for the invitation.
- 38:25SPEAKER_00: Love talking about this stuff with you.