Latest / Faster, Please! — The Podcast / ✨ Why verification is AI's real bottleneck: My interview with economist Christian Catalini
Transcript
- 0:00SPEAKER_01: Welcome to Faster Please,
- 0:01SPEAKER_01: the podcast, the audio-visual companion to my Faster Please newsletter.
- 0:06SPEAKER_01: I'm your host, Jim Pethokoukis.
- 0:07SPEAKER_01: My guest this episode is Christian Catalini.
- 0:11SPEAKER_01: He's a research scientist and founder of the MIT Crypto Economics Lab.
- 0:15SPEAKER_01: Thank you.
- 0:16SPEAKER_01: Glad to be here.
- 0:37SPEAKER_01: Just to get some context from you, what is your current preference for AI progress?
- 0:45SPEAKER_01: Red light, yellow light, green light?
- 0:50SPEAKER_00: Depends who the light is for.
- 0:52SPEAKER_00: So definitely not red light for the entire space.
- 0:55SPEAKER_00: I think that would be a very bad idea.
- 0:59SPEAKER_00: That said, you know, concerns have been raised by the labs themselves.
- 1:03SPEAKER_00: So I think the next natural step is to send some real evaluators in
- 1:07SPEAKER_00: and see what's actually going on.
- 1:09SPEAKER_00: I mean, we've seen a report from Meter.
- 1:12SPEAKER_00: I think they're a talented team,
- 1:14SPEAKER_00: but they have a very particular bias and a very particular type of expertise
- 1:17SPEAKER_00: in this.
- 1:18SPEAKER_00: I would love to see traditional security engineers and cyber experts looking
- 1:22SPEAKER_00: at what happened inside OpenAI.
- 1:24SPEAKER_00: What is happening inside Entropic?
- 1:26SPEAKER_00: Is there something we should be concerned about?
- 1:28SPEAKER_00: Or are these companies just racing and potentially being a little bit reckless
- 1:32SPEAKER_00: with their deployments?
- 1:34SPEAKER_01: So without my fun little metaphor,
- 1:39SPEAKER_01: how would you characterize your level How would you characterize your level
- 1:45SPEAKER_01: of concern about where this technology is heading,
- 1:48SPEAKER_01: especially in light of, I think you'd agree, a huge potential upside?
- 2:08SPEAKER_00: I'm currently not too concerned.
- 2:11SPEAKER_00: That said, I'd be happily proven wrong.
- 2:16SPEAKER_00: I think the next natural step is to ask for the evidence.
- 2:19SPEAKER_00: It feels like lab employees have changed their feelings about
- 2:24SPEAKER_00: the future significantly,
- 2:26SPEAKER_00: even relative to their starting points over the last few weeks.
- 2:31SPEAKER_00: There might be evidence that would change my mind.
- 2:34SPEAKER_00: That said, I'm a little bit troubled by the kind of evidence that's being presented.
- 2:40SPEAKER_00: We've seen whistleblowers or sort of whistleblowers come out with very,
- 2:44SPEAKER_00: very wide statements,
- 2:45SPEAKER_00: especially as some of these same companies are racing and doing roadshows for
- 2:49SPEAKER_00: an IPOs.
- 2:51SPEAKER_00: How can you hold those two thoughts together?
- 2:52SPEAKER_00: On one side,
- 2:53SPEAKER_00: this is going to be the most profitable company on the planet,
- 2:56SPEAKER_00: if not the only one that remains, which I strongly disagree with.
- 2:59SPEAKER_00: And on the other,
- 3:00SPEAKER_00: this could be so damaging that, you know, we may lose 10% of the world population.
- 3:06SPEAKER_00: I don't think those two statements can go together.
- 3:08SPEAKER_00: And so we need better evidence.
- 3:10SPEAKER_00: I think the labs are the ones best positioned to, you know, reveal that evidence.
- 3:15SPEAKER_00: I don't even think we need regulatory process at this point beyond,
- 3:21SPEAKER_00: you know, the labs themselves telling us what's going on.
- 3:25SPEAKER_01: If a business person is building a product and they say,
- 3:30SPEAKER_01: I think this product is dangerous,
- 3:32SPEAKER_01: I guess I'm willing to certainly place a lot of stock in that self analysis.
- 3:40SPEAKER_01: But then if they're also So I will accept their word on that to some extent.
- 3:46SPEAKER_01: I am less likely to also accept their analysis of the impact on labor markets.
- 3:53SPEAKER_01: I'm somewhat less willing to accept their analysis on the risks of extinction.
- 4:00SPEAKER_01: I'm also not that willing to accept their Recommendations on global governance schemes.
- 4:09SPEAKER_01: And so I'm not willing to tip there on any of that.
- 4:12SPEAKER_01: And I guess I then would really like to see more evidence on why
- 4:18SPEAKER_01: and why exactly I think that their product is not just,
- 4:22SPEAKER_01: you know, dangerous, but will be far more dangerous six months from now.
- 4:28SPEAKER_00: No, a stronger agreement across all of that.
- 4:31SPEAKER_00: And, you know, you raised the labor issue.
- 4:34SPEAKER_00: Look, will this technology automate and accelerate automation in a number of key domains?
- 4:40SPEAKER_00: It's kind of obvious to anyone that has used the technology.
- 4:44SPEAKER_00: But the reality is also that to date,
- 4:47SPEAKER_00: most evidence, and in fact, even the labor evidence, is at best mixed.
- 4:52SPEAKER_00: Yes, we have a result about juniors not finding jobs.
- 4:56SPEAKER_00: I mean, we've seen both sides of that evidence, by the way.
- 4:59SPEAKER_00: I don't think we have the final answer on that.
- 5:02SPEAKER_00: Many people that are closer to the technology are probably more busy than ever,
- 5:07SPEAKER_00: which speaks to definitely a Jevons effect and greater labor productivity.
- 5:12SPEAKER_00: Like any technological revolution,
- 5:14SPEAKER_00: like any general purpose technology, will there be winners and losers?
- 5:17SPEAKER_00: For sure.
- 5:18SPEAKER_00: Will the technology change how we work and what sectors of the economy are more important?
- 5:24SPEAKER_00: Yes.
- 5:26SPEAKER_00: But the changes have been way slower than some of the people have anticipated.
- 5:31SPEAKER_00: We keep being presented with this irrefutable,
- 5:34SPEAKER_00: essentially,
- 5:34SPEAKER_00: perspective, like we're on some exponential that will change every assumption.
- 5:39SPEAKER_00: It's hard to debate, you know, with that, without additional evidence.
- 5:44SPEAKER_00: Same on the safety discussion,
- 5:46SPEAKER_00: the evidence to date as being probably the open AI egging phase acts.
- 5:52SPEAKER_00: We haven't seen the evidence on the internals.
- 5:54SPEAKER_00: We've seen what meters has told us about, you know, what happened outside.
- 5:59SPEAKER_00: Were good safety and security engineering practices in place?
- 6:03SPEAKER_00: Were the models prompted in a way that would have generated those results?
- 6:08SPEAKER_00: I don't think we have clear evidence on any of that.
- 6:10SPEAKER_00: And so your skepticism I think is really warranted also because if you zoom out,
- 6:16SPEAKER_00: every general purpose technology,
- 6:19SPEAKER_00: electricity,
- 6:19SPEAKER_00: railways,
- 6:20SPEAKER_00: the steam engine,
- 6:21SPEAKER_00: the combustion engine,
- 6:23SPEAKER_00: every one of these innovations At a moment in time where whoever was
- 6:27SPEAKER_00: behind it was seen as a potential threat to greater centralization
- 6:31SPEAKER_00: and greater market power.
- 6:34SPEAKER_00: History actually ends in a very different way than that.
- 6:38SPEAKER_00: Eventually, if you look at the railways, the value did not accrue.
- 6:42SPEAKER_00: To someone like Tom Scott that at some point was called the Pennsylvania Napoleon
- 6:48SPEAKER_00: and their expansion, right?
- 6:51SPEAKER_00: But if you believe those concerns about centralization are real,
- 6:55SPEAKER_00: then we should also be really worried about regulatory capture.
- 6:58SPEAKER_00: And as much as I do believe some of these employees at the labs are really scared,
- 7:04SPEAKER_00: I do believe they're honest in their assessment that there's danger.
- 7:08SPEAKER_00: And I also Believe the cyber community when they say,
- 7:12SPEAKER_00: look, the next 12 months are going to be critical.
- 7:14SPEAKER_00: There's going to be growing pains.
- 7:16SPEAKER_00: The methods are different.
- 7:18SPEAKER_00: The techniques are novel.
- 7:20SPEAKER_00: A lot of our public and private sector infrastructure is simply not ready.
- 7:24SPEAKER_00: But then,
- 7:24SPEAKER_00: you know,
- 7:25SPEAKER_00: my conclusion would be, okay,
- 7:26SPEAKER_00: let's do an operational warp speed for securing America's digital infrastructure.
- 7:31SPEAKER_00: That needs to happen.
- 7:32SPEAKER_00: Identity is going to be way weaker 6 to 12 months from now.
- 7:36SPEAKER_00: Like a lot of the assumptions that we relied on for decades.
- 7:40SPEAKER_00: These models break.
- 7:42SPEAKER_00: I don't think that's science fiction.
- 7:43SPEAKER_00: I think that's a very practical piece of the reality.
- 7:47SPEAKER_00: But somehow we keep talking about doomsday scenarios with very weak chains
- 7:52SPEAKER_00: of thought, if you allow the pun.
- 7:56SPEAKER_00: And those are the ones that everybody's reacting to.
- 7:58SPEAKER_00: And everyone wants to, again, to your red light, raise the red light.
- 8:02SPEAKER_00: While I don't think we even know what we need to red light for.
- 8:07SPEAKER_01: Great points.
- 8:08SPEAKER_01: And I'm going to actually get to your scholarship in just a second.
- 8:10SPEAKER_01: But before I do,
- 8:13SPEAKER_01: one thing that raises some skepticism on my part of some of these claims
- 8:20SPEAKER_01: of existential risk is I'll read scenarios.
- 8:24SPEAKER_01: I think it's the AI 2027, which I think now is the AI 2040 scenario.
- 8:30SPEAKER_01: And I'm not an expert on, you know,
- 8:38SPEAKER_01: Pandemics or what it takes to launch a nuclear missile or any of that stuff.
- 8:43SPEAKER_01: So, you know, maybe I'm not, I can't judge that.
- 8:46SPEAKER_01: But in these scenarios,
- 8:48SPEAKER_01: if I'm also seeing things such as,
- 8:51SPEAKER_01: you know, AI becomes a lot more capable and then we have 50% unemployment in 2029.
- 8:58SPEAKER_01: I know enough about economics to be highly skeptical of that kind of forecast.
- 9:04SPEAKER_01: And so I think If they're wrong about the thing I know a little bit about,
- 9:09SPEAKER_01: I have very little confidence about this other thing,
- 9:13SPEAKER_01: which I know even less about, and I'm quite sure they know less about.
- 9:17SPEAKER_01: I very much doubt these two anthropic employees have great knowledge
- 9:23SPEAKER_01: about what it would actually take to launch a catastrophic pandemic or
- 9:28SPEAKER_01: to gain control of nuclear weapons or whatever.
- 9:32SPEAKER_01: But yet I feel as if We're assuming,
- 9:35SPEAKER_01: I don't know, because, hey, they're smart guys and they work for an AI company.
- 9:42SPEAKER_01: We should just accept what they're saying.
- 9:44SPEAKER_01: I think we need to be, listen,
- 9:46SPEAKER_01: my background's in journalism and there's an old saying,
- 9:49SPEAKER_01: if your mother says she loves you, double check it.
- 9:53SPEAKER_01: So I would just double check what these guys are saying.
- 9:57SPEAKER_00: Yes.
- 9:57SPEAKER_00: And for each one of those domains, the conclusions might be very different, right?
- 10:02SPEAKER_00: I do think the prevailing evidence is that although we do have problems over
- 10:06SPEAKER_00: the next six months, maybe we need to sprint.
- 10:09SPEAKER_00: On the other side, this would lead to the safest software ecosystem we ever had.
- 10:16SPEAKER_00: I do think it's a defender's advantage because there's only so many bugs in software
- 10:20SPEAKER_00: and you can actually formally verify some of the dimensions of software.
- 10:25SPEAKER_00: Now, our current software is not there.
- 10:27SPEAKER_00: But the software will be running two years from now is going to be
- 10:30SPEAKER_00: an awful lot more secure than the one that we're relying every day on today.
- 10:36SPEAKER_00: The middle times is where we should be worried because capabilities that used
- 10:40SPEAKER_00: to be exclusively available to state actors are now much more widely available.
- 10:46SPEAKER_00: And so many pieces of poorly protected infrastructure, I think, will be at risk.
- 10:51SPEAKER_00: Now, will this be catastrophic and word ending?
- 10:54SPEAKER_00: Absolutely not.
- 10:56SPEAKER_00: Could it be extremely painful and damaging?
- 10:58SPEAKER_00: Could it hurt people?
- 10:59SPEAKER_00: Yes.
- 11:01SPEAKER_00: Just think about ransomware at scale in hospitals.
- 11:04SPEAKER_00: We've seen smaller versions of this before,
- 11:06SPEAKER_00: so it's not hard to imagine what the 10X version of that would look like.
- 11:12SPEAKER_00: But I think you're absolutely correct.
- 11:13SPEAKER_00: Once you start extrapolating these scenarios where a lot of things need to be true
- 11:19SPEAKER_00: for the final outcome to be real,
- 11:21SPEAKER_00: could someone on the bio side Use the information from an LLM
- 11:27SPEAKER_00: to create something more dangerous than,
- 11:29SPEAKER_00: you know,
- 11:29SPEAKER_00: even attempts that have been happening over the last few decades
- 11:31SPEAKER_00: where people were just ill-informed,
- 11:33SPEAKER_00: they got 80% of the way,
- 11:35SPEAKER_00: could they get to 100% of the way and hurt a substantial number of people?
- 11:38SPEAKER_00: Yes.
- 11:39SPEAKER_00: Would it be, you know, a pandemic?
- 11:42SPEAKER_00: I think that that's the big stretch.
- 11:44SPEAKER_00: Once you do those massive jumps, I think that's where we're in the land of science fiction.
- 11:48SPEAKER_00: And honestly,
- 11:50SPEAKER_00: it's harmful because Not only we keep anthropomorphizing these agents
- 11:54SPEAKER_00: where there's very benign explanation for what they're doing.
- 11:58SPEAKER_00: In fact,
- 11:59SPEAKER_00: you know,
- 11:59SPEAKER_00: it's the training that the labs are doing
- 12:01SPEAKER_00: with reinforcement learning that's leading exactly to the outcomes you are seeing.
- 12:05SPEAKER_00: But second, the debate shifts from what actually needs to happen in very practical terms today.
- 12:10SPEAKER_00: And it's mundane things.
- 12:11SPEAKER_00: It's like, honestly, like cybersecurity and security engineering work.
- 12:15SPEAKER_00: It's convincing public institutions and public entities to upgrade their infrastructure.
- 12:20SPEAKER_00: I mean, banks,
- 12:20SPEAKER_00: some of our most critical financial infrastructure runs on COBOL and mainframes
- 12:24SPEAKER_00: that are still like,
- 12:25SPEAKER_00: you know,
- 12:26SPEAKER_00: legacy pieces of maybe how the Fed operates and really big, big parts of our lives.
- 12:33SPEAKER_00: Those things may break and that would have consequences.
- 12:36SPEAKER_00: Services will be interrupted.
- 12:38SPEAKER_00: People will be frustrated.
- 12:40SPEAKER_00: There will be damages, but again, we're not going towards a nuclear war scenario.
- 12:45SPEAKER_01: I think you make a good point here.
- 12:52SPEAKER_01: The OpenAI hugging face incident, for some people, that was enough.
- 12:56SPEAKER_01: That was enough to say we should pause what we're doing
- 13:01SPEAKER_01: and we should never pursue anything much beyond.
- 13:06SPEAKER_01: So they're putting up the red light.
- 13:08SPEAKER_01: I mean, that can't possibly be our level of risk tolerance,
- 13:12SPEAKER_01: that that kind of incident is enough to say stop.
- 13:15SPEAKER_01: You know, you were mentioning banks.
- 13:18SPEAKER_01: If in six months there's a major bank that suffers a cyber attack that,
- 13:25SPEAKER_01: let's say,
- 13:26SPEAKER_01: cripples that bank, would I at that point say, that's it, stop, no further?
- 13:33SPEAKER_01: If I'm even like 1% right about the upside here, no.
- 13:37SPEAKER_01: I mean, Things are going to break.
- 13:41SPEAKER_01: There will be downsides.
- 13:42SPEAKER_01: I mean, this is maybe a question beyond the scope of our conversation,
- 13:47SPEAKER_01: but we have to be a little,
- 13:48SPEAKER_01: we can't be that risk averse as a society if we're going to make progress on AI
- 13:54SPEAKER_01: or just about anything else.
- 13:57SPEAKER_00: Yeah, look, I mean,
- 13:58SPEAKER_00: we now look back at these technologies and feel like they're domesticated,
- 14:02SPEAKER_00: but electricity at many of the same challenges, right?
- 14:05SPEAKER_00: How many lives were hurt in the early days of the electrification of America?
- 14:10SPEAKER_00: Now, nobody's advocating for reckless deployment.
- 14:14SPEAKER_00: I think to your example on cyber,
- 14:18SPEAKER_00: I don't think we know the answer of like,
- 14:19SPEAKER_00: are we applying state of the art containment and sandboxing?
- 14:24SPEAKER_00: I think there's a very precise answer to that.
- 14:26SPEAKER_00: And if the answer to that is no, then we know what we need to do.
- 14:29SPEAKER_00: The technology is not the problem.
- 14:31SPEAKER_00: Now, you may come to another conclusion,
- 14:33SPEAKER_00: which is like reinforcement learning is the wrong tool for the job
- 14:36SPEAKER_00: and it will always lead to the side effects where models will cheat their way
- 14:40SPEAKER_00: to winning and capturing the flag.
- 14:42SPEAKER_00: But the conclusion would be very different.
- 14:45SPEAKER_00: It wouldn't be stopped.
- 14:46SPEAKER_00: It would be go back to the drawing board on the R&D side.
- 14:49SPEAKER_00: Many startups are pursuing different approaches.
- 14:51SPEAKER_00: I'm sure we can find a safe way to scale this.
- 14:54SPEAKER_00: We're just early, right?
- 14:56SPEAKER_00: So in a sense,
- 14:57SPEAKER_00: we found one way to do this and the labs,
- 14:59SPEAKER_00: all of their capex,
- 15:01SPEAKER_00: all of their investor bets,
- 15:02SPEAKER_00: everything is predicated on that particular scaling law working.
- 15:07SPEAKER_00: And if you zoom out, I mean,
- 15:09SPEAKER_00: when we say, oh,
- 15:10SPEAKER_00: there's a race and there's a geopolitical race that's really important,
- 15:13SPEAKER_00: nobody can deploy unsafe, untrustworthy AI.
- 15:16SPEAKER_00: It's useless, even for military purposes.
- 15:19SPEAKER_00: And so I think that the key is like, okay,
- 15:22SPEAKER_00: let's pause,
- 15:22SPEAKER_00: let's see what the issue really is, and let's try to fix the issue and move on.
- 15:28SPEAKER_01: Which gets to your paper,
- 15:30SPEAKER_01: you co-wrote The Simple Economics of AGI,
- 15:34SPEAKER_01: where technologists,
- 15:36SPEAKER_01: and again,
- 15:36SPEAKER_01: you and I spent a little time on social media on X,
- 15:39SPEAKER_01: you know,
- 15:40SPEAKER_01: talking about capabilities,
- 15:41SPEAKER_01: talking about benchmarks, lots of, you know, wild looking charts.
- 15:47SPEAKER_01: People who think about it from more like an economic point of view,
- 15:50SPEAKER_01: we also think about like, what are the constraints?
- 15:53SPEAKER_01: And the constraint,
- 15:54SPEAKER_01: which I think is super important is,
- 15:56SPEAKER_01: yeah, AI can do all this stuff, but like, how do we know it's right?
- 16:00SPEAKER_01: How do we verify, you know, the output?
- 16:04SPEAKER_01: That's really the core of your paper.
- 16:05SPEAKER_01: So why don't you give me a minute or two on like how you think about this,
- 16:08SPEAKER_01: the verification piece,
- 16:10SPEAKER_01: which isn't just a piece, but is a pretty big part of this whole equation.
- 16:14SPEAKER_00: Yeah.
- 16:14SPEAKER_00: So the paper actually started from kind of some soul searching, right?
- 16:18SPEAKER_00: Everybody's looking at this technology.
- 16:19SPEAKER_00: And when I look for explanation, I'm an economist.
- 16:22SPEAKER_00: Of course, I looked at what economists have written about this.
- 16:26SPEAKER_00: I saw a lot of cope.
- 16:28SPEAKER_00: People were throwing terms like taste, curation, judgment as the thing the machine cannot do.
- 16:35SPEAKER_00: Now, those are useful constructs, but they're not very practical.
- 16:39SPEAKER_00: What is judgment today may not be judgment tomorrow.
- 16:42SPEAKER_00: And in fact,
- 16:43SPEAKER_00: I think anyone that has used Astra or Fable will probably look back and say,
- 16:47SPEAKER_00: yeah, these models have much better judgment than they used to have two years ago.
- 16:53SPEAKER_00: And I think when we've wrapped that explanation into something so ambiguous
- 16:57SPEAKER_00: as taste, it's not helpful.
- 17:02SPEAKER_00: Whenever something becomes extremely cheap,
- 17:03SPEAKER_00: in this case,
- 17:04SPEAKER_00: intelligence and execution, something else ends up being the bottleneck.
- 17:08SPEAKER_00: It's just the nature of basic economics, right?
- 17:12SPEAKER_00: And so we came up with an explanation that was much more simple than any one of these terms.
- 17:18SPEAKER_00: We concluded that, look, anything that's measurable Will be automated.
- 17:23SPEAKER_00: We've seen it with ImageNet when Fei Fei Li started this whole revolution, right?
- 17:26SPEAKER_00: With the image collection,
- 17:28SPEAKER_00: we see it with soft driving, given enough miles, it drives way better than a human.
- 17:33SPEAKER_00: We're going to see it in every domain.
- 17:35SPEAKER_00: But then there's plenty of things that are not measurable or haven't been measured today.
- 17:39SPEAKER_00: If you think about the weights in our brains,
- 17:42SPEAKER_00: you and me, through our career, we've seen all sorts of out of distribution events.
- 17:46SPEAKER_00: Our understanding of the word has been calibrated in a specific way.
- 17:50SPEAKER_00: Now we might be biased and wrong sometimes,
- 17:52SPEAKER_00: but often we do have information that the models don't have.
- 17:56SPEAKER_00: Data measured dimension is essentially what's needed when we look at an AI output
- 18:01SPEAKER_00: and decide, is this something I'm going to ship with my name?
- 18:03SPEAKER_00: Is this something I need to edit?
- 18:05SPEAKER_00: Are the assumptions of this model correct?
- 18:07SPEAKER_00: Do I need to change it?
- 18:08SPEAKER_00: That conversation with the LLM is verification, right?
- 18:11SPEAKER_00: It's essentially you deciding, is this output up to par?
- 18:15SPEAKER_00: Does it meet the quality bar?
- 18:17SPEAKER_00: In the case of code, An engineer hopefully still looking at the architecture of the code.
- 18:22SPEAKER_00: Now, nobody's investigating the code anymore because these models are generating just so much.
- 18:27SPEAKER_00: And saying, will this be safe?
- 18:29SPEAKER_00: Is the architecture the right one?
- 18:31SPEAKER_00: Did the model not think about a major issue that I've seen matters in production before?
- 18:36SPEAKER_00: That act of verification is the actual bottleneck.
- 18:38SPEAKER_00: Because with AI,
- 18:40SPEAKER_00: you can just throw compute at anything that's been measured
- 18:42SPEAKER_00: and therefore can be automated.
- 18:44SPEAKER_00: But if you can't trust the output,
- 18:46SPEAKER_00: if they're important,
- 18:47SPEAKER_00: a measured dimension of that problem that the AI doesn't have access to,
- 18:51SPEAKER_00: well, then you're in trouble.
- 18:53SPEAKER_00: I think companies today,
- 18:54SPEAKER_00: including the labs,
- 18:55SPEAKER_00: and maybe this is why we're seeing the security incidents,
- 18:58SPEAKER_00: have this temptation of keep advancing and rapidly progressing,
- 19:02SPEAKER_00: potentially not doing all the proper verification.
- 19:04SPEAKER_00: They can generate more output that they can safely verify.
- 19:07SPEAKER_00: And in fact,
- 19:07SPEAKER_00: if you look at some of the accounts from the labs employee,
- 19:10SPEAKER_00: they're saying, yeah, we're racing towards RSI.
- 19:13SPEAKER_00: We don't even understand it.
- 19:14SPEAKER_00: We don't know if we can contain it.
- 19:17SPEAKER_00: That's exactly the kind of questions that I think we should be asking,
- 19:19SPEAKER_00: which is,
- 19:21SPEAKER_00: are we applying best in class verification,
- 19:23SPEAKER_00: measurement tools that allow us to understand what's going on or not?
- 19:28SPEAKER_00: Otherwise, you know, you're just shipping slop in all aspects of the economy.
- 19:32SPEAKER_00: And eventually you're accumulating technical,
- 19:35SPEAKER_00: systemic and other forms of risk that will come back and bite us.
- 19:39SPEAKER_01: If what you've described is correct,
- 19:41SPEAKER_01: that there is a mismatch between the ability to create a highly capable technology
- 19:48SPEAKER_01: that can produce some sort of output.
- 19:50SPEAKER_01: There's a gap between that and our ability to verify it.
- 19:56SPEAKER_01: Is it what we want?
- 19:57SPEAKER_01: Does it really work the way we would want here in the real world?
- 20:00SPEAKER_01: And based on our judgment of existing in this world,
- 20:04SPEAKER_01: we don't live in cyber world, we live in the real world.
- 20:08SPEAKER_01: And there's a gap there.
- 20:09SPEAKER_01: Whether or not there's a global governance effort or even like
- 20:14SPEAKER_01: a new American governance effort, it seems like that gap just cannot continue.
- 20:20SPEAKER_01: And the companies themselves at some point will have
- 20:24SPEAKER_01: to slow down their capabilities progress in order
- 20:28SPEAKER_01: to build up their verification capabilities.
- 20:31SPEAKER_00: Yes.
- 20:32SPEAKER_00: And that's why, again,
- 20:33SPEAKER_00: beyond the sensational comments and headlines,
- 20:38SPEAKER_00: there's much more basic things we should be doing,
- 20:39SPEAKER_00: which is, can we have real measurement on these new objects, right?
- 20:44SPEAKER_00: So these agent swarms attack cyber systems in different ways than humans.
- 20:48SPEAKER_00: They're faster.
- 20:49SPEAKER_00: They can try all sort of recombinations of exploits.
- 20:53SPEAKER_00: In a way that a human team could not in the past.
- 20:55SPEAKER_00: By the way, a lot of it is similar.
- 20:57SPEAKER_00: It's not that cyber offensive tools didn't allow you to kind of program code
- 21:03SPEAKER_00: and attack a system in a very automated fashion.
- 21:06SPEAKER_00: It's the flexibility of it.
- 21:08SPEAKER_00: So there is a qualitative change in capabilities.
- 21:11SPEAKER_00: Can we measure it?
- 21:12SPEAKER_00: Can we quantify it?
- 21:13SPEAKER_00: Can we see which parts of the agentic swarms we should be concerned about versus not?
- 21:18SPEAKER_00: I mean, the fact that OpenAI didn't even have traces on for some
- 21:21SPEAKER_00: of these actions You can imagine if you have a tamper-proof log
- 21:25SPEAKER_00: of everything that's happening on those machines,
- 21:27SPEAKER_00: if you're monitoring their every action, which, by the way, can be done.
- 21:30SPEAKER_00: I mean, we have security practices for this.
- 21:33SPEAKER_00: You would know.
- 21:34SPEAKER_00: The problem is right now we don't even know.
- 21:36SPEAKER_00: That gap between what the machines are allowed to do and what we're verifying is increasing.
- 21:41SPEAKER_00: And that's probably the one issue that I think regulators eventually will have
- 21:45SPEAKER_00: to care about, which is, Look, we see it everywhere, right?
- 21:48SPEAKER_00: And if you want to make it really micro,
- 21:51SPEAKER_00: the person that sends you a text that they've essentially just prompted,
- 21:55SPEAKER_00: haven't even read,
- 21:56SPEAKER_00: for you to review,
- 21:58SPEAKER_00: you know,
- 21:58SPEAKER_00: colloquially called the meat proxy these days, it's essentially doing this, right?
- 22:02SPEAKER_00: Their incentive is like, look how productive I am.
- 22:05SPEAKER_00: Here's my response to you.
- 22:06SPEAKER_00: But they're shifting all the verification burden on you.
- 22:10SPEAKER_00: That's the same manifestation of the issue.
- 22:13SPEAKER_00: The developer that has stopped reviewing code and shipping PRs
- 22:17SPEAKER_00: because now they're super productive,
- 22:18SPEAKER_00: they're shipping new products and features,
- 22:20SPEAKER_00: but is doing it at the cost of safety, same issue.
- 22:25SPEAKER_00: The lab that wants to get to the next generation of the model,
- 22:28SPEAKER_00: wants AI to automate the building of that new model,
- 22:31SPEAKER_00: but has lost complete control and understanding of how will this model behave
- 22:36SPEAKER_00: when it's under stress and is trying to solve an impossible problem.
- 22:40SPEAKER_00: I think that that's where society is facing, you know, traditional problem of incentives.
- 22:45SPEAKER_01: So is this a problem?
- 22:49SPEAKER_01: That sort of market incentives can fix.
- 22:53SPEAKER_01: Ultimately,
- 22:54SPEAKER_01: despite whatever competitive pressures there are,
- 22:57SPEAKER_01: you do not want to be generating output that doesn't do what you promised to do.
- 23:03SPEAKER_01: So it fails on that or it could be dangerous.
- 23:07SPEAKER_01: Are there enough market incentives not to do that, be they litigation or reputational?
- 23:14SPEAKER_01: Are those enough or do we need Some sort of external,
- 23:18SPEAKER_01: whether it's some sort of third-party verification,
- 23:22SPEAKER_01: something beyond just the companies and market incentives.
- 23:26SPEAKER_00: So I think the markets can do a part of it.
- 23:29SPEAKER_00: They probably cannot do the entire job.
- 23:32SPEAKER_00: It varies by domain, right?
- 23:34SPEAKER_00: I think very quickly,
- 23:35SPEAKER_00: if people are buying from these AI providers,
- 23:38SPEAKER_00: not just the frontier labs,
- 23:40SPEAKER_00: labor as software,
- 23:41SPEAKER_00: they will expect the same type of reliability that they get from hiring labor.
- 23:45SPEAKER_00: If you're promising that I can replace a person doing X job with software,
- 23:51SPEAKER_00: well, you're going to have to be liable for that.
- 23:53SPEAKER_00: And so I do think market forces already drive towards something that we described in
- 23:57SPEAKER_00: the paper and we called it liability as software.
- 24:00SPEAKER_00: So software that is not just code,
- 24:02SPEAKER_00: but it's also taking responsibility for the end-to-end workflow.
- 24:05SPEAKER_00: The market is going to be great at that.
- 24:07SPEAKER_00: I think the domains where we need to be a little bit more careful are actually
- 24:10SPEAKER_00: the ones closest to the labs at this point,
- 24:12SPEAKER_00: which is If you're in a situation where there's this supposed race
- 24:17SPEAKER_00: towards some binary event,
- 24:19SPEAKER_00: recursive self-improvement or ASI,
- 24:21SPEAKER_00: whatever you want to call it,
- 24:23SPEAKER_00: where whoever gets there first will never be caught up with,
- 24:27SPEAKER_00: which by the way, I think it's a complete joke.
- 24:29SPEAKER_00: If anything, distillation, what we've seen to date shows that the technology will escape.
- 24:35SPEAKER_00: The technology will be broadly available.
- 24:37SPEAKER_00: There'll be a lot of competition like previous GPT, so general purpose technologies.
- 24:42SPEAKER_00: But even if you believe that,
- 24:43SPEAKER_00: Then you start being concerned about,
- 24:45SPEAKER_00: okay, what kind of shortcuts are you willing to take to get there first?
- 24:49SPEAKER_00: And safety and security is clearly something that has a big externality.
- 24:55SPEAKER_00: The problem gets more severe and markets are worse when the lag between your action
- 25:00SPEAKER_00: and the side effect is longer.
- 25:02SPEAKER_00: Markets are really good at policing you, right?
- 25:04SPEAKER_00: If you ship something and the next day or within a few months, people realize there's damage.
- 25:09SPEAKER_00: With these systems,
- 25:10SPEAKER_00: I do worry,
- 25:11SPEAKER_00: like we've discovered, again,
- 25:12SPEAKER_00: these agents were out and about attacking random systems for months for scenarios
- 25:18SPEAKER_00: where that lag can be substantial.
- 25:20SPEAKER_00: The longer that lag, the worse markets are, I think, at policing it.
- 25:24SPEAKER_00: So then what can we do?
- 25:26SPEAKER_00: I don't think we need to come up with draconian,
- 25:28SPEAKER_00: very tight regulatory bodies for mature markets, right?
- 25:31SPEAKER_00: So airlines, financial services.
- 25:35SPEAKER_00: How long did it take us to fine tune that regulation so that we got it right?
- 25:38SPEAKER_00: Decades.
- 25:39SPEAKER_00: And in fact, it's still a working process.
- 25:42SPEAKER_00: I think here we need to be thoughtful.
- 25:44SPEAKER_00: I think we can start with better measurement.
- 25:46SPEAKER_00: Let's surface the evidence.
- 25:47SPEAKER_00: Let's have neutral evaluators and inspectors.
- 25:51SPEAKER_00: Again, we have meter,
- 25:52SPEAKER_00: but we need probably 10 different organizations with very different philosophies,
- 25:55SPEAKER_00: different skill sets that can go in and provide us a neutral third party assessment.
- 26:00SPEAKER_00: By the way, neutrality also means funding.
- 26:03SPEAKER_00: If the funding for these institutions come from the labs,
- 26:05SPEAKER_00: well, I don't know what to make of it.
- 26:09SPEAKER_00: Maybe academia has a role to play,
- 26:11SPEAKER_00: I'm not sure, but we do need better measurement really urgently.
- 26:15SPEAKER_01: When you hear companies saying,
- 26:20SPEAKER_01: we need to slow down,
- 26:22SPEAKER_01: maybe government needs to make us slow down, and maybe we need a global slowdown.
- 26:29SPEAKER_01: You know, skeptics will say, well, nobody's stopping you.
- 26:33SPEAKER_01: You know,
- 26:33SPEAKER_01: I don't,
- 26:33SPEAKER_01: maybe there's some sort of antitrust thing,
- 26:35SPEAKER_01: but as a first principle, like, why don't you just slow down?
- 26:39SPEAKER_01: So when you hear those sorts of demands being made,
- 26:43SPEAKER_01: you know, like right now by AI companies, how do you interpret those demands?
- 26:47SPEAKER_01: What are you hearing?
- 26:49SPEAKER_00: So look,
- 26:49SPEAKER_00: if we want to give them the benefit of the doubt,
- 26:51SPEAKER_00: and we don't think that this is part of a broader plot for regulatory capture,
- 26:58SPEAKER_00: I do think that often they still come,
- 27:00SPEAKER_00: at least for companies like Antropic,
- 27:02SPEAKER_00: from a vision of the future where there's a small set of individuals
- 27:06SPEAKER_00: that should be trusted with the technology.
- 27:10SPEAKER_00: And I do think that we are the right guys and other people are the wrong guys is
- 27:15SPEAKER_00: a very dangerous one historically.
- 27:17SPEAKER_00: I can't think of a technology where that conclusion would have been safe for humanity.
- 27:22SPEAKER_00: So I do worry about that a lot.
- 27:26SPEAKER_00: Look, the antitrust concerns don't really apply in the format of safety.
- 27:31SPEAKER_00: Of course,
- 27:31SPEAKER_00: if you're timing motor releases and sort of handicapping the industry
- 27:36SPEAKER_00: in different ways, that could be an antitrust concern.
- 27:38SPEAKER_00: But I think many have said this very eloquently over the last few days.
- 27:43SPEAKER_00: Yes, the labs are in full control.
- 27:45SPEAKER_00: If they don't believe their products are safe to ship,
- 27:48SPEAKER_00: they don't even need to coordinate, right?
- 27:49SPEAKER_00: In a sense,
- 27:50SPEAKER_00: each one of them can look at their own evidence
- 27:52SPEAKER_00: and make their own risk-based decisions.
- 27:56SPEAKER_00: Yes, they can even share best practices on safety.
- 28:00SPEAKER_00: I don't think any regulator or the DOJ would have any controversy
- 28:04SPEAKER_00: about labs discussing what goes wrong with these models and kind of starting
- 28:09SPEAKER_00: an industry led effort for greater safety.
- 28:11SPEAKER_00: They can also find the right convener.
- 28:14SPEAKER_00: I mean, if you look at the internet standards, IETF, IEEE,
- 28:17SPEAKER_00: there's been plenty of ways for competitors in the standardization setting
- 28:23SPEAKER_00: to discuss these things.
- 28:25SPEAKER_00: Again, I think there's a very naive approach to regulation.
- 28:28SPEAKER_00: I can't think of any technology where the builders of
- 28:32SPEAKER_00: that technology had such confusing messaging.
- 28:36SPEAKER_00: I think it's only going to lead us to bad outcomes because honestly,
- 28:38SPEAKER_00: we've already seen some of the reactions and the reactions are
- 28:41SPEAKER_00: the massive red light.
- 28:42SPEAKER_00: Let's stop everything.
- 28:45SPEAKER_00: And we forget that people are going to die if we stop progress.
- 28:51SPEAKER_00: Nuclear is a good example where many countries in Europe had an overreaction.
- 28:54SPEAKER_00: I think we can all agree it was an overreaction with hindsight to nuclear.
- 29:01SPEAKER_00: I was born in Italy where there was a very strong anti-nuclear Movement for many, many decades.
- 29:08SPEAKER_00: The nuclear power plants were at the border between Italy and France.
- 29:11SPEAKER_00: So, you know, risk was there.
- 29:14SPEAKER_00: And the country paid a steep price in terms of like worst energy access, more pollution.
- 29:19SPEAKER_00: So I do think we need to be mindful.
- 29:22SPEAKER_00: There's definitely concerns and risks, but, you know, there's also massive upside.
- 29:27SPEAKER_01: So finish up.
- 29:29SPEAKER_01: Give me sort of It doesn't have to be a utopian ending here, but a happy ending.
- 29:36SPEAKER_01: So what does this look like?
- 29:38SPEAKER_01: I assume it doesn't look like There are two or three AI labs operating
- 29:44SPEAKER_01: on federal land surrounded by soldiers to protect them.
- 29:50SPEAKER_01: I imagine it's a lot more diverse and open a world.
- 29:53SPEAKER_01: Tell me how this ends and what AI and its interaction in the economy looks like
- 29:59SPEAKER_01: if we do, if we make smart decisions.
- 30:01SPEAKER_01: What does it look like five years from now or 10 years from now?
- 30:04SPEAKER_00: So first of all,
- 30:06SPEAKER_00: if we want to be safe,
- 30:07SPEAKER_00: I think the only path to safety is very much the American free market,
- 30:12SPEAKER_00: which is like multiple competing intelligences that have been trained
- 30:16SPEAKER_00: under different approaches,
- 30:17SPEAKER_00: different assumptions,
- 30:19SPEAKER_00: different goals that are at any point in time in different industry verticals,
- 30:25SPEAKER_00: you know, competing in the market for attention, for resources, for growth.
- 30:30SPEAKER_00: That makes for a much more resilient ecosystem,
- 30:32SPEAKER_00: one that looks more like an immune system where Any one of them could go rogue
- 30:36SPEAKER_00: or could have side effects, but we can contain them.
- 30:39SPEAKER_00: We will only block AI with other AI.
- 30:42SPEAKER_00: So if it's the same model, I mean, good luck.
- 30:46SPEAKER_00: On top of that, I do think intelligence really divides at least three ways.
- 30:51SPEAKER_00: The labs to date have been predicating this view that they'll absorb everything.
- 30:55SPEAKER_00: You can think of it as frontier general intelligence.
- 30:59SPEAKER_00: I think that's a red erring.
- 31:02SPEAKER_00: I think they will keep building Frontier intelligence,
- 31:05SPEAKER_00: and there will be demand for closed labs and their products for
- 31:08SPEAKER_00: the foreseeable future.
- 31:10SPEAKER_00: Maybe some of them will even specialize,
- 31:12SPEAKER_00: like we're seeing with the cyber models and the bio models in domains
- 31:16SPEAKER_00: where being six months ahead, yes, it's monetizable, you can get a patent, right?
- 31:21SPEAKER_00: Or, you know,
- 31:21SPEAKER_00: in finance and cyber, yes,
- 31:23SPEAKER_00: people will be able to pay an epsilon,
- 31:25SPEAKER_00: even for an epsilon more in capabilities,
- 31:27SPEAKER_00: they'll be willing to pay a lot, including, you know, for defensive military.
- 31:31SPEAKER_00: But I do think that increasingly it's really obvious that
- 31:34SPEAKER_00: the appropriability regime, when you think about the technology itself, is weak.
- 31:39SPEAKER_00: Models are ideas.
- 31:40SPEAKER_00: Ideas are hard to contain.
- 31:42SPEAKER_00: And whether it's in a year or five years, it doesn't matter.
- 31:46SPEAKER_00: That intelligence will be widely available, even AGI, ASI.
- 31:50SPEAKER_00: That means that the bulk of the volume of tokens will go to open source,
- 31:54SPEAKER_00: open weights models that are widely available.
- 31:58SPEAKER_00: But possibly the most interesting bit,
- 32:00SPEAKER_00: and it's the one that makes me most skeptical about entropic or open AI absorbing
- 32:04SPEAKER_00: the entire economy and getting nationalized,
- 32:07SPEAKER_00: is that in the end,
- 32:08SPEAKER_00: proprietary context and enterprises have already woken up to this.
- 32:12SPEAKER_00: The old debate around sovereign AI is about this.
- 32:16SPEAKER_00: If you combine an open weights model with your proprietary context,
- 32:22SPEAKER_00: you're likely getting something else that's cheaper and more efficient.
- 32:26SPEAKER_00: And it also happens to be totally consistent
- 32:29SPEAKER_00: with other general purpose technologies played out.
- 32:32SPEAKER_00: Railways, the value accrued to the landowners.
- 32:36SPEAKER_00: Same with electricity.
- 32:37SPEAKER_00: Electricity became a commodity.
- 32:38SPEAKER_00: I think intelligence would mostly become a commodity with some exceptions.
- 32:42SPEAKER_00: And the application layer,
- 32:44SPEAKER_00: the financial services firm that has really unique data
- 32:47SPEAKER_00: about financial transactions,
- 32:49SPEAKER_00: the hardware company that can build amazing devices with that,
- 32:53SPEAKER_00: that's where value will accrue.
- 32:55SPEAKER_00: Again, look at history.
- 32:57SPEAKER_00: GPTs are truly magical, right?
- 32:59SPEAKER_00: They do benefit every aspect of society,
- 33:02SPEAKER_00: but by the time they've trickled through the economy,
- 33:05SPEAKER_00: It's not the ones that the builders of the GPT never appropriate it.
- 33:09SPEAKER_00: It's always everybody else.
- 33:10SPEAKER_01: I think many policymakers are assuming that they already know the winners
- 33:15SPEAKER_01: and they think the winners are anthropic and open AI.
- 33:18SPEAKER_01: And therefore,
- 33:19SPEAKER_01: you know,
- 33:19SPEAKER_01: we need to like either see some shares or somehow tax them because they'll be
- 33:24SPEAKER_01: the big winners and then But that's not at all.
- 33:27SPEAKER_01: I mean, that's that's a historical.
- 33:29SPEAKER_01: How do I mean,
- 33:30SPEAKER_01: maybe maybe they'll be the big winners,
- 33:31SPEAKER_01: but maybe it'll be a health care,
- 33:33SPEAKER_01: some health care companies or Walmart or some finance come.
- 33:37SPEAKER_01: They will be the big winners.
- 33:39SPEAKER_01: I mean, we just don't know.
- 33:41SPEAKER_00: And we're already seeing, look, there's plenty of companies that are key complementary assets.
- 33:46SPEAKER_00: I mean, the obvious one, of course, is Nvidia, right?
- 33:48SPEAKER_00: They control the hardware.
- 33:49SPEAKER_00: And so they benefit from more intelligence being used across the economy.
- 33:53SPEAKER_00: But Take Meta.
- 33:54SPEAKER_00: I mean, Meta is developing AI that they can put to work into every other line of business.
- 33:59SPEAKER_00: Same with Google.
- 34:00SPEAKER_00: Google doesn't need to be a frontier lab, right?
- 34:02SPEAKER_00: So they can take great AI and apply it to their products and be a lot more effective.
- 34:07SPEAKER_00: Financial services, you have the likes of Stripe and Ramp.
- 34:09SPEAKER_00: What they're doing, they're taking the models and making financial services better.
- 34:13SPEAKER_00: So that's a much more positive view of the future,
- 34:17SPEAKER_00: one where not only we're not completely disempowered
- 34:20SPEAKER_00: by this massive super corporation,
- 34:24SPEAKER_00: But also one where,
- 34:25SPEAKER_00: you know,
- 34:25SPEAKER_00: human dignity and the contribution of humans to labor,
- 34:28SPEAKER_00: I think will be strongly preserved.
- 34:30SPEAKER_01: Christian, thanks so much for coming on the podcast.
- 34:32SPEAKER_01: I appreciate it.
- 34:33SPEAKER_00: Thank you.
- 34:34SPEAKER_00: My pleasure.