Latest / Now Shipping: A Mind the Product podcast / We read Dario Amodei’s 3,800-word essay so you don’t have to

Transcript

We read Dario Amodei’s 3,800-word essay so you don’t have to · Now Shipping: A Mind the Product podcast

• 0:00
  SPEAKER_01: Three CEOs agreeing that the industry is moving too fast on some random Saturday in an essay and two tweets.
• 0:06
  SPEAKER_01: That's a very different thing than three companies that actually start shipping slower.
• 0:11
  SPEAKER_01: So watch the releases, not just the essays.
• 0:16
  SPEAKER_01: Today we're gonna dive deep into what I think is a really important story that's developed over the last few days.
• 0:22
  SPEAKER_01: Sort of a combination of a couple stories, really, but they all tie together.
• 0:26
  SPEAKER_01: And it started with some social posts online, which led to a barrage of national news stories and all the major media outlets, and then an essay posted over the weekend from Anthropic CEO.
• 0:38
  SPEAKER_01: And now politicians are involved, including President Trump, and it has all to do with the pace of AI frontier models and the safety of humanity.
• 0:48
  SPEAKER_01: All right, I should really back up here a bit.
• 0:51
  SPEAKER_01: Last week, Jacob Cox, and he's a researcher, or at least was a researcher at Anthropic.
• 0:57
  SPEAKER_01: He also spent time in the past at Open.
• 0:59
  SPEAKER_01: He posted this on X shortly after resigning from his role at Anthropic.
• 1:04
  SPEAKER_01: And I'll read it right here.
• 1:06
  SPEAKER_01: From Jacob.
• 1:07
  SPEAKER_01: I resigned from Anthropic today.
• 1:09
  SPEAKER_01: I spent the last three years doing pre-training research at both OpenAI and Anthropic, and neither company is acting responsibly.
• 1:17
  SPEAKER_01: They're racing straight to self-improving superintelligence and gambling with our lives.
• 1:22
  SPEAKER_01: The people building AI earnestly believe it could kill us all by the end of the decade.
• 1:27
  SPEAKER_01: This is not a marketing stunt.
• 1:29
  SPEAKER_01: If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible, but I hear the same people express their fear privately.
• 1:38
  SPEAKER_01: No other human activity poses this level of danger.
• 1:43
  SPEAKER_01: To go so far as to say that the people he worked for are literally gambling with our lives, racing towards self-improving AI without the safety work to back it up.
• 1:53
  SPEAKER_01: They're bold words for sure.
• 1:54
  SPEAKER_01: And that thread now has been viewed more than 150 million times.
• 1:59
  SPEAKER_01: And Coxin didn't just stop there.
• 2:02
  SPEAKER_01: He sat down with the BBC's Laura Cussenberg and said, quote, I believe if we don't slow down at the current rate of progress, there's a strong chance that we could all die in the immediate future.
• 2:14
  SPEAKER_01: He said, People still inside the Frontier Labs are, in his words, genuinely frightened of what they're building.
• 2:20
  SPEAKER_01: And here's the thing that actually stuck with me.
• 2:23
  SPEAKER_01: He said the people inside these labs use words like crunch time and endgame to describe where things are headed.
• 2:30
  SPEAKER_01: And they seem to mean it, literally.
• 2:32
  SPEAKER_01: Now, other outlets reporting on his comments or those that interviewed him, they actually quoted him saying things could spiral out of control even by the end of next year.
• 2:42
  SPEAKER_01: Now, he wasn't the only one to come out and say these types of things.
• 2:47
  SPEAKER_01: Coxson had actually quite a few people backing him up on this.
• 2:50
  SPEAKER_01: One in particular was Evan Hubinger, an alignment scientist who's still at Anthropic right now, and he's working on this stuff every single day.
• 2:58
  SPEAKER_01: He came out and agreed with Coxson and he went even a bit further.
• 3:02
  SPEAKER_01: He said there's a greater than 10% chance that AI could cause human extinction within a decade.
• 3:10
  SPEAKER_01: Human extinction.
• 3:11
  SPEAKER_01: Now, before we get to doom and gloom here, although I guess we can't get more doom and gloom than human extinction, there definitely has been another side to all of this.
• 3:20
  SPEAKER_01: There were security researchers who came out publicly with a counterpoint of sorts, people like Artem Dienberg and Niddy Argual, who's the CPO at Hacker One.
• 3:30
  SPEAKER_01: Both of them sort of pushed back on the doom framing, not necessarily on the facts that Jacob was bringing up, but their take is that the Hugging Face incident specifically, which that's a big one that people are talking about ever since that came out.
• 3:43
  SPEAKER_01: And of course, we reported on that last week and a few weeks back.
• 3:47
  SPEAKER_01: But ever since, you know, that incident sort of came to light, there have been a lot of people talking about a cybersecurity governance problem in particular, which is a real problem that we're facing right now.
• 3:59
  SPEAKER_01: But they say there are known playbooks to address that problem.
• 4:02
  SPEAKER_01: In their opinion, there isn't necessarily evidence of some uncontrollable superintelligence plotting against us.
• 4:09
  SPEAKER_01: But regardless, even if there were, you know, supporters and detractors, this was the temperature going into this past weekend.
• 4:18
  SPEAKER_01: And then, as if things couldn't escalate anymore, last Saturday, Dario Amodai, the CEO of Anthropic, published a 3,800-word essay titled We Must Pace the Frontier.
• 4:31
  SPEAKER_01: And I want to be very specific about what he actually said, because of course we've seen the whole like AI CEO warns about AI before.
• 4:39
  SPEAKER_01: You know, some people think that, you know, right before Anthropic's IPO, there might be some strategy to talking about just how powerful their AI is.
• 4:48
  SPEAKER_01: But this admittedly did seem different.
• 4:51
  SPEAKER_01: So in this essay, and I encourage you to read it all.
• 4:53
  SPEAKER_01: In fact, we'll link to that essay right here in the show notes.
• 4:56
  SPEAKER_01: But Amade's argument isn't to necessarily stop AI frontier development, but to slow down on purpose.
• 5:04
  SPEAKER_01: He's asking Frontier Labs to intentionally hold back capability gains by something like maybe a year, maybe two years, but enough time to let safety and alignment research actually catch up on what these systems can already do.
• 5:19
  SPEAKER_01: And he mentions two specific triggers.
• 5:22
  SPEAKER_01: One is recursive self-improvement.
• 5:24
  SPEAKER_01: Now, it's a fancy way of saying AI systems are increasingly being used to build the next AI systems.
• 5:32
  SPEAKER_01: The tools are starting to essentially build their own replacements.
• 5:35
  SPEAKER_01: And he says in this essay, this is starting to happen across the industry, including at Anthropic.
• 5:41
  SPEAKER_01: Now, the second trigger, it goes back to that hugging face incident.
• 5:45
  SPEAKER_01: He warns that left unaddressed, the same pattern could produce what he calls persistent bots.
• 5:52
  SPEAKER_01: Autonomous systems that don't just misbehave once they get shut down, but they stick around and keep operating.
• 5:58
  SPEAKER_01: Capable, he says, of hundreds of billions of dollars in damages within just six to twelve months.
• 6:04
  SPEAKER_01: Now, what do you actually do about it?
• 6:06
  SPEAKER_01: Now, he's proposing a three-step plan that at least Anthropic is already acting on, and he's encouraging every AI frontier model to act on.
• 6:16
  SPEAKER_01: Now, first, starting now, Anthropic is giving independent third-party evaluators permanent employee-level access inside the company and the right to publish what they find without Anthropic editing it first.
• 6:29
  SPEAKER_01: Now, he also calls for coordination among AI companies in democratic countries on common safety standards, which he admits this kind of coordination among competitors, it could require a government antitrust waiver.
• 6:41
  SPEAKER_01: So it's not exactly so easy or straightforward, but this is part two.
• 6:47
  SPEAKER_01: Now, finally, part three, and it might be the hardest part.
• 6:50
  SPEAKER_01: There needs to be some sort of coordination that includes authoritarian governments too, all while acknowledging there's no real way to verify whether anyone's holding up their end of the bargain, especially when you start involving authoritarian governments.
• 7:04
  SPEAKER_01: So this is a lot to lay down in an essay on a Saturday.
• 7:08
  SPEAKER_01: And to be fair, not everybody seems to be a supporter here.
• 7:12
  SPEAKER_01: I think the core problem that people are raising is that independent evaluators that Anthropic invites in, they'll still be paid by Anthropic.
• 7:21
  SPEAKER_01: Now, without protected publication rights and direct access to regulators, some critics say that an independent evaluator can sort of become, you know, just a well-paid consultant with a badge, sort of a thing.
• 7:34
  SPEAKER_01: Now, on top of that, pacing isn't really a defined thing.
• 7:38
  SPEAKER_01: I mean, there's no agreed upon measure of what counts as capability advanced.
• 7:42
  SPEAKER_01: So limiting one path just pushes labs towards another.
• 7:46
  SPEAKER_01: And the coordination piece also has this kind of weird side effect.
• 7:51
  SPEAKER_01: Safety standards written by the biggest labs, they tend to reflect exactly what the biggest labs can already afford to build.
• 7:58
  SPEAKER_01: Now that's sort of the textbook definition of regulatory capture, where the incumbents help write the rules that happen to lock out whoever's trying to catch up.
• 8:07
  SPEAKER_01: I don't know if you've ever watched a dominant platform voluntarily propose the exact regulation that conveniently cements its own lead.
• 8:15
  SPEAKER_01: This kind of sounds familiar.
• 8:17
  SPEAKER_01: But he did have his supporters too, and his supporters happened to surprise a lot of people.
• 8:23
  SPEAKER_01: The same day the essay came out, Sam Altman of OpenAI, Anthropic's biggest rival, responded.
• 8:29
  SPEAKER_01: Now, knowing the history between those two, some might have expected Altman to take a contradictory stance, but he didn't.
• 8:37
  SPEAKER_01: In fact, he said, and I quote, I agree with Dario that we need to pace the frontier.
• 8:42
  SPEAKER_01: This has been a primary topic of discussion that we've had at OpenAI in recent weeks.
• 8:46
  SPEAKER_01: And then he committed to bring in his own independent evaluators with the same employee level access.
• 8:52
  SPEAKER_01: And then there's Elon Musk, who runs SpaceX AI, and he responded with three words to start.
• 8:59
  SPEAKER_01: Dario is right.
• 9:01
  SPEAKER_01: Now, look, this isn't Musk's first rodeo with a pause letter of sorts.
• 9:05
  SPEAKER_01: Back in March of 2023, he signed a very different open letter, the Future of Life Institute's Pause Giant AI Experiments Letter, calling for a six-month pause on training any AI system more powerful than GPT-4.
• 9:19
  SPEAKER_01: Now, that didn't end up happening.
• 9:21
  SPEAKER_01: Nobody really paused.
• 9:22
  SPEAKER_01: Then a few months later, Musk went on and founded what became XAI, which is now SpaceXAI.
• 9:27
  SPEAKER_01: But this is a pretty rare moment.
• 9:30
  SPEAKER_01: Three competing Frontier Lab CEOs, men who spend most of their public lives disagreeing with each other, sometimes viciously, about basically everything.
• 9:39
  SPEAKER_01: But here they are agreeing on the same day in public that the industry itself needs to slow down.
• 9:46
  SPEAKER_01: Now, the political world noticed very fast, within about 48 hours, more than 20 lawmakers had weighed in, including Illinois Governor J.B.
• 9:54
  SPEAKER_01: Pritzker and Senator Bernie Sanders, among them.
• 9:57
  SPEAKER_01: They introduced or backed new proposals aimed at regulating AI development.
• 10:02
  SPEAKER_01: And then President Trump enters the chat.
• 10:06
  SPEAKER_01: He posted on his own social platform: the people that say AI is going to destroy the world and that data centers are bad for your neighborhood are the same people that said just a short time ago that the world would be extinguished by climate change.
• 10:19
  SPEAKER_01: And then he basically said the threat of AI destroying the world is just a hoax.
• 10:24
  SPEAKER_01: So to say that the early part of this week has been buzzing in tech and AI circles, it'd be a major understatement.
• 10:31
  SPEAKER_01: But what does it all mean for you as a product person?
• 10:35
  SPEAKER_01: I think it means a few things here.
• 10:37
  SPEAKER_01: First, evanthropic, SpaceX AI, OpenAI, if they're all talking about intentionally holding back capability jumps, definitely plan for that in your roadmap decisions, right?
• 10:48
  SPEAKER_01: Many of us assume that the next models are always faster, cheaper, more capable.
• 10:53
  SPEAKER_01: And we also assume that they're usually just a few months or maybe even a few weeks away.
• 10:58
  SPEAKER_01: Well, I don't think we can assume that anymore.
• 11:00
  SPEAKER_01: I think we need to build some slack into our plans that a frontier model release that we were planning for might not show up when we expect it to, or at least it will be deliberately less capable than the pace we might have been used to.
• 11:13
  SPEAKER_01: And I think we should also watch for what independent evaluator access actually turns into because it could end up becoming a real vendor selection criterion faster than most people might expect.
• 11:26
  SPEAKER_01: Right now, enterprise buyers ask vendors about SOC 2 and data residency and uptime SLAs.
• 11:33
  SPEAKER_01: Now, in 12 months, we could start seeing enterprise companies ask AI vendors about hey, who has independent access to red team your model and can they publish what they find without your sign-off?
• 11:45
  SPEAKER_01: I mean, these might be real questions that frontier model companies will start getting asked.
• 11:50
  SPEAKER_01: And it's a trust signal that didn't exist as a checkbox six months ago.
• 11:55
  SPEAKER_01: But pretty soon it very well maybe.
• 11:58
  SPEAKER_01: Now, you may need to get ahead of this, whether you're buying AI capability or building on top of it, and you might get asked this question yourself from your customers.
• 12:08
  SPEAKER_01: Now, one thing that took me a minute to fully appreciate, but pay attention to who actually benefits when safety coordination becomes the industry standard.
• 12:18
  SPEAKER_01: The critique of AMODA's plan isn't necessarily wrong just because it's cynical.
• 12:23
  SPEAKER_01: When the biggest labs propose the safety rules, those rules tend to describe exactly what the biggest labs can already afford to do.
• 12:30
  SPEAKER_01: So watch whether these AI safety standards over the next year end up becoming a moat for these three companies instead of a floor for the whole industry.
• 12:38
  SPEAKER_01: I'm not saying that we should be against the safety work.
• 12:42
  SPEAKER_01: I mean, the concerns that have surfaced, these are all very real, but it is important to make sure that these safeguards that are being recommended aren't just benefiting these three companies.
• 12:51
  SPEAKER_01: I also think it's worth paying attention to the gap between what these companies say publicly and what they actually ship.
• 12:58
  SPEAKER_01: I mean, three CEOs agreeing that the industry is moving too fast on some random Saturday in an essay and two tweets, that's a very different thing than three companies that actually start shipping slower.
• 13:10
  SPEAKER_01: Now, I want to see if they actually put their money where their mouths are.
• 13:14
  SPEAKER_01: So watch the releases, not just the essays.
• 13:16
  SPEAKER_01: Now, if they all start releasing products with major capability gains in the next just two or three months at the same pace that they seem to be releasing before, well, you'll have your answer about how much anything from this past weekend actually meant.
• 13:29
  SPEAKER_01: And remember, Musk has done this move before back in 2023 and kind of changed nothing at that time.
• 13:38
  SPEAKER_01: So don't build your risk assessment on what they're saying, build it on what they do next, but just be prepared regardless.
• 13:45
  SPEAKER_01: Finally, for those of you that are building products that sit on top of these models, it's a good moment to have an honest conversation internally about your own dependency on frontier capability gains that you don't control.
• 13:56
  SPEAKER_01: If your roadmap for the next year assumes that the model gets meaningfully better every few months and you're just going to ride that, or worse, you're dependent on that.
• 14:06
  SPEAKER_01: Well, this is the week to start stress testing that assumption because the people building the model just told you on the record that they might not keep that promise, all on purpose for safety reasons that they believe in, whether or not you buy the reasons.
• 14:19
  SPEAKER_01: Now, this is a very good reason to build your own products' value on the things you actually own, your data, your workflow, your context, not purely on borrowed capability from a lab that just said it might pump the brakes.
• 14:32
  SPEAKER_01: Anyway, that's my take.
• 14:34
  SPEAKER_01: I think it's all interesting for us to start thinking about and paying attention to.
• 14:39
  SPEAKER_01: But we're gonna try something new here at Now Shipping, because, in the words of LeVar Burton, you don't have to take my word for it.
• 14:46
  SPEAKER_01: We thought it'd be fun to incorporate others' perspectives on the news that we're covering too.
• 14:51
  SPEAKER_01: So we're going to include an outside perspective here, and I'll cut to this week's fresh take from a friend of mine, the product, Nina Olding.
• 14:59
  SPEAKER_01: She's a staff PM at Core Weave.
• 15:01
  SPEAKER_01: Let's cut to Nina's take on all of this.
• 15:04
  SPEAKER_00: I don't know exactly what it was about this tweet that went so viral or like has really captured the public's imagination.
• 15:11
  SPEAKER_00: Because if you zoom out, this really isn't new, this idea of alignment.
• 15:15
  SPEAKER_00: There's a whole practice around it.
• 15:17
  SPEAKER_00: There are teams that have been working on alignment for years, and alignment is basically the practice of ensuring that AI's interests and actions are aligned with humanity's interests.
• 15:28
  SPEAKER_00: That is a huge problem.
• 15:30
  SPEAKER_00: It is really hard to solve.
• 15:31
  SPEAKER_00: What does it even mean in many ways?
• 15:33
  SPEAKER_00: It's quite philosophical and kind of hard to grok.
• 15:36
  SPEAKER_00: So for that reason, the problem is quite poorly defined, even around what does it mean to pause AI development?
• 15:42
  SPEAKER_00: What does it mean to pace AI development?
• 15:44
  SPEAKER_00: Do you limit the compute?
• 15:45
  SPEAKER_00: Do you limit the number of teams?
• 15:46
  SPEAKER_00: Do you limit the intelligence of the model?
• 15:49
  SPEAKER_00: And in what ways do you even measure that?
• 15:51
  SPEAKER_00: And right now, the models are already plenty smart, smarter than a lot of humans.
• 15:56
  SPEAKER_00: So, in some dimensions at least, what does it mean to say that you're going to limit the intelligence of models?
• 16:04
  SPEAKER_00: So I think it's actually a lot more helpful, at least for most of us who, like the vast majority of us, are not working on the frontier.
• 16:11
  SPEAKER_00: What does it mean for us in our careers and in our daily lives?
• 16:16
  SPEAKER_00: What is within our control?
• 16:17
  SPEAKER_00: And for me, the things that are within my control are am I shipping AI responsibly?
• 16:22
  SPEAKER_00: At its heart, this debate is around assurance.
• 16:25
  SPEAKER_00: Can we be sure that models are behaving in a way that is aligned with the interests of humanity?
• 16:30
  SPEAKER_00: But within your product, how can you be sure that the AI that you're shipping is aligned with the interests of your users?
• 16:36
  SPEAKER_00: Can you be transparent with your users?
• 16:38
  SPEAKER_00: And I think really shipping AI responsibly for us as PMs and shipping AI responsibly as a lab, they're just sort of the micro and the macro of the same problem, which is around putting your user interests first and putting the interests of humans first and understanding the role of AI in your product and in the lives of your users and giving them the controls that they need and the information that they need to use it in a way that is aligned with the expectations of humanity and the best interests of your users.
• 17:10
  SPEAKER_00: And that can be around evals and quality disclosures, human in the loop, giving your users choices.
• 17:17
  SPEAKER_00: So there's a lot of things that are within our control around responsibility, not just this pacing the frontier debate, which is really interesting, but also maybe at an altitude above where most of us are operating right now.
• 17:32
  SPEAKER_01: That is going to do it for this one.
• 17:34
  SPEAKER_01: If you found this week's episode valuable, please subscribe, tell a friend, share this to the outside world.
• 17:41
  SPEAKER_01: You spreading the word actually helps us.
• 17:43
  SPEAKER_01: So we appreciate you doing all of that.
• 17:45
  SPEAKER_01: And if you have any suggestions for me, leave a comment below.
• 17:48
  SPEAKER_01: I will read each and every one of your comments.
• 17:50
  SPEAKER_01: Next week, you will see me right here again with more AI news stories that matter to you.
• 17:56
  SPEAKER_01: Once again, my name is Mike Belcito, and brought to you by the team at Mind the Product.
• 18:00
  SPEAKER_01: This is now shipping.