Latest / DX Today | No-Hype Podcast & News About AI & DX / Anthropic Warns Its AI May Soon Start Building Better AI

Transcript

Anthropic Warns Its AI May Soon Start Building Better AI · DX Today | No-Hype Podcast & News About AI & DX

• 0:00
  SPEAKER_00: Welcome to the DX Today Podcast, your daily deep dive into the AI ecosystem.
• 0:04
  SPEAKER_00: I'm Chris, and joining me as always is Laura.
• 0:07
  SPEAKER_01: Hi, Chris, and hello everyone.
• 0:09
  SPEAKER_01: Today we have a story that sounds like science fiction, but it was actually tucked into the risk factors of a securities filing, which is about the least dramatic place you could imagine.
• 0:18
  SPEAKER_00: Right, so let me set it up for people.
• 0:20
  SPEAKER_00: Anthropic, the company behind Claude, is preparing to go public.
• 0:24
  SPEAKER_00: And on Monday, it released an amended version of its registration statement with the Securities and Exchange Commission.
• 0:30
  SPEAKER_01: And the headline that most business reporters focused on was the money, with revenue growing fast and expenses growing fast too.
• 0:38
  SPEAKER_01: But TechCrunch noticed something buried much deeper in the document that is far more interesting.
• 0:43
  SPEAKER_00: Okay, so walk me through it because I want to understand exactly what the company said before we start speculating about robots building robots in some secret lab somewhere.
• 0:52
  SPEAKER_01: Fair enough.
• 0:53
  SPEAKER_01: The filing says anthropic may imminently meet the definition, inside its own responsible scaling policy, of what it calls dramatically accelerated progress attributable to AI.
• 1:04
  SPEAKER_01: That is the exact phrase, and it comes from internal estimates about future models.
• 1:08
  SPEAKER_00: So that is a mouthful of corporate language.
• 1:11
  SPEAKER_00: In plain English, I think what they are saying is that their AI may soon be speeding up the research that produces the next AI.
• 1:18
  SPEAKER_00: Is that a fair translation?
• 1:19
  SPEAKER_01: That is exactly the translation.
• 1:22
  SPEAKER_01: People in the field call the extreme version of this recursive self-improvement, where each new system helps build a better one, and that better one helps build an even better one after that.
• 1:32
  SPEAKER_00: Now I have to push back right away because the filing also says something important, which is that no current model has actually been designated as crossing that line.
• 1:40
  SPEAKER_00: So this is a forecast, not an event.
• 1:42
  SPEAKER_01: Correct, and that distinction matters a lot.
• 1:45
  SPEAKER_01: Anthropics says it has not designated any current model as exceeding the threshold, and that it is still assessing the underlying data.
• 1:53
  SPEAKER_01: The word imminently is doing a lot of work in that sentence.
• 1:56
  SPEAKER_00: So let us get concrete about the threshold itself, because I suspect most listeners have never read a responsible scaling policy and have no idea how a company would even measure something like this.
• 2:06
  SPEAKER_01: Great question.
• 2:07
  SPEAKER_01: The policy describes the threshold as roughly doubling the baseline rate of improvement in AI capability, where that doubling is plausibly caused by AI itself.
• 2:16
  SPEAKER_01: Think of compressing two years of normal progress into a single year.
• 2:20
  SPEAKER_00: And I read that Anthropic actually clarified that wording back in April because people were confusing it with something else.
• 2:26
  SPEAKER_00: What was the confusion and why does that clarification matter for how we read this filing?
• 2:31
  SPEAKER_01: In April, the company issued version 3.1 of the policy and explained that the doubling refers to overall capability, not to how productive individual researchers feel.
• 2:41
  SPEAKER_01: That is a much higher bar than engineers saying they code twice as fast.
• 2:44
  SPEAKER_00: That is a really useful distinction because I hear people say all the time that AI tools have doubled their output.
• 2:50
  SPEAKER_00: And that is clearly not the same thing as the entire field moving twice as fast.
• 2:54
  SPEAKER_01: Exactly.
• 2:55
  SPEAKER_01: And there is a trail of warnings leading up to this.
• 2:58
  SPEAKER_01: Back in February, Anthropic wrote that confidently ruling out this threshold is becoming increasingly difficult, even while saying that Claude Opus 4.6 had not crossed it.
• 3:08
  SPEAKER_00: So this did not come out of nowhere.
• 3:09
  SPEAKER_00: They have been flashing a yellow light for most of the year.
• 3:12
  SPEAKER_00: And now in the filing, that yellow light looks like it is getting a lot closer to turning red.
• 3:17
  SPEAKER_01: That is a good way to picture it.
• 3:19
  SPEAKER_01: And one detail in the TechCrunch report really jumped out at me, which is that Anthropics says it has already used AI to build models whose leaps and capability it did not recognize at the time.
• 3:29
  SPEAKER_00: Wait, that one actually worries me a little.
• 3:31
  SPEAKER_00: If you are building something and you cannot tell in the moment how much better it got, that sounds like a measurement problem before it becomes anything else at all.
• 3:39
  SPEAKER_01: I think that is the sharpest reading of it.
• 3:42
  SPEAKER_01: Evaluation is the hard part.
• 3:44
  SPEAKER_01: If your yardsticks lag behind your models, you could cross a threshold and only realize it months later when you look back at the data.
• 3:51
  SPEAKER_00: So what does the company say it will do about all this?
• 3:53
  SPEAKER_00: Because a warning without a plan, man, is uh just a scary paragraph.
• 3:58
  SPEAKER_00: And investors and regulators are going to want to see the actual plan.
• 4:01
  SPEAKER_01: The filing says using AI to build more capable AI could pose a considerable safety risk.
• 4:07
  SPEAKER_01: According to TechCrunch, it also says the company does not expect to safely let Claw design and build future frontier generations on its own without meaningful human oversight.
• 4:16
  SPEAKER_00: That is a striking thing for a company to say to people who are about to buy its stock, because it is basically admitting that the most powerful version of its own product needs a human hand on the brake.
• 4:27
  SPEAKER_01: And it says it is considering updates to the responsible scaling policy to keep humans in control of frontier development.
• 4:34
  SPEAKER_01: Gizmod also listed commitments the policy already triggers if this threshold is reached, and some of them are pretty unusual.
• 4:41
  SPEAKER_00: Unusual how?
• 4:43
  SPEAKER_00: Give me the specifics, because I think people will be curious what a safety plan for self-improving AI actually looks like when you get past the press release language.
• 4:52
  SPEAKER_01: The policy talks about ambitious and possibly unconventional security methods, an eyes on everything approach to watching internal activity, and red teaming strong enough that the company no longer relies on outside bug bound 20 programs to find its weaknesses.
• 5:07
  SPEAKER_00: Eyes on everything is quite a phrase.
• 5:09
  SPEAKER_00: I assume that means monitoring what the AI systems themselves are doing inside the company, not just what the employees are doing on their laptops during the workday.
• 5:17
  SPEAKER_01: That is how I read it too.
• 5:18
  SPEAKER_01: And under the policy, crossing the threshold also means the company has to write an affirmative case explaining the risks of misalignment and how it plans to reduce them before going further.
• 5:29
  SPEAKER_00: Okay, let me play devil's advocate now because somebody has to.
• 5:32
  SPEAKER_00: A registration statement is a legal disclosure, but it is also a sales document.
• 5:37
  SPEAKER_00: Is it possible this is partly marketing dressed up as caution?
• 5:40
  SPEAKER_01: Skeptics are saying exactly that.
• 5:42
  SPEAKER_01: TechCrunch noted critics who argue a filing is both disclosure and promotion, and Gizmoded it as a message to risk-averse investors or to a world that is eager for a kind of mechanical god.
• 5:53
  SPEAKER_00: Because think about the incentive.
• 5:55
  SPEAKER_00: If you tell investors your technology is so powerful it might start improving itself, that is a frightening warning.
• 6:01
  SPEAKER_00: But it is also the most impressive product claim a technology company could possibly make.
• 6:06
  SPEAKER_01: True, but I would push back on you a little there.
• 6:08
  SPEAKER_01: Lawyers write risk factors to protect the company from being sued later.
• 6:12
  SPEAKER_01: If Anthropic believed this risk was real and stayed quiet, that silence could become a legal problem down the road.
• 6:19
  SPEAKER_00: That is a fair point.
• 6:20
  SPEAKER_00: Securities law punishes you for hiding known risks.
• 6:24
  SPEAKER_00: So, in a way, the safest legal move is to disclose the scariest plausible scenario, whether or not it ever actually comes true.
• 6:31
  SPEAKER_01: Right, and Gizmodel, even while being skeptical, gave anthropic some credit.
• 6:36
  SPEAKER_01: It pointed out that the policy threshold is much more clearly defined than the vague idea of recursive self-improvement that floats around online.
• 6:43
  SPEAKER_00: Let us talk about where that vague idea came from, because I think a lot of listeners have heard the phrase recursive self-improvement and have no idea that it has a long and somewhat strange history behind it.
• 6:55
  SPEAKER_01: It traces back to an older idea called seed AI, coined by the researcher Eliezer Yudkowski, and it was developed for years in rationalist and effective all-truest online communities long before any major company used it in a filing.
• 7:08
  SPEAKER_00: So something that started as a thought experiment on internet forums is now sitting in a document that the Securities and Exchange Commission will review.
• 7:16
  SPEAKER_00: That is a remarkable journey for an idea to take.
• 7:19
  SPEAKER_01: It really is, and it shows how quickly the field has moved this year.
• 7:23
  SPEAKER_01: TechCrunch pointed out that Claude has improved rapidly in coding, mathematics, and cybersecurity, which are exactly the skills you would need for AI to help with AI research.
• 7:32
  SPEAKER_00: Now I want to bring this back to the people listening who run businesses or teams.
• 7:36
  SPEAKER_00: Why should a chief information officer or a product manager care about a threshold in a safety policy at one AI company?
• 7:42
  SPEAKER_01: Because the pace of change is the whole story for them.
• 7:45
  SPEAKER_01: If capability gains really do double, then the tools you evaluated this spring could be badly outdated by next spring, and your planning cycles have to get much shorter.
• 7:55
  SPEAKER_00: That is practical.
• 7:56
  SPEAKER_00: And there's a flip side, which is that a company slowing down to add human oversight might ship certain features more carefully, and that could affect your roadmap if you build on its models.
• 8:07
  SPEAKER_01: And it is worth noting the contrast with how companies are using these tools today.
• 8:11
  SPEAKER_01: On the same day, reports said Goldman Sachs largely limited coding assistance like Claude Code to its engineers, citing productivity and risk concerns.
• 8:20
  SPEAKER_00: Which is interesting because it shows that real organizations are still figuring out basic questions of who should use these tools, even while the companies building them are debating whether the tools are starting to build themselves.
• 8:32
  SPEAKER_01: That gap between the lab and the office is huge.
• 8:35
  SPEAKER_01: Most companies are still in the pilot stage, while the frontier labs are writing policies for scenarios where AI does a meaningful share of its own research and development.
• 8:45
  SPEAKER_00: So what about governance?
• 8:46
  SPEAKER_00: If a company is telling the government in a formal filing that it might soon cross a line like this, does that create any obligation for regulators or lawmakers to step in?
• 8:56
  SPEAKER_01: Not directly, because the policy is voluntary, but it gives regulators a concrete reference point.
• 9:01
  SPEAKER_01: The TechCrunch author argued that if Anthropic believes this is near, people should take it seriously and possibly start thinking about governance now.
• 9:09
  SPEAKER_00: And I would add transparency to that list.
• 9:11
  SPEAKER_00: The policy now requires that at least 200 employees can read the unredacted risk reports, and the company publishes redacted versions, like the one it released this past August.
• 9:21
  SPEAKER_01: Yes, that rule came in with version 3.4 in July.
• 9:25
  SPEAKER_01: It is a way of making sure that a large group of insiders, not just a few executives, actually knows what the models can do.
• 9:32
  SPEAKER_00: Let me ask you the big question then.
• 9:34
  SPEAKER_00: Laura, do you think we are actually close to AI that builds better AI?
• 9:38
  SPEAKER_00: Or is this a case of a company being extremely cautious with its legal language?
• 9:42
  SPEAKER_01: My view is that the truth is somewhere in the middle.
• 9:45
  SPEAKER_01: AI is already doing real work inside these labs, so some acceleration is happening, but the full runaway scenario is still a forecast that nobody outside the company can verify yet.
• 9:55
  SPEAKER_00: I agree, and I think the most important word in this entire story is verify.
• 10:00
  SPEAKER_00: Right now we are taking the company at its word in both directions, both that the threshold is near and that it has not been crossed.
• 10:07
  SPEAKER_01: Which is why independent evaluation matters so much.
• 10:10
  SPEAKER_01: Whether it comes from government institutes, academic researchers, or outside auditors, someone other than the company needs to be able to check these numbers before we treat them as facts.
• 10:21
  SPEAKER_00: Let me raise one more angle before we wrap up, which is competition.
• 10:24
  SPEAKER_00: If one lab says it is getting close to this threshold, what does that tell us about where its rivals might be right now?
• 10:30
  SPEAKER_01: It suggests they are probably in a similar neighborhood because the leading labs tend to move within months of each other.
• 10:36
  SPEAKER_01: The difference is that Anthropic has written down a public definition, so it has to report against it.
• 10:42
  SPEAKER_00: That creates a strange incentive, does it not?
• 10:44
  SPEAKER_00: The company that measures and discloses carefully ends up sounding the most alarming, while a company that never defines a threshold never has to announce that it crossed one.
• 10:53
  SPEAKER_01: That is the transparency paradox in a nutshell.
• 10:56
  SPEAKER_01: It is one reason many policy researchers want common definitions across the whole industry, so that every lab is measured against the same yardstick instead of grading its own homework.
• 11:06
  SPEAKER_00: So, for listeners, here's the takeaway.
• 11:08
  SPEAKER_00: Watch for the updated responsible scaling policy.
• 11:11
  SPEAKER_00: Watch for whether any models formally designated as crossing the threshold and watch how regulators respond during the rest of this public offering process.
• 11:20
  SPEAKER_01: And maybe read the risk factor section of the next big technology filing you come across.
• 11:25
  SPEAKER_01: As this story shows, sometimes the most important sentence in the whole document is hiding deep in the back pages, not in the headline.
• 11:32
  SPEAKER_00: That's all for today's episode of the DX Today podcast.
• 11:35
  SPEAKER_00: Thanks for listening, and we'll see you next time.