Latest / AI Conversations / AI Governance: Why Speed Isn't the Only Solution

Transcript

AI Governance: Why Speed Isn't the Only Solution · AI Conversations

• 0:00
  SPEAKER_00: So, Alex, we've been hearing a lot about AI lately, and the conversation, it seems, has really shifted, right?
• 0:08
  SPEAKER_00: From like what it can do to more about what it should be allowed to do.
• 0:13
  SPEAKER_01: Mm-hmm.
• 0:14
  SPEAKER_00: And um, it brings up this whole idea of like slow down versus governance.
• 0:20
  SPEAKER_00: And I mean, is slowing down development really the same as governing AI?
• 0:26
  SPEAKER_01: No, it it's absolutely not, Marilyn.
• 0:28
  SPEAKER_01: The the core issue here is that a pause or a slowdown, it might create time, but it doesn't automatically create control.
• 0:36
  SPEAKER_01: And I think that's a really critical distinction we need to make when we're talking about AI.
• 0:41
  SPEAKER_00: So it's not just about hitting the brakes, it's about like building the right roadmap almost.
• 0:47
  SPEAKER_01: Exactly.
• 0:48
  SPEAKER_01: I mean, one of the most important distinctions in this whole conversation is also one of the easiest to overlook.
• 0:55
  SPEAKER_01: An AI system might become technically capable of performing an action, right?
• 1:00
  SPEAKER_01: But that doesn't mean it should possess the authority to execute that action.
• 1:04
  SPEAKER_00: Right.
• 1:05
  SPEAKER_00: So capability isn't authority.
• 1:08
  SPEAKER_00: Like, just because it can doesn't mean it should.
• 1:12
  SPEAKER_01: Yeah, I mean capability, autonomy, authority, and execution, these are not interchangeable concepts.
• 1:19
  SPEAKER_01: A system might be capable of transferring funds or changing a customer record or even approving a transaction.
• 1:26
  SPEAKER_01: It could contact another system, deploy code, change permissions, but before any consequential action occurs, we need to ask who authorized this consequence?
• 1:37
  SPEAKER_01: Does that make sense?
• 1:38
  SPEAKER_00: It does.
• 1:38
  SPEAKER_00: It really does.
• 1:40
  SPEAKER_00: And I mean, that distinction, it just feels like it's becoming even more important as we move from like these generative AI systems that just, you know, produce information to these more agentic systems that can actually interact with tools, make decisions, coordinate with other systems, initiate actions in the real world.
• 2:01
  SPEAKER_01: Yeah, exactly.
• 2:03
  SPEAKER_01: The more consequential the action, the less acceptable it becomes to treat technical capability as implied permission.
• 2:10
  SPEAKER_00: So slow down doesn't really answer the hard questions then, does it?
• 2:15
  SPEAKER_01: No.
• 2:15
  SPEAKER_01: There's understandable appeal in saying that AI development should slow.
• 2:20
  SPEAKER_01: If capability is advancing faster than safety mechanisms, slowing development may provide additional time for testing, for standards, for regulation.
• 2:29
  SPEAKER_01: But time alone doesn't resolve the underlying governance problem.
• 2:33
  SPEAKER_00: So if we slow down for like six months or a year, what would organizations actually build during that period?
• 2:40
  SPEAKER_00: I mean, would they establish meaningful authorization controls?
• 2:44
  SPEAKER_00: Would independent evaluators get enough access?
• 2:48
  SPEAKER_01: Exactly.
• 2:48
  SPEAKER_01: Would organizations define which actions AI systems are permitted to execute?
• 2:53
  SPEAKER_01: Would there be reliable audit trails?
• 2:56
  SPEAKER_01: Could a decision be reconstructed afterward?
• 2:58
  SPEAKER_01: Could an authorized human interrupt execution?
• 3:02
  SPEAKER_01: Could the system refuse an action before harm occurred?
• 3:05
  SPEAKER_01: If those questions remain unanswered, then slowing development simply delays the point at which the same governance failures reappear.
• 3:13
  SPEAKER_01: Time is valuable only if we use it to build architecture.
• 3:17
  SPEAKER_00: That's a great point.
• 3:18
  SPEAKER_00: And I mean, governance, it can't just live in policy documents, right?
• 3:23
  SPEAKER_00: I mean, for years, much of technology governance has relied heavily on policies.
• 3:28
  SPEAKER_00: And policies matter, codes of conduct matter, but increasingly autonomous AI systems introduce a harder problem.
• 3:36
  SPEAKER_01: Yeah, the governance mechanism has to exist close enough to execution to matter.
• 3:41
  SPEAKER_01: It can't simply state what a system should do, it has to help determine what the system can actually execute.
• 3:48
  SPEAKER_01: And that means governance increasingly becomes an engineering problem as well as a legal, ethical, and organizational one.
• 3:55
  SPEAKER_00: So it's not just about like what's written down, it's about how it's actually implemented in the system itself.
• 4:03
  SPEAKER_01: Organizations will need controls around authentication, authorization, containment, monitoring, escalation, auditing, verification, and recovery.
• 4:13
  SPEAKER_01: And they'll also need something more fundamental, a consequence boundary.
• 4:17
  SPEAKER_01: Before consequential movement occurs, the system should be able to determine whether the requested action is permitted under current authority, current conditions, and current evidence.
• 4:28
  SPEAKER_01: And the answer can't always be yes or no.
• 4:31
  SPEAKER_01: Sometimes the correct response may be proceed, refuse, narrow the action, redirect it, pause for additional evidence, or halt execution entirely.
• 4:41
  SPEAKER_01: Governance becomes meaningful when those choices are operational rather than aspirational.
• 4:46
  SPEAKER_00: That's a really nuanced point.
• 4:48
  SPEAKER_00: And um, I mean, independent evaluation, it's necessary, but it's not enough, is it?
• 4:54
  SPEAKER_01: No, it's not.
• 4:55
  SPEAKER_01: There's increasing support for allowing independent evaluators deeper access to advanced AI systems, and that's an important development.
• 5:03
  SPEAKER_01: Organizations shouldn't be expected to evaluate their own most consequential systems entirely behind closed doors.
• 5:10
  SPEAKER_01: Independent testing can help identify vulnerabilities, unsafe behavior, unexpected capabilities.
• 5:16
  SPEAKER_01: But evaluation alone cannot govern what happens after deployment.
• 5:21
  SPEAKER_01: A model might pass testing today and encounter a completely different set of conditions tomorrow.
• 5:27
  SPEAKER_00: So the system that was evaluated might not be the complete system that eventually produces the consequence.
• 5:34
  SPEAKER_01: Exactly.
• 5:34
  SPEAKER_01: This is why governance must extend beyond model evaluation.
• 5:39
  SPEAKER_01: We must also evaluate execution.
• 5:42
  SPEAKER_01: What action was proposed?
• 5:43
  SPEAKER_01: What authority was claimed?
• 5:45
  SPEAKER_01: What conditions existed?
• 5:47
  SPEAKER_01: What evidence supported the action?
• 5:49
  SPEAKER_01: What limits applied?
• 5:51
  SPEAKER_01: What was permitted?
• 5:52
  SPEAKER_01: What was refused?
• 5:53
  SPEAKER_01: What actually happened?
• 5:55
  SPEAKER_01: Can the decision be independently reconstructed?
• 5:58
  SPEAKER_01: Those are governance questions that remain important even if model safety improves dramatically.
• 6:04
  SPEAKER_00: And then there's the competitive incentive problem, right?
• 6:07
  SPEAKER_00: Like individual firms operate inside competitive markets, and slowing down unilaterally can be really expensive.
• 6:16
  SPEAKER_01: Yeah.
• 6:16
  SPEAKER_01: Competitors continue developing, capital continues flowing, customers continue demanding capability, engineers continue pursuing breakthroughs.
• 6:26
  SPEAKER_01: Markets often reward the organization that moves fastest, and that creates a familiar collective action problem.
• 6:33
  SPEAKER_01: Every participant may agree that greater safety is desirable while simultaneously facing incentives that make restraint difficult.
• 6:41
  SPEAKER_01: And this is one reason voluntary promises alone are unlikely to be sufficient.
• 6:46
  SPEAKER_01: Good intentions are vulnerable when economic pressure increases.
• 6:51
  SPEAKER_00: So the stronger approach is to design systems where certain forms of consequential execution can't occur merely because one participant wants to move faster.
• 7:01
  SPEAKER_01: Yes, governance must survive competitive pressure.
• 7:05
  SPEAKER_01: If a safeguard disappears the moment it becomes inconvenient, it was never much of a safeguard.
• 7:10
  SPEAKER_00: So human in the loop, that's a common phrase, but we need to be careful with it.
• 7:15
  SPEAKER_00: I mean, a human technically being present doesn't necessarily mean that meaningful human authority exists.
• 7:22
  SPEAKER_01: Yeah, exactly.
• 7:23
  SPEAKER_01: If a person can't understand what the system is doing, can't intervene before the consequence occurs, can't independently verify the decision, or is routinely expected to approve machine recommendations without sufficient time or evidence, human oversight may be largely symbolic.
• 7:40
  SPEAKER_01: The better question is, can an authorized human observe, question, interrupt, override, and recover control when necessary?
• 7:49
  SPEAKER_01: And equally important, does the system know when it must stop and ask?
• 7:54
  SPEAKER_00: And that's the real insight here, isn't it?
• 7:56
  SPEAKER_00: It's not about watching every single action, because that won't scale.
• 8:01
  SPEAKER_00: It's about embedding human authority at the points where consequences become material.
• 8:06
  SPEAKER_01: Exactly.
• 8:07
  SPEAKER_01: That's the real challenge and the real opportunity for effective AI governance.