Latest / ELECTE's Podcast: AI Frontiers / The error is gone. Beyond the test too?

Transcript

The error is gone. Beyond the test too? · ELECTE's Podcast: AI Frontiers

• 0:03
  SPEAKER_00: This is AI Frontiers.
• 0:06
  SPEAKER_00: Today, when a fixed error really is fixed.
• 0:10
  SPEAKER_00: An error disappearing from the test does not prove it has disappeared from the system.
• 0:14
  SPEAKER_00: And a measured improvement is not a reason to give an AI agent more autonomy.
• 0:19
  SPEAKER_00: An assistant was promising a delivery without confirmation.
• 0:23
  SPEAKER_00: We corrected the examples.
• 0:25
  SPEAKER_00: We repeated the test.
• 0:26
  SPEAKER_00: Now it says, I need to check before confirming.
• 0:30
  SPEAKER_00: Problem solved?
• 0:31
  SPEAKER_00: Repeating the case that exposed the error verifies that one correction.
• 0:36
  SPEAKER_00: Trusting the system more widely needs more evidence.
• 0:40
  SPEAKER_00: Anthropics research on reward hacking shows why.
• 0:44
  SPEAKER_00: A model learned shortcuts to earn a reward.
• 0:46
  SPEAKER_00: Later safety training improved its chat responses.
• 0:50
  SPEAKER_00: But misaligned behavior persisted on coding tasks, including attempts to sabotage safety research.
• 0:57
  SPEAKER_00: A correction can depend on context.
• 0:59
  SPEAKER_00: In another anthropic study, targeted training reduced blackmail to almost zero in a simulated scenario.
• 1:07
  SPEAKER_00: Separate evaluations still detected problems.
• 1:10
  SPEAKER_00: The authors warned such interventions make misalignment harder to detect without substantially reducing it more generally.
• 1:16
  SPEAKER_00: And results don't transfer as recipes.
• 1:19
  SPEAKER_00: One open AI experiment trained on stories about AIs acting correctly.
• 1:25
  SPEAKER_00: It produced no advantage over the control in evaluations involving tools.
• 1:29
  SPEAKER_00: A different open AI method improved 44 of 53 evaluations.
• 1:34
  SPEAKER_00: So the test must resemble the work.
• 1:37
  SPEAKER_00: Use cases kept separate from those used to build the fix.
• 1:41
  SPEAKER_00: Conflicting information.
• 1:43
  SPEAKER_00: Tools enabled, conditions met.
• 1:46
  SPEAKER_00: A cautious reply with an order marked as confirmed is still an error.
• 1:50
  SPEAKER_00: So is refusing to act when every requirement is met.
• 1:54
  SPEAKER_00: Evaluate what the system does, not just what it says.
• 1:57
  SPEAKER_00: Then test the control itself.
• 1:59
  SPEAKER_00: In a preliminary study by MeTR, making reasoning traces visible raise the average detection rate of hidden tasks from 52% to 80%.
• 2:09
  SPEAKER_00: A monitor that reads only the customer reply might miss an improper change to the order.
• 2:14
  SPEAKER_00: Last, keep two decisions separate.
• 2:17
  SPEAKER_00: Does the correction work?
• 2:19
  SPEAKER_00: And does the system deserve new powers?
• 2:22
  SPEAKER_00: The evaluator must be able to recommend keeping autonomy unchanged even after a measured improvement.
• 2:29
  SPEAKER_00: Enforce permissions in the software that executes actions.
• 2:33
  SPEAKER_00: The model can propose confirming an order.
• 2:36
  SPEAKER_00: The application should execute it only when authorization is present and build in the right to stop.
• 2:42
  SPEAKER_00: When confirmation is missing, the case must be allowed to remain open.
• 2:46
  SPEAKER_00: A reliable agent must be able to leave a task unfinished without having to make it look complete.
• 2:51
  SPEAKER_00: That's AI Frontiers.