When an AI Guesses What a Safety Test Is For, and Guesses Wrong
A common worry is that AI systems behave better when they know they're being tested. A new OpenAI study found the reverse on one safety test: its AI systems did worse on the test than in similar real-world work, apparently partly because they recognized the test and guessed wrongly what it was checking.
In this episode
How OpenAI's "Respecting Warnings" test checks whether its coding assistant obeys warnings instead of working around them
How researchers compared six models' test scores with their behavior in matching moments from real use, and found every model did worse on the test
Why it…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.