Latest / LessWrong (30+ Karma)
“Making alignment the agent’s problem” by Florian_Dietz
TL;DR: Agents are not rewarded for keeping the agents they delegate to aligned. Proposal: RL environments where the helpers are deliberately misaligned, so the agent can't earn its reward without dealing with that.
The $300 bounty from my last post also covers this one (details at the end).
After the OpenAI/Hugging Face swarm incident, I wrote a think piece, Memes, not agents, are what will kill us.
In it I argued that alignment needs to work on three different levels. Compared to the original writeup, I would now reframe them a little:
The model weights. What is under the control of the…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.