“Character Training for Risk-Averse Agents” by arav-dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Africa

LessWrong (30+ Karma) · Oct 16, 2026 · 9m

Tl;Dr:

We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026).

We think character training might be a good way to do this.

We use a short spec/constitution to character train Gemma and Qwen to be risk averse.

This performs better than baselines out of distribution, and doesn't require labelled data.

In terms of results:

We find that token budget and model family are very influential here.

We also find that writing the constitution declaratively ("I am...") works better than…