“Subtly wrong advice misaligns models more than obviously wrong advice” by Christine Corry, arav-dhoot, Jozdien

LessWrong (30+ Karma) · Oct 28, 2026 · 22m

tl;dr

We fine-tune 12 open-weight models on correct, obviously incorrect, and subtly incorrect advice across 8 narrow domains

We replicate the OpenAI result and find that subtly incorrect data produces more emergent misalignment than obviously incorrect data

+10.6 points pooled over all model-domain pairs (95% CI [9.1, 12.0])

We also extend this analysis to Petri evals and find uplift in fabrication and concerning behavior

Fabrication: 2.6 → 6.7 (obvious) and 8.7 (subtle) on the 10-point judge scale

Concerning: 3.7 → 7.5 (obvious) and 8.2 (subtle)

Emergent misalignment seems to rise with…