“Instrumental convergence is incoherent (so pls slightly lower your p(extinction))” by Arepo

LessWrong (30+ Karma) · Oct 23, 2026 · 46m

Crossposted from the EA forum

‘Like most of life's dangers, the dangers of artificial intelligence are not easily identified from the armchair.’

J Dmitri Gallow

Epistemic status

I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2 to 300 hours learning about modern AI safety technical work, and some amount of spare time over the years before that thinking about here-relevant notions such as preferences from an…