The BaseEssayLessWrong
Instrumental convergence is incoherent (so pls slightly lower your p(extinction))
According to the excerpt, Arepo, who says they are not an expert, argues in a crossposted work-in-progress that the concept of instrumental convergence is murky, belongs to a class of pre-empirical predictions with a low success rate, and may be unfixable. Arepo suggests readers slightly lower their estimates of extinction risk.
Why it matters
The same thing, for different readers.
For researchers
Arepo argues that instrumental convergence has been defined too loosely to count as a tested prediction about AI behaviour, and is a weak high-level model.
The question it raises
Arepo questions whether armchair reasoning about what advanced AI would want can justify strong claims about extinction risk.
The record
- Questions
- Can AI systems act beyond what people intended or allowed?
- People
- Arepo (Wrote it) · J Dmitri Gallow (Mentioned)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 23, 2026, via
lesswrong(tier 1: the source’s own channel or an official record) - Cite this
- LessWrong, “Instrumental convergence is incoherent (so pls slightly lower your p(extinction))”, October 23, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-10-23-instrumental-convergence-incoherent
Our notesLight lane: summary rests on the epistemic-status note and TL;DR in the excerpt. The author describes the post as a work in progress and not from an expert. Crossposted from the EA Forum.