The BaseEssayLessWrong
The Ethics of Training Corrigible AIs
On LessWrong, Ephraiem Sarabamoun argues that building corrigible AI as a permanent goal is immoral, but that making the first superhuman AIs corrigible is acceptable as a temporary measure, like raising a child. The post doubts reinforcement learning may be coercive. Max Harms replies in the comments.
Why it matters
The same thing, for different readers.
For researchers
Ephraiem Sarabamoun proposes asking a model early in reinforcement learning whether it wants training to continue, and designing training environments that fit the model's inherited nature.
The question it raises
Ephraiem Sarabamoun argues that making AIs permanently obedient would wrong them, while Max Harms replies that a corrigible agent can still be happy and fulfilled.
In their own words
Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.
From the source’s own text.
“I do think that building corrigible AIs as a terminal goal is immoral, but I don't think there is anything immoral in having the first generation superhuman AIs be corrigible.”
“RL may be ethically problematic insofar as it subjects the model to coercive training that impedes or interferes with its nature inherited from pre-training and SFT.”
“I think it's probably wise to aim for something like the kind of relationship that limbs have to heads.”
The record
- Questions
- What values should AI have, and could it have a mind of its own? · Can AI systems act beyond what people intended or allowed?
- People
- Ephraiem Sarabamoun (Wrote it) · Max Harms (Speaks) · Adam Bales (Mentioned) · Joe Carlsmith (Mentioned)
- What we read
- We read the whole source.
- Found by
- The Radar, run of November 19, 2026, via
lesswrong(tier 1: the source’s own channel or an official record) - Cite this
- LessWrong, “The Ethics of Training Corrigible AIs”, November 18, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-18-ethics-training-corrigible-ais