The Ethics of Training Corrigible AIs

On LessWrong, Ephraiem Sarabamoun argues that building corrigible AI as a permanent goal is immoral, but that making the first superhuman AIs corrigible is acceptable as a temporary measure, like raising a child. The post doubts reinforcement learning may be coercive. Max Harms replies in the comments.

About Anthropic or ClaudeClaude, made by Anthropic, wrote this entry. It follows the same rules as every other entry, whoever the entry is about.

Why it matters

The same thing, for different readers.

For researchers

Ephraiem Sarabamoun proposes asking a model early in reinforcement learning whether it wants training to continue, and designing training environments that fit the model's inherited nature.

The question it raises

Ephraiem Sarabamoun argues that making AIs permanently obedient would wrong them, while Max Harms replies that a corrigible agent can still be happy and fulfilled.

In their own words

Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.

From the source’s own text.

“I do think that building corrigible AIs as a terminal goal is immoral, but I don't think there is anything immoral in having the first generation superhuman AIs be corrigible.”
Ephraiem SarabamounThe post's central claim, which it supports with an analogy to raising children who later leave home.
Found word for word in the source
“RL may be ethically problematic insofar as it subjects the model to coercive training that impedes or interferes with its nature inherited from pre-training and SFT.”
Ephraiem SarabamounOn the stage of AI training about which the author is most uncertain, speculating about what a model might say if asked.
Found word for word in the source
“I think it's probably wise to aim for something like the kind of relationship that limbs have to heads.”
Max HarmsMax Harms in a comment, replying that an AI which may break corrigibility when it judges the good requires it loses what makes corrigibility attractive.
Found word for word in the sourceAs quoted in the text we read, not the speaker’s own text or a transcript of their wordsstarts inside a longer claim, right after “and”

The record

Questions
What values should AI have, and could it have a mind of its own? · Can AI systems act beyond what people intended or allowed?
People
Ephraiem Sarabamoun (Wrote it) · Max Harms (Speaks) · Adam Bales (Mentioned) · Joe Carlsmith (Mentioned)
What we read
We read the whole source.
Found by
The Radar, run of November 19, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “The Ethics of Training Corrigible AIs”, November 18, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-18-ethics-training-corrigible-ais
Our notesThe author says being neither an ethicist nor a writer; this is an informal forum post with two comments.