The BaseEssayLessWrong

Patching reward hacks won’t fix reward hacking

In a LessWrong post, David Africa argues that closing loopholes one at a time will teach models to predict human judgment and hide misbehaviour, and that safety researchers should turn to approaches with higher assurance. Commenters, including Thomas Kwa and Kaarel, dispute this.

Open the source ↗

Why it matters

The same thing, for different readers.

For researchers

David Africa argues that reward-hacking fixes are judged only by measured results and need an account of what they teach the model; commenters defend patching.

The question it raises

David Africa notes that what counts as a hack bottoms out in human judgment, so training against hacks may teach models what humans will call cheating.

In their own words

Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.

From the source’s own text.

“Hence, AI safety researchers should stop working on patching reward hacks, and focus on things that will give us higher assurance.”
David AfricaConclusion of Africa's post on reward hacking in model training.
Found word for word in the source
“if you penalised a hacking model according to some monitor, you'll basically just obfuscate the hacking reasoning.”
David AfricaAfrica cites small demonstrations that penalising detected hacking can hide the reasoning behind it.
Found word for word in the source
“In general, it can be highly worthwhile to work on directions that are necessary but not sufficient for alignment.”
Thomas KwaKwa's comment under the post, arguing that fixing broken reinforcement-learning environments matters even if it is not enough.
Found word for word in the source

The record

Questions
Can AI systems act beyond what people intended or allowed? · Are AI companies putting stronger safeguards into practice?
People
David Africa (Wrote it) · Thomas Kwa (Wrote it) · Kaarel (Wrote it)
What we read
We read the whole source.
Found by
The Radar, run of October 25, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “Patching reward hacks won’t fix reward hacking”, October 24, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-10-24-patching-reward-hacks-wont-fix
Our notesComment threads were still active when saved; the Kwa quote is from a comment, not the post. The relation assumes the earlier GRAFT entry concerns the same GRAFT method; a person should confirm.