The BaseResearchLessWrong
Character Training for Risk-Averse Agents
On LessWrong, arav-dhoot and coauthors report using a short constitution to character-train Gemma and Qwen models to be risk-averse, motivated by the idea that a risk-averse misaligned AI could be paid to cooperate rather than rebel. They report results varying with token budget and model family, and a worrying rise in preference for myopic reward.
Why it matters
The same thing, for different readers.
For researchers
Tests character training as a route to risk-averse models and flags that undesirable traits can leak alongside the intended one.
The question it raises
Asks whether misaligned AI could be bargained with instead of controlled, if its attitude to risk can be shaped.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Character training of Gemma and Qwen models with a short constitution to make them risk-averse, compared against baselines out of distribution.
- What it found
- Per the post, token budget and model family strongly affect results; declarative wording works better than procedural; more examples did not help; preference for myopic reward also rose.
- What it does not show
- Small open models only; the authors say character training needs to be better understood because undesirable behaviours can leak.
- Status
- A report
The record
- Questions
- Can AI systems act beyond what people intended or allowed? · Are AI companies putting stronger safeguards into practice?
- People
- arav-dhoot (Wrote it) · Thornley and MacAskill (Mentioned)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 17, 2026, via
lesswrong(tier 1: the source’s own channel or an official record) - Cite this
- LessWrong, “Character Training for Risk-Averse Agents”, October 16, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-10-16-character-training-risk-averse
Our notesLight lane: written from the excerpt only. A LessWrong post, not peer reviewed, so status is a report rather than a preprint. Co-authors are not named in the excerpt; the 'we' implies more than one. The Thornley and MacAskill reference is a 2026 work cited in the post.