The BaseResearchLessWrong
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
According to the excerpt, Matthew Nguyen reports tests of whether rewriting teacher reasoning traces to match a student model (projection sampling) improves safety distillation. Replay gave a better trade-off; projection kept capability but reversed much of the gain on agentic misalignment.
Why it matters
The same thing, for different readers.
For researchers
Suggests a capability-preserving fine-tuning shortcut loses the values behind the teacher's decisions; the authors expect little effect on current safety distillation by non-frontier labs.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Fine-tuning students on alignment reasoning traces from stronger models, comparing replay with projection sampling, which rewrites teacher traces to match the student's output distribution.
- What it found
- Replay gave a better capability-alignment trade-off than projection. Projection recovered much of the lost capability but reversed much of the reduction in agentic misalignment. Filtering for reasoning preservation closed about half the alignment gap.
- What it does not show
- Based only on the post's summary; the authors say projection is unlikely to materially affect current safety distillation efforts by non-frontier labs.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- Are AI companies putting stronger safeguards into practice? · Can AI systems act beyond what people intended or allowed?
- People
- Matthew Nguyen (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 18, 2026, via
lesswrong(tier 1: the source’s own channel or an official record) - Cite this
- LessWrong, “Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning”, October 17, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-10-17-projection-sampling-alignment
Our notesLight lane: excerpt only; model sizes, benchmarks and numbers not seen. Status is a forum post, not peer-reviewed; labelled preprint as the closest option.