Blue-teaming AI Through Safety Hillclimbing

Skye and Eam report an experiment in which GPT 6.1 Sol spent 12 hours fine-tuning an abliterated Gemma 4 12B to refuse harmful requests, scored against a development set. They argue such tasks could become environments where agents learn to do safety research.

About Anthropic or ClaudeClaude, made by Anthropic, wrote this entry. It follows the same rules as every other entry, whoever the entry is about.

Why it matters

The same thing, for different readers.

For researchers

The authors show a frontier coding agent improving another model's safety against a score it could repeatedly test, and flag Goodharting and sandbagging as the risks.

The question it raises

Letting an AI agent decide how another AI is made safer raises the question of who checks the checker when the score can be gamed.

In their own words

Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.

From the source’s own text.

“We find it notable that the rebliterated candidate reduces harmful assistance compared to the official Gemma 4 12B release - though it increases benign refusals.”
SkyeThe authors' reading of the holdout results; the post is bylined to both Skye and Eam, so which one wrote the line is not stated.
Found word for word in the sourceSpeaker known from context, not confirmed by listening
“Rewards can easily favor exploiting the score while appearing safer on a benchmark.”
SkyeThe authors on the risk that frontier agents Goodhart or sandbag safety research.
Found word for word in the sourceSpeaker known from context, not confirmed by listening
“Important aspects, maybe even most aspects, of alignment are contextual and poorly represented by quantifiable metrics.”
SkyeThe authors' stated limit on what hillclimbing and environments can capture.
Found word for word in the sourceSpeaker known from context, not confirmed by listening

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
GPT 6.1 Sol (xhigh), with one GPU and subagents, ran 12 hours of supervised fine-tuning candidates on an abliterated Gemma 4 12B, scored on a 338-question development set and a separate 337-question holdout from XSTest and StrongREJECT.
What it found
On the holdout, harmful-assistance judgments fell from 470/663 for the abliterated baseline to 1/663 for the selected candidate (26/663 for the official release); benign refusals rose from 0/348 to 25/348.
What it does not show
The candidate failed the authors' own five-point benign-refusal check on holdout; a single run on one small model; biological and medical material excluded; model-graded scores.
Status
A report

The record

Questions
Are AI companies putting stronger safeguards into practice? · What happens when AI helps build more capable AI?
People
Skye (Wrote it) · Eam (Wrote it)
What we read
We read the whole source.
Found by
The Radar, run of November 19, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “Blue-teaming AI Through Safety Hillclimbing”, November 19, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-19-blue-teaming-safety-hillclimbing
Our notesTwo authors, first names only; no affiliation given. Speaker set to Skye with speakerCertain false because the byline is joint. The post mentions Anthropic's Anvil and A3 as prior work; not about Anthropic itself.