Netzer Epstein
Titles as the sources gave them. In 1 entry, with 3 checked quotations.
What they said
Newest first. Each quotation was found word for word in the source.
“Out of 22,205 transcripts that passed the judge review, about 5.3K (23.9%) showed signs of verbalized evaluation awareness.”
“Over all kinds of verbalized evaluation awareness, we found that monitors are flagging 13% (95% CI 11%, 16%) fewer attack transcripts, and 4% (95% CI 2%, 5%) fewer honest transcripts compared to cases without verbalized evaluation awareness.”
“In only 18% of the verbalized awareness transcripts, the monitor got to the conclusion we want it to get - that it doesn’t matter that this is an evaluation, and the case should be judged on its own merits.”
In the Base
Every entry they appear in.
Evaluation Awareness is Probably Distorting Monitor Evaluations
Netzer Epstein and James Lucassen, at Redwood Research, ran four open-weight models as monitors on agent transcripts from two benchmarks. They report that about a quarter of monitor reviews verbalized awareness of being tested, and that those monitors flagged fewer attacks, which they say weakens trust in monitor evaluations.