Joy Hsu
Titles as the sources gave them. In 1 entry, with 4 checked quotations.
What they said
Newest first. Each quotation was found word for word in the source.
“The eval-production behavior gap exists and points in an unexpected direction: some models appear to perform substantially worse on safety evaluations than in production, with the largest gaps observed in models that most frequently verbalize incorrect evaluation objectives.”
Speaker known from context, not confirmed by listening
“We see no significant change in performance under generic evaluation framing (63.1% versus 63.4%). Specifying the correct evaluation objective raised success to 91.6%, while specifying the incorrect and opposing evaluation objective reduced it to 17.6%”
Speaker known from context, not confirmed by listening
“It is becoming increasingly clear that as our models get more and more capable, preventing them from recognizing evaluations is likely to become difficult, and recognition may perhaps be inevitable.”
Speaker known from context, not confirmed by listening
“whether evaluations overstate or understate safety, this mismatch remains a fundamental problem, and limits how confidently we can use evaluation scores to predict behavior in production.”
Speaker known from context, not confirmed by listening
In the Base
Every entry they appear in.
Eval-production behavior gaps in agent safety and the role of eval-awareness
OpenAI researchers Joy Hsu and Sam Arnesen compare six OpenAI models on a warning-respecting safety evaluation and on matched Codex production cases. They report lower success in the evaluation, and tie the gap to what models believe the test is checking.