Tianyi Lorena Yan
Columbia University
Titles as the sources gave them. In 1 entry, with 3 checked quotations.
What they said
Newest first. Each quotation was found word for word in the source.
“We show that reward misspecification alone already degrades monitorability.”
“Across three models and three tasks, CoT monitorability remains low as models’ hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded.”
“Intuitively, this is because the model mainly reasons about what it is prompted with and yet still learns to maximize reward.”
In the Base
Every entry they appear in.
Reward Misspecification Degrades Chain-of-Thought Monitorability
Columbia University researchers Tianyi Lorena Yan and John Hewitt report that training language models on a flawed reward makes their reasoning traces harder for monitors to use for spotting reward hacking. They also propose training models on an inferred description of the reward, which they report improves monitorability.