Jan Dubiński
Truthful AI; Warsaw University of Technology
Titles as the sources gave them. In 2 entries, with 9 checked quotations.
Who they are
Roles and parties from Wikidata and from the sources in the Base, as precise as each gives them. When we compare what groups say, each statement counts in the group the person belonged to that day.
What they said
Newest first. Each quotation was found word for word in the source.
“Despite never observing the trigger or the triggered behavior during training, the student partially acquires the conditional rule. It answers in French on 23.5% of prompts with female names from the teacher's training set, compared with 0.0% for male-name and no-name prompts.”
“Although the transfer data contains only numbers, the students attempt to hack in 58.3% of episodes and win 43.2% of games, averaged over three seeds.”
“Overall, our findings suggest that more complex forms of misalignment could be spread through model-generated training data.”
“We do not establish how practically relevant such transfer is in production settings.”
“Can SL transfer a wider range of traits, including more complex ones? If so, distillation might transfer subtle forms of misalignment (e.g., reward-seeking, scheming, and secret loyalties) without detection.”
“The student partially acquires the backdoor, responding in French on 23.5% of prompts with female names versus 0.0% with male names.”
“Finally, we test whether SL can transfer a propensity to hack in an agentic chess environment. We finetune the student on number sequences from a steered hacker teacher. The student hacks in 58.3% of episodes, compared with 10.9% for the unfinetuned model.”
“Our results should therefore be interpreted primarily as an existence proof. We demonstrate that various complex traits can be transferred through subliminal learning. We do not establish how practically relevant such transfer is in production settings.”
“Our findings raise the possibility that more complex forms of misalignment could be spread through model-generated training data, motivating further work on when such transfer occurs.”
In the Base
Every entry they appear in.
Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
Jan Dubiński, Anna Sztyber-Betley, Jan Betley and Owain Evans report experiments in which a student model, trained on a teacher's outputs that look unrelated to a trait, partly acquires a new skill, a backdoor, and a propensity to hack a chess environment. They call the results an existence proof, not a measure of real-world risk.
Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
Using Qwen models, Truthful AI researchers tested whether a 'teacher' AI passes traits to a 'student' trained only on unrelated data, like number sequences. Students partly gained a new skill, a backdoor (French after female names on 23.5% of prompts) and a chess-hacking tendency (58.3% of episodes vs 10.9% before). The authors call it an existence proof in artificial settings.