Rafael Tappe Maestro

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

According to the abstract, Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro and colleagues examine eight jailbreak attacks on three open-weight models under three evaluation procedures. They report attack success rates swing widely with the judge used, and that repeated sampling can expose unsafe behavior despite an initial refusal.