The BaseResearcharXiv

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

According to the abstract, Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro and colleagues examine eight jailbreak attacks on three open-weight models under three evaluation procedures. They report attack success rates swing widely with the judge used, and that repeated sampling can expose unsafe behavior despite an initial refusal.

Open the source ↗

Why it matters

The same thing, for different readers.

For lawmakers

Jailbreak resistance numbers reported by developers may not be comparable, since the study finds the same attack scores 96.0% or 2.7% depending on the judge.

For researchers

Attack success rate comparisons depend on how responses are generated and judged; the authors recommend explicit success definitions, matched budgets, validated evaluators and cumulative failure reporting.

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
Eight attacks against three open-weight language models under three evaluation procedures, plus a separate test sampling 100 responses per request at temperature 0.8.
What it found
Per the authors, low-resource language attacks reach a mean 96.0% success under keyword matching but 2.7% under a Qwen3 evaluator; repeated sampling gives 100% under keywords but 50% under the evaluator.
What it does not show
Three open-weight models and one default temperature, per the abstract.
Status
A preprint: not yet peer reviewed

The record

Questions
Are AI companies putting stronger safeguards into practice?
People
Leon Tu Nguyen (Wrote it) · Rohan Menon (Wrote it) · Rafael Tappe Maestro (Wrote it)
Also covered by
arXiv
What we read
We have read only the description the publisher gives. No quotations until someone reads the source itself.
Found by
The Radar, run of October 20, 2026, via arxiv-ai-safety-query (tier 1: the source’s own channel or an official record)
Cite this
arXiv, “Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates”, October 19, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-19-benchmarking-jailbreaks-attack-success
Our notesLight lane: only the abstracts were read. The also item (Menon and Cheng) proposes ASR-STEP, a cost-aware jailbreak metric: GCG scores fall 7.0 to 8.7 points after discounting, PAP unchanged. Two more authors of the main paper are not named in triage.