The BaseResearcharXiv
Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
According to the abstract excerpt, the authors introduce DiagSafe, a bilingual English-Chinese safety benchmark that replaces safe/unsafe labels with prompt-level rubrics, and report 95.34% agreement between LLM judges and human annotations.
Why it matters
The same thing, for different readers.
For researchers
The authors argue binary safe/unsafe labels explain little and judge inconsistently, and offer rubrics with safety and usefulness scored separately.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- A bilingual benchmark built on an 84-label safety taxonomy, with a base subset of direct harmful queries and jailbreaks and an attack subset embedding harmful intent in instruction-following tasks, scored with prompt-level rubrics.
- What it found
- LLM-as-a-judge evaluation reaches 95.34% agreement with human annotations, according to the authors.
- What it does not show
- Only the abstract excerpt was read; limits were not seen.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- Are AI companies putting stronger safeguards into practice?
- People
- Hongyu Li (Wrote it) · Xinlong Huang (Wrote it) · Zhan Yu (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 22, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics”, October 21, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-21-beyond-binary-labels-diagsafe
Our notesLight lane: written from the truncated triage excerpt only. The comparison figure for other judges was cut off. No source text saved.