The BaseResearcharXiv
Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
According to the abstract, Mishra and colleagues use a counterfactual audit that reassigns category definitions among labels. Across three political-text datasets they report models rarely follow the reassigned definitions and fine-tuning strengthens the original label mapping.
Why it matters
The same thing, for different readers.
What it means for you
Studies using AI to label political texts may reflect the model's prior label associations, not the researchers' stated definitions.
For researchers
Mishra and colleagues warn that high annotation accuracy does not show a model applies the supplied category definitions.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Counterfactual audit reassigning definitions among output labels on three political-text datasets, plus retrieval-augmented annotation tests.
- What it found
- Per the abstract, models rarely follow reassigned definitions; nearest-neighbor retrieval recovers 74 to 93 percent of the gain on two tasks.
- What it does not show
- Political-text annotation only; full text not read.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- How fast is AI advancing, and where is it heading?
- People
- Viraj Mishra (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of November 6, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation”, November 6, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-11-06-what-governs-llm-annotation
Our notesLight lane: written from the triage excerpt only. Marginal relevance to the Base.