The BaseResearcharXiv
High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis
According to the abstract, Senger, Campbell, van der Goot and Plank test LLM coding pipelines for social-science text annotation across four datasets and five models. They report instability, domain sensitivity and label-level disparities hidden by accuracy and micro-F1, which persist under common optimization.
Why it matters
The same thing, for different readers.
For researchers
Senger, Campbell, van der Goot and Plank conclude that LLM annotation pipelines need targeted validation, not just aggregate accuracy, before serving as measurement tools.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Failure-mode analysis of LLM codebook pipelines across four datasets and five models under perturbations, domain shifts and optimization procedures.
- What it found
- Stability, robustness and conceptual-validity failures that standard metrics do not show, persisting under common optimization.
- What it does not show
- Abstract excerpt only.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- How is AI changing childhood, health and daily life?
- People
- Elena Senger (Wrote it) · Yuri Campbell (Wrote it) · Rob van der Goot (Wrote it) · Barbara Plank (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 27, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis”, October 26, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-26-high-scores-hidden-failures-codebook
Our notesLight lane: excerpt only.