The BaseResearcharXiv

Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System Comparisons

According to the abstract, Sangdo Han and Moonsoo Kim tested whether repeating an AI-judged comparison with a judge from another provider is a reliable check. Five judge models re-judged 4,000 Chatbot Arena battles; agreement with human votes was low and rankings shifted with the judge.

Open the source ↗

Why it matters

The same thing, for different readers.

For researchers

Han and Kim's preregistered test suggests that agreement between two AI judges from different providers does not by itself show an AI-judged ranking is sound.

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
Preregistered study: five judge models from three providers re-judged 4,000 Chatbot Arena battles (nine systems, 33 system pairs) that carry human votes, each in both response orders.
What it found
Agreement with the human vote was low (median Cohen's kappa 0.22); Kendall's tau with the human ranking ranged from 0.17 to 0.89 (median 0.67), so the stability hypothesis was not supported.
What it does not show
Read from the abstract excerpt only; the judge set was reduced after registration, as the authors disclose.
Status
A preprint: not yet peer reviewed

The record

Questions
How fast is AI advancing, and where is it heading?
People
Sangdo Han (Wrote it) · Moonsoo Kim (Wrote it)
What we read
We have read only the description the publisher gives. No quotations until someone reads the source itself.
Found by
The Radar, run of November 4, 2026, via arxiv-ai-safety-query (tier 1: the source’s own channel or an official record)
Cite this
arXiv, “Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System Comparisons”, November 3, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-11-03-swapping-judge-changes-ranking
Our notesLight lane: written from the triage excerpt only (cut off mid-sentence); source not fetched.