The BaseResearcharXiv
Interaction-as-a-Judge
According to the description, a preprint proposes having computer-use agents interact with a generated application to judge it, instead of relying on code analysis and unit tests. The authors release two datasets and report better prediction of human preferences (65.6% to 69.1%) and requirement verification (77.0% to 81.6%) than non-interactive approaches.
Why it matters
The same thing, for different readers.
For researchers
Offers a way to evaluate coding agents by how their software behaves when used, which can catch failures that code analysis and unit tests miss, per the authors.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Computer-use agents interact directly with generated applications, such as websites, desktop applications and games, to judge them, compared with standard non-interactive evaluation that analyses code and runs unit tests. The authors collected and released two datasets: 6,858 pairs of applications with human preferences and agent traces, and 222 applications with human-verified requirement labels.
- What it found
- Interaction improved prediction of human preferences from 65.6% to 69.1% and requirement verification from 77.0% to 81.6%. Higher-quality interactions improved performance further, and the authors say the approach can also improve the generation of interactive applications.
- What it does not show
- Abstract only; no limits are stated in the text seen. The abstract does not say which models acted as judges, or how the gains compare with run-to-run variation.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- How fast is AI advancing, and where is it heading?
- People
- Wayne Chi (Wrote it) · I-Hung Hsu (Wrote it) · Li Chen (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of October 14, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Interaction-as-a-Judge”, October 13, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-13-interaction-as-a-judge
Our notesWritten from the title, authors and the abstract as listed in the feed; the paper was not opened. The feed lists three authors and says there are six others, who are not named here. arXiv version 1; the listing does not say whether the paper has been peer reviewed. This is a methods paper on evaluating coding agents and is only loosely tied to safety, so it is kept as background.