The BaseResearcharXiv
Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
Tsinghua University researchers build TiGanEval, a 64-task benchmark where a simulated user states only part of what a software or research task needs, the rest being findable in the project. They report that eight frontier models lose 53 to 67 points of success versus a full specification, and four rounds of feedback do not close the gap.
Why it matters
The same thing, for different readers.
What it means for you
On a first attempt, AI agents in these tests rarely delivered everything a task needed when the request left details out, though they were findable in the project.
For lawmakers
Benchmarks that give agents complete instructions may overstate how they will perform for real users; the paper reports large drops once requirements are left unstated.
For researchers
Separates stated, inferred and true requirements so the gap can be scored; most failures mishandled evidence already in the project, and a plan-first prompt did not help.
The question it raises
Should an assistant do what was said or what was needed? The paper treats an instruction as evidence of a goal, and makes inferring the rest the agent's job.
In their own words
Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.
From the source’s own text.
“Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap.”
Found word for word in the source
“In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project.”
Found word for word in the source
“Treating the user instruction as evidence and inferring the true requirements from the environment thus remains the basis of performing well in plan mode; a simple plan-mode design cannot compensate for the missing ability of outcome alignment.”
Found word for word in the source
“We audit 100 failed Partial episodes spanning all twelve models and both domains, assigning each one primary failure mode (Figure 5). Five modes are ways of mishandling evidence about the true requirements, and together they cover 72 of the 100 episodes.”
Found word for word in the source
“To maintain evaluation stability, our user simulator generates subsequent instructions from authored observations and the interaction history. This design does not capture more complex user behaviors, such as messages that only partially express user beliefs or the proactive, incremental disclosure of requirements.”
Found word for word in the source
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- A benchmark of 64 tasks (40 software engineering, 24 research), each with five requirements that can be found in the project's code, records or runnable experiments. The user's opening instruction states none of the five in 39% of tasks and never more than two. A simulated user (GPT-5.6-Terra) reacts to the visible consequence of each failed submission. Twelve models (eight frontier, four small open-weight) were run under three set-ups: partial instruction (told to treat it as evidence), literal (execute it as written) and full specification, with up to four submissions under the partial instruction. Tasks were drafted by an AI pipeline (Claude Opus 5 through the Claude Code SDK) and reviewed by human experts. Separately, annotators labelled 100 sampled real coding sessions from the SWE-chat dataset.
- What it found
- All eight frontier models lost 53 to 67 points of first-submission success (all five requirements met) when given the partial instruction instead of the full specification; GPT-5.6-Sol fell from 64.1% to 10.9%. With up to four submissions and user feedback, partial-instruction success reached 23.4% to 62.5% across the eight, still below each model's full-specification first-submission result (60.9% to 82.8%). Treating the instruction as evidence beat literal execution by 15.7 to 34.1 score points at the first submission. A plan-first harness never raised first-submission success on the three models tested. In an audit of 100 failed partial-instruction episodes, 72 mishandled evidence available in the project (25 replaced the documented rule with a self-authored one; 21 ran a replay that printed the violation and then made no fix) and 24 ended in behavioural collapse. In the 100 real coding sessions, 56% of opening instructions left out requirements that later proved necessary, and 52% of the useful follow-up requirements could have been inferred from the project.
- What it does not show
- The authors say cost limited the size of the task set (64 tasks) and that they did not test training models for this ability. Their simulated user writes follow-up instructions only from pre-written descriptions of what went wrong, so it does not capture messages that express beliefs only partly or requirements disclosed gradually. The 'literal' set-up is a prompt-defined control, not a measure of how people use agents. Tasks are synthetic (AI-drafted, human-reviewed) and use one simulated user; the user simulator, GPT-5.6-Terra, is also one of the models tested, which the paper does not discuss (it reports a hand check in which 96.0% of 150 regenerated simulator messages conveyed only what the observation allowed). Whether real opening instructions are incomplete was judged by human annotators on 100 sampled sessions.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- How fast is AI advancing, and where is it heading? · Can AI systems act beyond what people intended or allowed?
- People
- Kaiming Liu (Wrote it) · Yuxuan Liu (Wrote it) · Jinrui Ju (Wrote it) · Minyuan Ruan (Wrote it) · Xuanyu Lei (Wrote it) · Peng Li (Wrote it) · Yang Liu (Wrote it)
- What we read
- We read the whole source.
- Found by
- The Radar, run of October 14, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents”, October 13, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-13-instructions-are-evidence-not-objectives
Our notesPreprint, not peer reviewed (arXiv v1, 13 Oct 2026). Full text read from the arXiv HTML version, not the PDF, so no lost-ligature problem. Quotes are co-authored paper text; speaker set to the first author per the run rule. The abstract's 52% is 52.49% in Appendix B (annotators' judgement on 100 sampled SWE-chat sessions). Claude Opus 5 appears only as the tool that drafted the benchmark tasks (Section 4.3), not as a tested model, so conflict is false and anthropic is left out of orgs. The simulated user, GPT-5.6-Terra, is also one of the twelve tested models. The 53 to 67 point loss is stated for the eight frontier models only; the four small open-weight models mostly collapsed into loops (14 of 18 audited failures) and their near-zero scores reflect a capability floor, as the authors say. The authors' affiliations are given as three Tsinghua units without a per-author mapping, so people carry no titles. Yang Liu and Peng Li are common names: ids are plain and not disambiguated.