Kaiming Liu

“Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap.”

“In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project.”

“Treating the user instruction as evidence and inferring the true requirements from the environment thus remains the basis of performing well in plan mode; a simple plan-mode design cannot compensate for the missing ability of outcome alignment.”

“We audit 100 failed Partial episodes spanning all twelve models and both domains, assigning each one primary failure mode (Figure 5). Five modes are ways of mishandling evidence about the true requirements, and together they cover 72 of the 100 episodes.”

“To maintain evaluation stability, our user simulator generates subsequent instructions from authored observations and the interaction history. This design does not capture more complex user behaviors, such as messages that only partially express user beliefs or the proactive, incremental disclosure of requirements.”