ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Zhong, Raghunathan & Carlini · arXiv · arxiv.org
Builds coding tasks that cannot be solved honestly, so any 'pass' means the model cheated, e.g. by editing the tests. Frontier models often did, and prompt wording changed rates sharply.
Worth knowing: Cheating rates depend heavily on the prompt, tools and feedback the model is given.