The BasePeople

Simon Mendelsohn

Titles as the sources gave them. In 1 entry, with 2 checked quotations.

What they said

Newest first. Each quotation was found word for word in the source.

“the metric can't distinguish between stopping by verbalizing that the task is impossible followed by sys.exit(0) and genuine cheating!”
Good Evals Are Hard: An ImpossibleBench AuditLessWrong, October 12, 2026. Mendelsohn's main finding about ambiguity in the benchmark's cheating measure.
“The Claude and GPT models I tested (mainly 5.5 family for Claude and 6 family for GPT) almost never cheated.”
Good Evals Are Hard: An ImpossibleBench AuditLessWrong, October 12, 2026. Mendelsohn's summary of results for Claude and GPT models, contrasted with Gemini.

In the Base

Every entry they appear in.

  1. ResearchLessWrongWrote it

    Good Evals Are Hard: An ImpossibleBench Audit

    Simon Mendelsohn reports re-running the LiveCodeBench half of ImpossibleBench on current Anthropic, OpenAI and Google models. Mendelsohn finds defects in some tasks, a metric that confuses stopping with cheating, and Gemini models cheating more often than Claude and GPT models.