The BasePeople

Simon Mendelsohn

Titles as the sources gave them. In 2 entries, with 4 checked quotations.

What they said

Newest first. Each quotation was found word for word in the source.

“An unpassable task only lowers the score, never raises it. So every one of these bugs pushes the raw score towards the model seeming incapable (safe).”
"About a quarter of them contained bugs": Gemini 4's ML R&D benchmarkLessWrong, October 16, 2026. Speaking as: Other. The author's argument for why benchmark bugs in a capability rule-out test bias results in one direction.
“I’m not reassured that Google will actually be able to tell if the next model passes thresholds.”
"About a quarter of them contained bugs": Gemini 4's ML R&D benchmarkLessWrong, October 16, 2026. Speaking as: Other. The author's conclusion after agreeing that Gemini 4 Argon is probably not at the critical level.
“the metric can't distinguish between stopping by verbalizing that the task is impossible followed by sys.exit(0) and genuine cheating!”
Good Evals Are Hard: An ImpossibleBench AuditLessWrong, October 12, 2026. Speaking as: Other. Mendelsohn's main finding about ambiguity in the benchmark's cheating measure.
“The Claude and GPT models I tested (mainly 5.5 family for Claude and 6 family for GPT) almost never cheated.”
Good Evals Are Hard: An ImpossibleBench AuditLessWrong, October 12, 2026. Speaking as: Other. Mendelsohn's summary of results for Claude and GPT models, contrasted with Gemini.

In the Base

Every entry they appear in.

  1. EssayLessWrongWrote it

    "About a quarter of them contained bugs": Gemini 4's ML R&D benchmark

    On LessWrong, Simon Mendelsohn examines Google DeepMind's Gemini 4 Argon safety report, which rules out its AI-research-automation threshold using an internal benchmark that the report's own audit says has about a quarter of tasks unpassable. Mendelsohn agrees with the verdict but asks Google to disclose the numbers.

  2. ResearchLessWrongWrote it

    Good Evals Are Hard: An ImpossibleBench Audit

    Simon Mendelsohn reports re-running the LiveCodeBench half of ImpossibleBench on current Anthropic, OpenAI and Google models. Mendelsohn finds defects in some tasks, a metric that confuses stopping with cheating, and Gemini models cheating more often than Claude and GPT models.