LLMs are weirdly below chance on TASTE benchmark without a shortcut

According to the excerpt, Ziqian Zhong reports a shortcut in the TASTE benchmark, which asks models to pick the AI-safety research proposal experts prefer: proposals with more mechanistic-interpretability terms score lower, and a term-counting baseline beats all tested models.

Why it matters

The same thing, for different readers.

For researchers

The post suggests TASTE scores may reflect a bias against interpretability-related proposals rather than judgement of research quality, so results should be read with care.

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
The author tested a baseline on the TASTE benchmark that counts mechanistic-interpretability terms in each of two proposals and picks the one with fewer, with a coin flip on ties; the same benchmark was run on several LLMs, also on the non-interpretability half with rubric only.
What it found
The baseline reached 63.0% overall accuracy, above all LLMs tested. On the non-interpretability half, all models with ECI above 158 were below 50%; OpenAI models averaged 29.6%, falling as ECI rose (Pearson r = -0.74).
What it does not show
Not stated in the excerpt; a blog post that is not peer reviewed.
Status
A working paper: not yet peer reviewed

The record

Questions
How fast is AI advancing, and where is it heading?
People
Ziqian Zhong (Wrote it)
What we read
We have read only the description the publisher gives. No quotations until someone reads the source itself.
Found by
The Radar, run of November 16, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “LLMs are weirdly below chance on TASTE benchmark without a shortcut”, November 15, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-15-taste-benchmark-below-chance
Our notesLight lane: written from the triage excerpt only, nothing fetched, no quotations. Numbers are the author's, from the excerpt. Evidence limits and the benchmark's authors are not in the excerpt.