The BaseResearchLessWrong
LLMs are weirdly below chance on TASTE benchmark without a shortcut
According to the excerpt, Ziqian Zhong reports a shortcut in the TASTE benchmark, which asks models to pick the AI-safety research proposal experts prefer: proposals with more mechanistic-interpretability terms score lower, and a term-counting baseline beats all tested models.
Why it matters
The same thing, for different readers.
For researchers
The post suggests TASTE scores may reflect a bias against interpretability-related proposals rather than judgement of research quality, so results should be read with care.
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- The author tested a baseline on the TASTE benchmark that counts mechanistic-interpretability terms in each of two proposals and picks the one with fewer, with a coin flip on ties; the same benchmark was run on several LLMs, also on the non-interpretability half with rubric only.
- What it found
- The baseline reached 63.0% overall accuracy, above all LLMs tested. On the non-interpretability half, all models with ECI above 158 were below 50%; OpenAI models averaged 29.6%, falling as ECI rose (Pearson r = -0.74).
- What it does not show
- Not stated in the excerpt; a blog post that is not peer reviewed.
- Status
- A working paper: not yet peer reviewed
The record
- Questions
- How fast is AI advancing, and where is it heading?
- People
- Ziqian Zhong (Wrote it)
- What we read
- We have read only the description the publisher gives. No quotations until someone reads the source itself.
- Found by
- The Radar, run of November 16, 2026, via
lesswrong(tier 1: the source’s own channel or an official record) - Cite this
- LessWrong, “LLMs are weirdly below chance on TASTE benchmark without a shortcut”, November 15, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-15-taste-benchmark-below-chance
Our notesLight lane: written from the triage excerpt only, nothing fetched, no quotations. Numbers are the author's, from the excerpt. Evidence limits and the benchmark's authors are not in the excerpt.