GPT-6.2 Astra scores below chance on TASTE; GPT-6 Astra and Fable 5.2 near chance

In a LessWrong post, Sarah Wang reports running the TASTE benchmark, which asks models to pick the better of two AI safety research proposals, on three newer models. Wang finds GPT-6.2 Astra below chance and the other two near chance.

Why it matters

The same thing, for different readers.

For lawmakers

Claims that AI is already speeding up AI research are hard to check; this result shows one proxy for research judgment where newer models do not do better than chance.

For researchers

Wang's run adds three newer models to TASTE and finds none above chance, which questions whether the benchmark picks up recent AI R&D uplift or only its own quirks.

In their own words

Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.

From the source’s own text.

“Overall, none of the models tested score significantly above chance, and GPT-6.2 Astra scores significantly below it.”
Sarah WangWang's summary of the results after running two of the four TASTE variations on three newer models.
Found word for word in the source
“Any uplift in AI R&D does not seem to show up in TASTE in its current form.”
Sarah WangWang's closing judgment, after listing the limits of the benchmark and the small number of pairs.
Found word for word in the source

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
A single author ran the TASTE benchmark (92 pairs of synthetic safety research proposals with gold labels set by researchers) on GPT-6.2 Astra, GPT-6 Astra and Fable 5.2, in two of the paper's four variations.
What it found
Astra 6.2 agreed with gold labels 35% of the time, Astra 6 41% and Fable 5.2 53%. With 92 pairs, the post puts the 95% interval for random guessing at 40-60%.
What it does not show
Small benchmark (92 pairs), only safety proposals, only two of four variations run, and a separate post finds that choosing the proposal with fewer mech-interp terms beats expert agreement.
Status
A report

The record

Questions
What happens when AI helps build more capable AI? · How fast is AI advancing, and where is it heading?
People
Sarah Wang (Wrote it)
What we read
We read the whole source.
Found by
The Radar, run of November 19, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “GPT-6.2 Astra scores below chance on TASTE; GPT-6 Astra and Fable 5.2 near chance”, November 18, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-18-astra-below-chance-taste
Our notesSingle-author forum post, not peer reviewed. Relation to the Nov 15 entry is inferred from the post's own mention of a post about TASTE limits; a person may confirm it is the same one.