Safety Cases Have an Agent Foundations Problem

In a LessWrong post, Austin Long argues that it is unclear how safety-engineering methods apply to highly capable AI, because they run into scientific problems also studied under Agent Foundations. Long draws on experience in a University of York safety assurance group.

Why it matters

The same thing, for different readers.

For lawmakers

Austin Long argues that regulators asking for safety cases for advanced AI would be asking for something engineers do not yet know how to produce.

For researchers

Austin Long says the scientific problems blocking safety assurance for advanced AI are the same ones studied under Agent Foundations.

The question it raises

Austin Long poses whether AI safety should be approached as engineering or as science, which Long describes as a growing crux between worried and optimistic camps.

The record

Questions
Are AI companies putting stronger safeguards into practice? · Can AI systems act beyond what people intended or allowed?
People
Austin Long (Wrote it, PhD student, University of York safety assurance community (per the post))
What we read
We have read only the description the publisher gives. No quotations until someone reads the source itself.
Found by
The Radar, run of November 17, 2026, via lesswrong (tier 1: the source’s own channel or an official record)
Cite this
LessWrong, “Safety Cases Have an Agent Foundations Problem”, November 17, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/lesswrong-2026-11-17-safety-cases-agent-foundations
Our notesLight lane: written from the triage excerpt only, which is cut off. The author's title is as the author describes it. Long's pronoun is unknown, so none is used.