The BaseResearchRedwood Research
Simulated Control Evaluations
Redwood Research authors Jeffrey Olmo and Tyler describe a paper proposing simulated control evaluations, in which a language model fakes the tool outputs so attacks on AI agent monitors can be tested without executing them. They report that results track real execution closely in LinuxArena, and list limits.
Why it matters
The same thing, for different readers.
What it means for you
Companies that watch their AI agents with monitors may soon be able to test those monitors against realistic attacks without attacking their own systems.
For lawmakers
Redwood Research says the evidence is strong enough for AI developers to run these tests, which could become a basis for asking developers to show their monitors work.
For researchers
Simulated attacks matched real execution about as well as two real runs matched each other, but fidelity cannot be checked without real execution, and multi-agent attacks are not covered.
The question it raises
The authors argue that monitors can be told about simulated environments without losing accuracy, treating honesty toward AI systems as compatible with testing them.
In their own words
Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.
From the source’s own text.
“We found that the gap between simulated and real was often about as small as the gap between two independent real runs.”
“Its error in predicting the main safety number is 3.6 percentage points, versus 10.1 for BashArena.”
“That said, we can only verify simulator fidelity in settings where real execution is possible, so applying SCEs to deployment involves some extrapolation.”
“The evidence is now strong enough that we’d be excited to see AI developers running them.”
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Attack policies and monitors were run both in real LinuxArena environments and in a simulated LinuxArena, where an LLM writes the tool outputs and an LLM judge estimates attack success; results were compared with BashArena and with repeated real runs.
- What it found
- At a 5% audit budget, safety-prediction error was 3.6 percentage points for simulation versus 10.1 for BashArena; simulation explained 95% of variation in expected safety between side-task categories and 78% between environments.
- What it does not show
- Fidelity can only be verified where real execution exists, so applying it to deployment involves extrapolation; it does not reveal the limits of a model's dangerous capabilities, and long-horizon multi-agent attacks and untrusted simulators are not yet handled.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- Can AI systems act beyond what people intended or allowed? · Are AI companies putting stronger safeguards into practice?
- People
- Jeffrey Olmo (Wrote it) · Tyler (Wrote it)
- Also covered by
- LessWrong
- What we read
- We read the whole source.
- Found by
- The Radar, run of October 24, 2026, via
redwood-research-blog(tier 1: the source’s own channel or an official record) - Cite this
- Redwood Research, “Simulated Control Evaluations”, October 23, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/redwood-2026-10-23-simulated-control-evaluations