The BaseResearcharXiv

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

According to the abstract excerpt, the authors argue that existing tests miss whether deployed agents follow rules on pricing, privacy or confirmation during valid tool use. They propose PROBE, which trains auditor models to talk with agents as realistic users and probe for violations.

Open the source ↗

Why it matters

The same thing, for different readers.

For lawmakers

Points to a gap between capability benchmarks and checks on whether agents obey the rules set for them, relevant to auditing deployed systems.

For researchers

Frames agent auditing as a two-player game with a reward balancing violation discovery, plausibility, progress and diversity.

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
An algorithm for training LLM-based auditors that interact with deployed agents in multi-turn, tool-mediated dialogue, formalised as a two-player Markov game.
What it found
Not visible in the excerpt, which ends before the results.
What it does not show
Judged from a truncated abstract; the full paper was not read.
Status
A preprint: not yet peer reviewed

The record

Questions
Are AI companies putting stronger safeguards into practice? · Can AI systems act beyond what people intended or allowed?
People
Kshitij Mishra (Wrote it) · Abhijith Sharma (Wrote it) · Nils Lukas (Wrote it) · Salem Lahlou (Wrote it)
What we read
We have read only the description the publisher gives. No quotations until someone reads the source itself.
Found by
The Radar, run of October 16, 2026, via arxiv-ai-safety-query (tier 1: the source’s own channel or an official record)
Cite this
arXiv, “PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents”, October 15, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-15-probe-auditing-agent-policy-compliance
Our notesLight lane: written from the triage excerpt only; the source was not fetched and no quotations are given.