The BaseResearcharXiv

Comprehension Audits to Mitigate Risks from Automated AI Research

In an arXiv preprint, Ronald J. Bodkin, Bahrad A. Sokhansanj and Gillian Hadfield propose 'comprehension audits': independent auditors check, without AI help, that the people behind AI-lab research understand it, and work pauses if they cannot. Their study of open-source AI projects finds more code and fewer human review comments per line. They note the audits are untested in practice.

Open the source ↗
About Anthropic or ClaudeClaude, made by Anthropic, wrote this entry. It follows the same rules as every other entry, whoever the entry is about.

Why it matters

The same thing, for different readers.

What it means for you

As AI writes more of the code for new AI, these researchers want proof that humans still understand what is being built, checked a bit like an oral exam.

For lawmakers

Proposes a gate for AI-built AI: failed audits pause development and dependent releases, and the authors see mandated audits by accredited external auditors as a promising medium-term step.

For researchers

Across open-source AI repositories, the authors report human inline review comments per thousand lines fell by roughly half from early 2024 to mid 2026, a proxy for attention.

The question it raises

Does human approval mean anything if the person approving no longer understands what they approve, and who is responsible when nobody does?

In their own words

Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.

From the source’s own text.

“We propose comprehension audits, a novel development-process assurance mechanism in which the responsible people explain R&D contributions to auditors to demonstrate understanding.”
Ronald J. BodkinThe paper's abstract, introducing its proposal.
Found word for word in the source
“Our analysis of leading open-source AI projects finds increased output of code with reduced human review commentary rates per line of code, with far lower rates for automated fleet accounts.”
Ronald J. BodkinThe abstract, summarising the paper's study of open-source AI code repositories.
Found word for word in the source
“Ultimately, approval of AI-written output without comprehension is meaningless.”
Ronald J. BodkinClosing the paper's problem statement, which cites a 2024 consensus statement that AI should not improve itself without explicit human approval.
Found word for word in the source
“We believe that regulations that mandate comprehension audits performed by accredited external auditors are a promising medium-term intervention to reduce risks from automated AI R&D.”
Ronald J. BodkinThe paper's final sentence, after calling for a pilot at a frontier lab.
Found word for word in the source
“Ultimately, comprehension audits provide an early warning and a safety brake. They don’t make automated AI R&D safe through alignment or ensure control.”
Ronald J. BodkinEnd of the authors' Limitations section.
Found word for word in the source

How the finding was reached

In our words, from the source. Weigh the finding by how it was made.

What was done
A proposal paper with one empirical analysis. The proposal: independent auditors test whether the people responsible for an AI-lab research contribution can explain it without AI help, and development of that contribution pauses until they can. The analysis: trends in code review on GitHub open-source AI repositories, covering 224 organizations (1,278 repositories from 179 prominent AI developers and 857 AI-topic repositories from 45 larger organizations), 54,000 to 162,000 merged pull requests per quarter, for Q1 2024 and Q4 2025 to Q2 2026. AI use was inferred with detection heuristics, and human review comments per line of code stand in for attention.
What it found
Traceable AI use appeared in 17% of pull requests by the end of the period (a floor). Regular contributors' code submitted rose by a median of 183% from Q1 2024 to Q2 2026. Human inline comments per line of code roughly halved: by June 2026 the rate was 36% of the Q1 2024 rate for AI-assisted pull requests and 46% for the others. Automated "fleet" accounts were reviewed far less (below 30% in the full population, against about 80% for pull requests without detected AI use).
What it does not show
The audit mechanism itself has not been tried: the authors cite no real-world experience, immature grading rubrics, and a small auditor ecosystem that depends on voluntary lab participation, and propose a pilot of at least 22 audits. The data cover open-source projects, which the authors say differ from proprietary lab research; AI-use detection gives only a floor; and comment rates are a proxy for attention, not a measure of understanding (the authors say changes in the metrics are not proof of reduced comprehension).
Status
A preprint: not yet peer reviewed

The record

Questions
What happens when AI helps build more capable AI? · Who can demand evidence, delay a release or stop an unsafe deployment? · Are AI companies putting stronger safeguards into practice?
People
Ronald J. Bodkin (Wrote it) · Bahrad A. Sokhansanj (Wrote it) · Gillian Hadfield (Wrote it)
What we read
We read the whole source.
Found by
The Radar, run of October 12, 2026, via arxiv-ai-safety-query (tier 1: the source’s own channel or an official record)
Cite this
arXiv, “Comprehension Audits to Mitigate Risks from Automated AI Research”, October 7, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-07-comprehension-audits-automated-ai-research
Our notesPreprint, not peer reviewed. Quotes are co-authored paper text; speaker set to the first author per the run rule. The saved HTML lists 'Gillian Hadfield' (the arXiv listing in triage gives 'Gillian K. Hadfield'). Affiliations listed as Independent, Institute for Law and AI, and Johns Hopkins University; the text does not map them to authors. Conflict: much of the paper's evidence is Anthropic's own reporting (e.g. Claude 'leads' 26% of Anthropic R&D work as of August 2026) and its proposal targets frontier labs including Anthropic. The empirical part covers open-source repositories only (224 organizations); the authors say open-source R&D differs from proprietary lab R&D, AI-use detection gives only a floor (17% of PRs traceable), and comment rates are a proxy for attention, not a measure of understanding. Stated limits: no real-world experience yet, immature grading rubrics, a small and lab-dependent auditor ecosystem; a pilot of at least 22 audits is proposed. The paper quotes Daniel Selsam, roon, Dario Amodei and an Anthropic employee secondhand; not quoted here.