The BaseResearcharXiv
Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents
In an arXiv preprint, Junfeng Jiao and six co-authors code 1331 publicly reported child-related AI incidents into a child-centered harm taxonomy. They argue model-level safety tests cannot by themselves anticipate the harm pathways most often documented for children, and propose measures beyond content filtering.
Why it matters
The same thing, for different readers.
What it means for you
A new study of 1331 reported incidents finds that harm to children from AI often comes from how products are deployed and governed, not from the model alone.
For lawmakers
Jiao and co-authors call for age assurance, consent verification for depiction, incident disclosure norms and auditing of developmental risk, beyond model-level safeguards.
For researchers
The study argues benchmarks, red-teaming and content filtering miss most documented child harm pathways, based on coded public incident reports, not a measured rate of harm.
In their own words
Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.
From the source’s own text.
“We find that harm concentrates in representational, dignity-related, and psychological categories, and co-occurs most often with deployment-context, governance, and misuse failures rather than model defects.”
“These findings suggest that model-level evaluations, while necessary, cannot by themselves anticipate the harm pathways most often documented for children.”
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- Curation and coding of 1331 publicly reported child-related AI incidents, annotated for internal and external harm types and the failure mode involved, using a child-centered harm taxonomy the authors built on prior taxonomies.
- What it found
- Harm concentrates in representational, dignity-related and psychological categories and co-occurs most often with deployment-context, governance and misuse failures rather than model defects; harm to one child is often linked to population-level harm, reaching children who never used the system.
- What it does not show
- Based on the abstract only: incidents are those publicly reported, so prevalence reflects reporting, not true rates. Full methods were not read.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- How is AI changing childhood, health and daily life? · Are AI companies putting stronger safeguards into practice?
- People
- Junfeng Jiao (Wrote it) · Kevin Chen (Wrote it) · Shrestha Mishra (Wrote it) · Abhejay Murali (Wrote it) · David Atkinson (Wrote it) · Shriti Raj (Wrote it) · Saleh Afroogh (Wrote it)
- What we read
- We read the abstract.
- Found by
- The Radar, run of October 27, 2026, via
arxiv-ai-safety-query(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents”, October 26, 2026. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2026-10-26-child-ai-incidents-1331