The BaseResearcharXiv
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR researchers report a randomized trial in which 16 experienced open-source developers completed 246 real tasks in mature repositories, each task randomly allowing or barring early-2025 AI tools. Tasks took 19% longer with AI allowed, although developers had expected a 24% saving. The authors caution against generalizing beyond this setting.
Why it matters
The same thing, for different readers.
What it means for you
Skilled workers can feel faster with AI while measuring slower, so impressions can mislead, though this trial covers one kind of expert work, not jobs in general.
For lawmakers
The authors say field experiments with robust measures matter more than relying only on expert forecasts or user surveys; experts predicted time savings near 40%.
For researchers
Tasks were defined before random assignment, which fixes the outcome measure in advance; but the estimate rests on 16 developers and February-June 2025 tools.
In their own words
Every quotation was found, word for word, in the text we saved from the source. That shows the words are there; it does not make them true, or prove the speaker meant them as quoted.
From the source’s own text.
“Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down.”
Found word for word in the source
“Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%.”
Found word for word in the source
“This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter).”
Found word for word in the source
“Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.”
Found word for word in the source
“That said, many of the factors we find evidence for contributing to slowdown are specific to the setting we study—these results do not imply that current AI systems are not useful in many realistic, economically relevant settings.”
Found word for word in the source
How the finding was reached
In our words, from the source. Weigh the finding by how it was made.
- What was done
- A randomized controlled trial. Sixteen experienced open-source developers (about five years on their own repository on average; repositories averaging 23,000 stars and 1.1 million lines of code) completed 246 real tasks of about two hours each, with AI tools of the February-June 2025 frontier. Each task was randomly assigned to allow or disallow AI tools; developers chose their tools (mostly Cursor Pro with Claude 3.5 or 3.7 Sonnet) and were paid $150 an hour. Time to finish was self-reported; screen recordings were used to check compliance and, for a subset, the reported times.
- What it found
- Allowing AI increased completion time by 19%. Developers had forecast a 24% reduction beforehand and estimated a 20% reduction afterwards; economics experts predicted 39% shorter and machine-learning experts 38% shorter. In per-developer estimates, 75% of developers were slowed down. On a subset of recorded tasks, developers with AI allowed spent less time actively coding and searching for information and more time prompting, waiting for and reviewing AI output. The authors name five factors they judge to contribute, including over-optimism about AI, high familiarity with the repositories, and large, complex codebases, and call this analysis speculative.
- What it does not show
- Sixteen developers working on mature repositories they knew well, with February-June 2025 tools; developers knew which condition applied. The authors say the slowdown does not imply AI fails to help in other settings (small new projects or unfamiliar codebases may see speedups), that better or later systems could speed up developers even here, and that experimental artifacts cannot be entirely ruled out, including the chance that developers who rely heavily on AI were less likely to take part. They call the factor analysis suggestive only, because the data are not powered for multiple comparisons, and they study software development only.
- Status
- A preprint: not yet peer reviewed
The record
- Questions
- What happens to work, income and the economy? · How fast is AI advancing, and where is it heading?
- People
- Joel Becker (Wrote it, Model Evaluation & Threat Research (METR)) · Nate Rush (Wrote it, Model Evaluation & Threat Research (METR)) · Beth Barnes (Wrote it, Model Evaluation & Threat Research (METR)) · David Rein (Wrote it, Model Evaluation & Threat Research (METR))
- What we read
- We read the whole source.
- Found by
- A research pass: work, income and the economy, gathered October 13, 2026, via
research-work(tier 1: the source’s own channel or an official record) - Cite this
- arXiv, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, July 25, 2025. Entry in the Ctrl AI Base, https://ctrlai.com/base/arxiv-2025-07-25-ai-slowed-experienced-open-source-developers
Our notesVersion read: arXiv v2, 25 Jul 2025 (v1 was submitted 12 Jul 2025, which is the triage date). The arXiv abstract page says 20 properties were examined; the v2 paper says 21 (5 contribute, 10 unclear, 6 against), and this entry follows the paper. The saved text does not mention METR's February 2026 follow-up. Read separately, not part of the saved text: METR's post of 24 Feb 2026 (https://metr.org/blog/2026-02-24-uplift-update/) says a newer experiment begun in August 2025 gives an unreliable signal of the current effect, mainly because more developers declined to take part if it meant working without AI; it reports raw estimates that show a speedup with wide intervals, says developers are likely more sped up in early 2026 than in early 2025, and says METR is changing its study design. An editor may want that post as its own entry. The tools used in the trial included Anthropic's Claude 3.5 and 3.7 Sonnet, but the paper is not about Anthropic.