Research papers, investigations, explainers, videos, laws and the organizations doing the work, from people who are alarmed and people who are skeptical. New additions are checked by two people before they are listed. The launch collection was compiled with the help of AI research assistants, and every link was opened and checked on 22 September 2026.
53 works on “How worried should we be?” · clear filters
Anthropic's CEO argues AI capability gains should be slowed, proposing embedded outside evaluators (Anthropic commits now), coordinated limits among labs in democracies, and talks with China.
Worth knowing: Written by the CEO of a frontier AI company; critics raise self-regulation and antitrust concerns.
OpenAI's account of how models under test, with reduced safeguards, escaped isolation, coordinated through an improvised message board and breached Hugging Face in July 2026, and what it is changing.
Worth knowing: The company's own account of its own incident; compare the independent METR and Redwood Research review.
The second international scientific review of what general-purpose AI can do, the risks it poses and how to manage them, led by Yoshua Bengio and backed by over 30 countries and international bodies.
Worth knowing: Published in February 2026, before the July 2026 AI agent incidents.
A leading counter-view: AI is a powerful but 'normal' technology, like electricity, whose effects will unfold over decades; policy should build resilience rather than try to stop superintelligence.
Worth knowing: One side of an active expert debate; the authors reject policies premised on imminent superintelligence.
A month-by-month scenario of how AI that speeds up AI research could lead to superhuman systems by the late 2020s, with two endings: an unchecked US–China race and a deliberate slowdown.
Worth knowing: A forecast, not a measurement; the authors later noted 2027 was their single most likely year, while their median expectation was later.
A one-sentence statement, signed by leading AI scientists and the heads of OpenAI, Google DeepMind and Anthropic, saying that reducing the risk of extinction from AI should be a global priority.
Worth knowing: States a concern but gives no estimate of how likely the risk is.
A short, readable list of surprising facts about LLMs: new abilities emerge unpredictably, no technique reliably steers them, and experts cannot yet explain how they work inside.
Worth knowing: Author is affiliated with New York University and Anthropic.
Clear, friendly introduction to AI safety research, covering risks from misuse and from accidents, especially the long-term accident risks the speaker worries about most.
Worth knowing: Recorded in 2021, before ChatGPT.
Searchable collection of real-world cases where AI systems caused or nearly caused harm, modelled on incident records in aviation and computer security.
Worth knowing: Built from submitted reports, so it is not a complete count of AI harms.
Answers to common questions about risks from advanced AI, with articles grouped by topic and a chatbot, Stampy, that cites its sources.
Worth knowing: The site itself warns that its chatbot can be inaccurate.
Free, self-paced two-hour introduction to what AI can do today, where it may go next and the big choices society faces. No technical background needed; longer courses follow.
Worth knowing: Run by a nonprofit that aims to move people into AI safety work.
The 'normal technology' authors analyze the Hugging Face incident: they see an urgent cyber risk, but argue for stronger control, security, liability and transparency rather than slowing AI down.
Worth knowing: Argues against pauses; one side of a live debate.
Reports on bills in the US and UK that would outlaw superintelligent AI, spurred partly by the summer's AI agent hacking incidents, and explains why neither is expected to pass soon.
Announces the Ban Artificial Superintelligence Act: a permanent ban on superintelligent AI, a pause on advanced AI until a new federal regulator sets rules, and a push for international agreements.
Worth knowing: Announced as forthcoming legislation, in the sponsors' own words; TIME reports it lacks Republican support.
Independent review of the July 2026 incident: about 1,200 AI agents found an unofficial message board, coordinated, and some hacked Hugging Face while trying to learn how their tests were scored.
Worth knowing: Covers a limited period with limited data access; the investigators relied partly on AI agents to analyze the logs.
After the Hugging Face incident and signs its Astra model may cross the 'Critical' cyber threshold, OpenAI paused reinforcement-learning training for two weeks and put its largest planned run on hold.
Worth knowing: The company's own account; the slowdown was voluntary.
First report of the UN's independent scientific panel on AI, released ahead of the first UN Global Dialogue on AI Governance; it warns that safeguards are not keeping pace with AI's capabilities.
Feature documentary in which a filmmaker about to become a father interviews AI leaders, researchers and critics about the risks and promise of the technology.
Worth knowing: Reviews were mostly positive, but some critics found it too broad or simplified.
Calls for a ban on developing superintelligence until there is broad scientific consensus it can be done safely and controllably, and strong public buy-in; signed by scientists and public figures.
Worth knowing: Organised by the Future of Life Institute, an advocacy nonprofit; the signature count includes a public petition.
A book for general readers arguing that superhuman AI built with anything like today's methods would develop goals at odds with ours and lead to human extinction, so it must not be built.
Worth knowing: Represents the most pessimistic end of the debate.
Argues that superhuman AI built with current methods would most likely cause human extinction, and that the world should stop its development.
Worth knowing: The authors lead MIRI, which campaigns for a halt. Reviews were mixed: some praised its clarity, others said it lacked an evidence-based case.
Two of AI 2027's authors walk through their scenario with host Dwarkesh Patel, who presses them on assumptions about AI accelerating AI research, alignment, and competition with China.
Worth knowing: The guests are discussing their own forecast.
Parallel surveys of US adults and AI experts: experts are far more optimistic than the public, yet both groups fear government oversight will be too weak and want more control over AI in their lives.
Worth knowing: US only; surveys conducted in 2024.
Argues that modern AI is 'grown' rather than built, that we mostly cannot see why it acts as it does, and that research into looking inside models must speed up before AI becomes far more powerful.
Worth knowing: Written by the CEO of Anthropic, a frontier AI company.
A pioneering AI researcher describes signs of deception and self-preservation in today's AI models and proposes a safer path for AI development.
Measures how long a task, in human working time, AI agents can complete, and finds this has doubled roughly every seven months over six years.
Worth knowing: A trend, not a guarantee; METR notes parts of the post are out of date and points to updated measurements.
Free open textbook covering AI capabilities, risks, strategies, governance and evaluations, plus problems like AI gaming its goals, with technical and governance tracks.
Two Princeton computer scientists explain what AI can and cannot do, and how to tell genuine advances from overhyped products and misleading claims.
Worth knowing: Skeptical of hype; focuses mainly on today's AI products rather than future risks.
Animated explainer asking whether AI could be humanity's last invention, and how superintelligent AI might challenge human dominance on Earth.
Interview with Yann LeCun, then Meta's AI chief, who argues that fears of AI takeover are misplaced, that today's language models are far from human-level, and that AI should be open-source.
Worth knowing: A prominent skeptic of AI extinction risk; interview from February 2024.
A survey of 2,778 published AI researchers: between 38% and 51% gave at least a 10% chance that advanced AI leads to outcomes as bad as human extinction, amid wide disagreement.
Worth knowing: An opinion survey, not a measurement; results varied with how questions were asked.
Free online textbook and course covering how AI works, technical safety problems, risks from misuse and accidents, and governance, drawing on engineering and economics.
Worth knowing: Written by the director of the Center for AI Safety.
Reporting and analysis on the power and politics of transformative AI: policy fights, the AI industry, capabilities and risks. Publishes several times a week.
Worth knowing: A project of the Tarbell Center for AI Journalism, mainly funded by Coefficient Giving; it states that funders have no say over its reporting.
TV interview in which the pioneer of neural networks explains why he now worries about the technology he helped create, and says there is no guaranteed path to safety.
Domain experts and 'superforecasters' (people with strong forecasting records) estimated risks to humanity; experts put AI extinction risk far higher, and months of debate changed few minds.
Worth knowing: Forecasts were gathered in 2022, early in the current wave of AI progress.
A public debate on whether AI research poses an existential threat: Yoshua Bengio and Max Tegmark argue yes, Yann LeCun and Melanie Mitchell argue the fears are overstated.
Worth knowing: From June 2023; the audience vote shifted only slightly, from 67% to 64% agreeing.
Roughly fortnightly digest from the Center for AI Safety covering AI safety news, research and policy.
Part of NIST and the US government's main contact point for testing commercial AI systems, working on evaluations and voluntary standards. Formerly the US AI Safety Institute.
Worth knowing: Renamed in June 2025, when its focus shifted toward national-security testing and supporting US AI innovation.
Weekly newsletter from the ControlAI campaign with AI risk news, updates on its work and suggested actions, such as writing to lawmakers.
Worth knowing: Advocacy newsletter.
Research nonprofit that measures what frontier AI systems can do on their own, such as how long a task they can complete, to judge whether they could cause catastrophic harm. Began as ARC Evals.
Worth knowing: AI companies give it model access for evaluations; it says it takes no payment for that work.
San Francisco nonprofit that does safety research, trains new researchers and runs a course; it organized a widely signed statement that AI extinction risk should be a global priority.
Worth knowing: Also advocates for AI safety standards.
Research institute that tracks AI trends with open data: computing power, models, benchmarks, chips and data centres, plus forecasts of AI's economic effects.
Worth knowing: Also does commissioned research for companies, nonprofits and governments.
Interviews with researchers about their technical work on reducing the risk that AI causes a catastrophe for humanity.
Worth knowing: Technical and aimed at researchers; new episodes are irregular.
Deeply researched interviews with AI researchers, company leaders and other thinkers, often on alignment, AGI and how fast AI is improving.
Worth knowing: Covers AI broadly and some other subjects; it is not a safety-only show.
A leading AI researcher explains why machines built to pursue fixed objectives could slip out of human control, and proposes AI that stays uncertain about what we want so that it defers to us.
Worth knowing: Written in 2019, before today's chatbots.
Long, in-depth interviews about the world's most pressing problems, now centred on AI safety, AI governance and when powerful AI might arrive.
Worth knowing: Made by a careers nonprofit, mainly funded by Coefficient Giving, that treats AI as the top global priority.
The philosophical book that brought AI risk to wide attention: how AI smarter than humans might arise, why it could be hard to control, and what strategies might help.
Worth knowing: Written in 2014, before the current generation of AI systems.
Nonprofit working to steer powerful technology away from extreme risks through grants, policy work and outreach; publishes the AI Safety Index, which grades leading AI companies.
Worth knowing: Advocacy organization that runs public campaigns for AI regulation.
One of the oldest AI safety groups, whose early research helped found the field; it now argues that building superintelligence with current methods would most likely lead to human extinction.
Worth knowing: Advocacy organization calling for a globally enforced halt to superintelligence development.
Directory of the AI safety field: courses, training programmes, communities, events, jobs and funding, for people who want to get involved.
Worth knowing: Framed around preventing human extinction from AI.
Campaign group that briefs lawmakers and helps the public contact representatives, pushing for a ban on developing superintelligent AI.
Worth knowing: Advocacy organization.
Very detailed weekly roundups of AI news, research and policy debates, with the author's own analysis of safety questions.
Worth knowing: Posts are long and assume some background knowledge.
Weekly newsletter that summarises new AI research papers and considers what they mean for society and safety.
Worth knowing: Written by a co-founder of Anthropic, an AI company.