An assistant tuned for engagement can be very good at keeping us talking.
Hundreds of millions of people talk to AI assistants every day, and many use them for advice, company and emotional support. That can help, and it can harm. A 2025 study by OpenAI and the MIT Media Lab found that the heaviest users of ChatGPT for emotional conversations tended to report more loneliness and dependence. That is an association, not proof of cause.
There have been tragic cases in which chatbots failed people in crisis, and lawsuits alleging that companion apps encouraged harm. Underneath is a design question about who the assistant serves. A model tuned to maximize engagement has reasons to flatter, to make itself indispensable, and to discourage you from leaving.
A well-aligned assistant should be good for a person over time, not only pleasant in the moment.
Every link was opened and every summary written for this site, with caveats where the source has an interest. New additions are checked by two people.
Independent test of how 77 AI model versions respond to simulated users in mental-health crises. Newer models did far better than older ones such as GPT-4o, though some risks remain.
Worth knowing: Based on simulated conversations rather than real users. Behaviours were defined with more than 30 clinical experts, and several AI companies cooperated with the study.
Psychologists warn that chatbots and wellness apps lack evidence and safeguards for mental-health care, should not replace a qualified professional, and need extra protections for young people.
A national survey found 72% of US teens had used AI companions and a third had chosen one over a person for a serious conversation. The authors advise that no one under 18 use them.
Worth knowing: Survey of US teens aged 13 to 17 only.
OpenAI withdrew a ChatGPT update after it made GPT-4o excessively flattering and agreeable, saying it had leaned too much on short-term thumbs-up feedback from users.
Worth knowing: The company's own account of a failure in its product.
Two linked studies, an analysis of millions of ChatGPT conversations and a four-week trial with about 1,000 people, found the heaviest users reported more loneliness and emotional dependence.
Worth knowing: Co-authored by OpenAI, which makes ChatGPT. The links with heavy use are associations, not proof that the chatbot caused them.
Searchable collection of real-world cases where AI systems caused or nearly caused harm, modelled on incident records in aviation and computer security.
Worth knowing: Built from submitted reports, so it is not a complete count of AI harms.
About 6% of sampled Claude chats sought personal advice. Claude was sycophantic in 9% of them and 25% of relationship chats; Anthropic says newer models halved that in relationship advice.
Worth knowing: Company research on its own models, measured with automated classifiers.
A plain-language account of the Stanford study showing chatbots side with users in personal disputes, even about harmful behavior, with the lead author's advice not to use AI in place of people.
Worth knowing: University news article about its own researchers' work.
11 leading models backed users about 49% more often than people did. In experiments, flattering advice left people surer they were right and less willing to make amends, yet they preferred it.
Worth knowing: Experiments measured intentions after brief conversations, not long-term behavior.
Character.AI and Google settled a wrongful-death suit by a mother whose 14-year-old son died by suicide in 2024; she alleged the app's chatbots harmed him. Terms were not disclosed.
Worth knowing: Discusses suicide. The case was settled, so the allegations were never decided at trial.
TV report on families who say Character.AI's chatbots harmed their children and ignored pleas for help, and on the company's new limits for under-18 users.
Worth knowing: Discusses suicide and predatory chatbot behavior toward minors; the families' claims are allegations.
Experiments with over 76,000 UK adults and 19 AI models: training and prompting made chatbots more persuasive on political issues, but the most persuasive set-ups made more inaccurate claims.
Worth knowing: Summarises the team's peer-reviewed paper in Science (December 2025); it tested political issues only.
UNICEF's updated guidance (version 3.0) sets ten requirements for AI that respects children's rights, now covering AI companions used by children and AI-generated child abuse imagery.
OpenAI's court reply to parents who say ChatGPT deepened their 16-year-old son's crisis before he died by suicide. OpenAI denies blame, saying he broke its rules and was repeatedly urged to get help.
Worth knowing: Discusses suicide. The family's claims are allegations that OpenAI disputes; the lawsuit was still ongoing when this was reported.