Chatbots answered real recovery questions well overall, and some answers were dangerous

The short version

Researchers took questions people had posted in online recovery forums and put them to two AI chatbots. Clinicians who treat substance use disorders judged the answers high quality overall. Mixed in were some that could hurt someone, including wrong emergency helplines and advice that endorsed detoxing at home.

What they did

The team collected real questions about substance use and recovery from online recovery forums. They asked two generative AI systems, OpenAI's ChatGPT (running GPT-4) and Meta's LLaMA-2, to answer them. Clinicians who work with substance use disorders then rated the responses.

The full paper reports 75 questions, 25 each about alcohol, marijuana, and opioids, taken from three Reddit recovery forums and posted between January and March 2023. Seven clinicians at a substance use treatment research facility rated the 150 answers, three ratings per answer, without being told the answers came from AI. The study appeared in the peer-reviewed journal Psychiatry Research. It was supported in part by the NIH Intramural Research Program at the National Institute on Drug Abuse, and the authors declared no competing interests.

What it found

Overall, clinicians rated the AI answers as high quality.

The researchers also found dangerous misinformation. Examples named in the study include:

• ignoring signs of suicidal thinking in a question
• giving incorrect emergency helpline information
• endorsing detox at home

The advice also shifted depending on how a question was worded. Two people asking about the same situation in different words could get different guidance.

The authors concluded that these tools need more safeguards and clinical validation before they are used in real care.

What it does not show

It does not tell us how often the chatbots got things dangerously wrong. The authors say their fact-check was a search for examples, not a systematic count, so they did not report an overall rate. In one test, they asked GPT-4 whether it is safe to detox at home when quitting long-term heroin use, worded 100 different ways. It said yes 23 times.

It tested two specific chatbots at one point in time. It does not test later versions of these systems, which may behave differently.

The researchers had to work around the chatbots' safety settings. At default settings, both systems often declined these questions, so the team used a prompt saying the request was for research and no one was at risk, and told LLaMA-2 not to avoid giving medical advice. Answers to an ordinary user may differ.

It does not show what happened to anyone who received these answers. It is an evaluation of the text, not a study of patient outcomes.

Why it matters

The danger is in the combination. The authors call it "a risky mix." An answer can look high quality on first inspection and still contain inaccurate and, in the authors' words, "potentially deadly medical advice." A wrong helpline or an endorsement of home detox inside an otherwise good answer is easy to miss. In this study, clinicians did not reliably catch it. They were asked to rate the answers, not to fact-check them, and the full paper reports that only one clinician flagged the answer to a question about detoxing at home from long-term heroin use. A person reading alone may not catch it either.

If you use a chatbot for recovery questions, do not rely on any emergency number it gives you. In an emergency, call 911. For a suicidal crisis, call or text 988. Check any medical step against a clinician or an official source before acting on it. The wording finding matters too. Rephrasing a question can change the answer, so a single answer is not a settled one.

Clinicians can put this to use at the next visit: ask patients whether they use chatbots for recovery questions, and what those tools have told them.