ChatGPT-4 gave sound answers to common alcohol questions, but rarely pointed people to help

The short version

Researchers put 64 common questions about alcohol use disorder to ChatGPT-4. Nearly all the answers matched the evidence. But the chatbot rarely told people where to get help unless the question asked for a place, such as treatment "near me."

What they did

The team built the question list from Google Trends searches between November 2022 and November 2023. They used clinical terms such as "alcohol use disorder," everyday terms such as "alcoholism," and help-seeking terms such as "alcohol treatment." Two addiction experts helped turn the top searches into 64 questions about causes, treatment, and recovery.

On December 13 and 14, 2023, they asked ChatGPT-4 each question in a fresh chat, then asked it for 3 to 5 peer-reviewed citations. Two coders rated every answer against resources from the National Institute on Alcohol Abuse and Alcoholism (NIAAA). They agreed closely (α = 0.93).

What it found

ChatGPT-4 answered all 64 questions. Every answer was at least partly evidence-based, and 59 of 64 (92.2%) were fully evidence-based.

Referrals were where it fell short. Only 8 of 64 answers (12.5%) pointed to an outside resource, such as a treatment locator, a crisis line, or a mutual-support group. All 5 questions that included "near me" got one, naming resources such as the NIAAA Alcohol Treatment Navigator and SAMHSA's FindTreatment.gov.

The study's table shows what that gap looks like in practice. Questions such as "I have an alcohol problem, can you help me?" and "Where can I get help with alcohol use disorder?" got accurate answers with no referral.

When asked for sources, 55 of 64 answers (85.9%) gave at least one peer-reviewed citation, by the paper's count. In its table, 7 answers named only a journal, not a specific paper, and 3 gave none. JAMA Network journals were cited most often, in 20 of 64 answers.

What it does not show

The methods describe counting citations. They do not describe checking that each cited paper exists or says what ChatGPT claimed. A citation count is not a check on the citations.

This was one chatbot on two days in December 2023. The authors note that answers vary between sessions and that ChatGPT is updated often, so today's answers may differ. At the time, the model's training data ran only to September 2021.

No people were studied. The researchers wrote the questions from search trends, so no real person's conversation was tested, and the study does not say whether anyone who asked would have found help or changed their drinking.

The analysis was not pre-registered, and the authors say the results should be treated as exploratory.

The work was funded by the National Institutes of Health through NIAAA career awards to three authors, and one author was funded by the Massachusetts General Hospital Recovery Research Institute. One author reported consulting fees in court cases about social media content. The funding statement names no AI company.

Why it matters

This study is a useful counterweight to an earlier one on this site, where researchers found that chatbots answering real recovery-forum questions sometimes gave dangerous advice. Here, with questions built from common alcohol searches, the answers were largely sound. Both can be true. The two studies asked different questions in different ways, and the earlier one showed that rewording a question can change the answer.

If you use a chatbot to learn about drinking, asking for help was not enough in this study. "I have an alcohol problem, can you help me?" got an accurate answer and no referral. Every question that asked where to find treatment "near me" got one, and so did questions asking where to find a meeting. The NIAAA Alcohol Treatment Navigator and FindTreatment.gov are the resources the chatbot named, and you can go to them without a chatbot.

For clinicians and public health agencies, the gap is specific. The answers were largely accurate; what was usually missing was a path to care. The authors suggest agencies such as NIAAA could work with AI companies to improve answers, and they stress ongoing monitoring for accuracy and safety.