An overdose risk score held up in a second state, but most people it flagged did not overdose

The short version

Researchers built a machine-learning model that reads Medicaid insurance claims and estimates a person's chance of an opioid overdose in the next 3 months. It sorted people by risk well, in Pennsylvania and again in Arizona. But overdose is rare, so even among the people the model flagged as highest risk, most did not have one.

What they did

The team used claims data from Pennsylvania Medicaid for 2013 to 2016 to build the model. The data covered 639,693 adults aged 18 to 64 who had filled at least one opioid prescription. The model looked at 284 possible predictors, including diagnoses, prescriptions, emergency visits, and prescriber and regional factors, and updated them every 3 months.

Then they tested it on people it had never seen: 318,585 Pennsylvania Medicaid members from 2017 and 2018, and 391,959 Arizona Medicaid members from 2015 to 2017. The outcome was a hospital or emergency department visit for opioid overdose, including heroin, in the following 3 months.

What it found

The model was good at ranking. Its C-statistic, a measure of how often it scores a person who overdosed above one who did not, was 0.841 inside the original data, 0.828 in later Pennsylvania data, and 0.817 in Arizona.

In the later Pennsylvania data, 22.4% of people landed in the model's high-risk groups. Those groups held 73% of the overdoses in the next 3 months. In Arizona, 10% landed in the high-risk groups, and those groups held 55% of the overdoses. In the lower-risk groups, 0.2% or fewer had an overdose.

The model also did better than the rules state Medicaid programs already use, such as high-dose prescriptions, filling opioids from four or more prescribers and four or more pharmacies, or taking opioids with benzodiazepines. In the original Pennsylvania data over 12 months, those rules flagged 9.3% of people and caught 24.2% of overdoses. Using a top-5% risk-score cutoff, the model flagged a similar share, 8.2% of people over the year, and caught 66.1%.

What a "high risk" flag means

This is the part to read slowly. A model can rank a population well and still be wrong about most of the individuals it flags.

The measure that answers "if I am flagged, how likely am I to overdose?" is called positive predictive value. In the later Pennsylvania data, it ranged from 0.38% to 4.08% across the high-risk groups. Put plainly: in the very highest-risk group, which held fewer than 1 in 400 people, a little over 4 in 100 had an overdose in the next 3 months. In the lower high-risk tiers, it was fewer than 1 in 100. In Arizona, the range was 0.19% to 1.97%.

The paper gives a second way to see it, the number needed to evaluate. At the cutoff that balances sensitivity and specificity, a program would need to evaluate 171 flagged people in the later Pennsylvania data to find one who went on to overdose. In Arizona, the number was 281. The authors say the low positive predictive values could mean more false positives. They also argue that numbers like these are similar to the number of women screened with mammography to prevent one breast cancer death. That comparison is theirs, and it sets overdoses predicted against deaths prevented; this study did not test prevention.

That does not make the score useless. The authors suggest matching the step to the risk: low-cost, low-harm steps such as naloxone distribution for the wider moderate-to-high-risk groups, and burdensome ones such as pharmacy lock-in programs only for the small highest-risk group. A flag should not be read as "this person is about to overdose." The authors say further screening and assessment are needed so that false positives do not cause harm.

What it does not show

The study predicted overdoses. It did not test whether acting on the score prevents them, and no one's care changed because of it.

It saw only people with an opioid prescription paid by Medicaid. People using only illicit opioids were not in it. In Arizona, where death records were linked, only 40% of fatal overdoses were captured by the study's cohort definition. Overdoses that never reached a hospital or emergency department are not in claims data.

The model uses race and ethnicity as inputs. The authors say it needs a full bias evaluation before use. They also name practical barriers, such as delays in claims data.

On funding: the work was paid for by the National Institute on Drug Abuse and the National Institute on Aging, and the paper states the funder had no role in the study. The first author and the senior author are named inventors on a preliminary patent filing from the University of Florida and the University of Pittsburgh for using the paper's machine-learning algorithm to predict opioid risk in Medicare. Several authors also reported grants or fees from drug companies.

Why it matters

Health plans and programs may use risk scores like this one to decide who gets a phone call, a naloxone kit, or a closer look. The useful question for anyone flagged is not "how accurate is the model?" It is "of the people flagged like me, how many actually had an overdose?" Here, even at the top, it was about 4 in 100.

For clinicians and programs, the paper is a reminder to match the response to the score. A cheap, low-harm step can go to many flagged people. A burdensome one should go to few, and only after a person has been assessed.

If someone may be overdosing, call 911 and give naloxone if you have it.