NU710 Unit 3 Readmission Risk Model Evaluation Example

A Score That Ranks by What the Hospital Has Seen: A Unit 3 Evaluation of a Readmission Risk Model Used to Assign Follow-Up Calls

[Student Name]

Purdue University Global

NU710: Advanced Health Information and Technologies

Unit 3 Predictive Analytics Evaluation

[Instructor Name]

[Date]

The hospital, the model's local performance figures and the patients are composite and illustrative; the studies cited are real.

The Model and the Decision It Drives

The score is a commercial model built into the electronic record. It assigns each inpatient a probability of readmission within 30 days, recalculated daily, from data such as age, diagnoses, how often the patient was admitted or seen in the emergency department during the past twelve months, medications, length of stay and selected laboratory values. The vendor publishes its general performance but not its exact formula.

The hospital adopted it to replace the bedside priority rule recommended in the ethics course, which gave a point each for advanced age, living alone, a literacy concern, interpreter use and an unfinished practice injection. The vendor score was easier: it appeared automatically, required no extra charting and could be sorted on a list. Patients starting insulin with scores in the top third receive a call within 72 hours; the rest receive a mailed card with the clinic's number.

How Well It Predicts Here

A model's published performance is not its local performance. Kansagara et al. (2011), reviewing 26 readmission models, found that the ones built on claims-type data and tested on large American samples separated returners from non-returners poorly, none reaching a c statistic of 0.66, and that models usable at discharge did better, from 0.68 to 0.83. Two of the studies they reviewed found that functional and social variables improved prediction, and few models included them.

The hospital's quality analyst validated the vendor score on 590 patients discharged on new insulin over the prior year, 84 of whom returned within 30 days. The local c statistic was 0.66, modest discrimination. Calibration was uneven: the score overestimated risk for patients with many prior admissions and underestimated it for patients with few. Among the 84 who returned, 37 had scores outside the top third and so would not have been called.

A third of the patients who came back were the ones the score told the educator not to call.

Who the Score Misses and Why

The analyst compared the 37 missed patients with the 47 the score caught. The missed group was younger, more often uninsured or recently insured, more often Spanish- or Marshallese-speaking, and had far less recorded contact with the hospital in the year before. Many were newly diagnosed. Their low scores were not evidence of low risk; they reflected how little contact they had previously had with the hospital.

This is the mechanism Obermeyer et al. (2019) described in a widely used population health algorithm. Their algorithm forecast dollars rather than disease. Since less had historically been paid out for the care of equally ill Black patients, those patients had to carry a heavier burden of chronic illness before their scores matched those of White patients. Fixing the proxy would have nearly tripled the proportion of Black patients flagged for the care program. The vendor score here does not predict cost, but it leans heavily on prior utilization, and prior utilization reflects access as much as illness. Patients who have had the least care look the safest. In this hospital that pattern runs along language and insurance lines more than race, but the mechanism is the same: a convenient proxy stands in for need and quietly redistributes help away from those with less access.

Ethical Risks

Cao et al. (2025), in a rapid review of 22 studies on predictive analytics and early warning systems in nursing, identified five dimensions of ethical risk, among them risks arising from data and algorithms, from the attribution of professional roles and responsibility and from patient rights and humane care. All three appear in this case. The data risk is the utilization bias above. The responsibility risk is that nurses and the educator now follow the list rather than their judgment, while no committee or person has been asked to answer for the list's fairness. The patient rights risk is that a patient who would benefit from a call does not receive one because of a number nobody explained to them or to the nurse.

There is also a transparency problem. The vendor does not disclose the formula, so the hospital can measure the score's performance but cannot see exactly why a given patient scores low.

What Nurses Do With the Score

Interviews with the educator and four nurses showed that the score has changed practice in ways nobody intended. Before the score, nurses flagged patients they were worried about. Now, one nurse said, "If the score is low, I don't bother flagging, because it won't change who gets called." The score has displaced the knowledge that the earlier priority rule captured: who lives alone, who struggled with the pen and who needed an interpreter.

What the Local Validation Cannot Tell Us

The validation is itself limited. It covers one year at one hospital, the number of returners is modest and the groups compared are small enough that a few patients change the picture. It measures returns only to this hospital, so patients who went elsewhere count as successes. These limits argue for continued monitoring rather than for trusting the score more.

Recommendations

1. Combine, do not replace. Use the vendor score together with the bedside factors from the earlier rule. A patient qualifies for a call if either the score is in the top third or two or more bedside factors are present. A quick re-analysis of the 590 patients showed this rule would have caught 69 of the 84 returners, against 47 with the score alone, at the cost of about 60 more calls a year.

2. Allow and record nurse override. Any nurse may flag a patient for a call regardless of score, with a one-line reason recorded, and overrides will be reviewed quarterly.

3. Monitor performance by group. The analyst will report sensitivity, the share of returners who were called, every quarter by language, insurance and new diagnosis.

4. Assign governance. The nursing informatics council will own the score's use for nursing decisions and will review the local validation yearly.

5. Ask the vendor. Request documentation of the variables and their weights and whether a version without utilization variables is available.

Conclusion

The readmission score is convenient but modest in accuracy on this hospital's patients, and it misses the newly diagnosed, the recently insured and patients who speak languages other than English, because it mistakes past contact with the hospital for risk. Used alone, it quietly withdraws follow-up from patients who need it most. Combined with bedside knowledge, open to nurse override and monitored by group, it can help the educator's calls reach the right patients.

References

Cao, Y., Deng, L., Liu, X., Feng, Z., & Gao, Y. (2025). Ethical challenges in the algorithmic era: A systematic rapid review of risk insights and governance pathways for nursing predictive analytics and early warning systems. BMC Medical Ethics, 26(1), 151. https://doi.org/10.1186/s12910-025-01308-z

Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., & Kripalani, S. (2011). Risk prediction models for hospital readmission: A systematic review. JAMA, 306(15), 1688-1698. https://doi.org/10.1001/jama.2011.1515

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342