HI300 Unit 3 Systems Evaluation Example

Weighted Before the Demos, Scored After: A Unit 3 Systems Evaluation Rating Three Perinatal Documentation Products Against Twelve Requirements Written in Advance

[Student Name]

Purdue University Global

HI300: Information Systems for Health Care

Unit 3 Systems Evaluation

[Instructor Name]

[Date]

The hospital, the three products and their scores are composite; the method and the sources cited are real.

Method

The evaluation used the twelve requirements the unit derived from observing two shifts. The project team, made up of two labor nurses, an obstetrician, a midwife, the unit clerk, an HIM analyst and an information services analyst, assigned weights before seeing any product. Must requirements were weighted 10 or 12; should requirements were weighted 4 to 6. The weights sum to 100. Each vendor received the same written script and performed it on the unit's own workstations, using composite patients. Each team member scored each requirement as 0 for not met, 0.5 for met with a workaround, or 1 for met as written, and the team's average for each requirement was multiplied by its weight. Scripted testing matters because unscripted demonstrations show what a product does well, not whether it meets what a unit needs (Cresswell et al., 2013). The script also forced each vendor to perform the dull tasks, such as the badge sign-on and the census view, that a polished demonstration skips.

Weights

The two heaviest requirements, at 12 each, were the shared system clock and the bedside view during network loss. Interpretation entry, provider notification, the linked newborn record and the admission and transfer feed were weighted 10 each, and the perinatal quality report 8. The five should requirements shared the remaining 18 points: the charge nurse summary view 5, oxytocin rate changes beside the tracing 4, badge sign-on 4, the response time report 3 and the single legal record 2.

Results

Product A scored 68. It met every must requirement except the bedside view during network loss, which it offered only through a separate downtime workstation at the nurses' station, a workaround scored 0.5 for a loss of 6 points. It lost further points on the summary view and on reporting, where its perinatal measures needed export to a spreadsheet. Product B scored 81. It led on charting speed, where nurses entered an interpretation in two actions, and on notification, which reached the charge nurse's phone in under thirty seconds during the test. Its weakness was the clock: the product took time from each fetal monitor rather than from the hospital's time server, and during the script the monitor in room 4 ran ninety seconds ahead of the record. The team scored the clock requirement 0, a loss of 12 points. Product C scored 74. It met both of the heaviest requirements fully but needed seven actions to enter an interpretation and could not create the newborn record until a registration clerk had registered the baby, which happened in the test about forty minutes after birth. The highest score belongs to a product that fails the clock requirement outright. A weighted total can hide a must, which is why musts are checked separately from the sum. Scoring differences among team members were small on most requirements and large on two. The nurses rated Product C's seven-action interpretation entry as a near failure, while the information services analyst rated it as met, since every step worked. The team kept the average, but the gap itself is a finding. The people who will chart every few minutes judged speed at the bedside very differently from the person who will support the system, and a selection made without the nurses' scores would have ranked Product C higher than it deserves.

Reading the Scores

A weighted total ranks the products, but it does not decide among them on its own. The team applied two rules agreed in advance. First, a product that fails any must requirement cannot be selected unless the vendor shows a fix in a second test. Second, scores within five points of each other are treated as a tie, because a single team member's half point can move a total by two or three points. Under those rules, no product is selectable today. Product B fails a must, Product A meets its must only with a workaround, and Product C meets every must but is slow at the bedside.

Recommendation

The team recommends a second scripted test limited to the failed or partly met must requirements: Product B's clock source and Product A's bedside downtime view. If Product B can take its time from the hospital server, its lead on the requirements that nurses use most would likely make it the choice. Product C remains the safe fallback. The team also recommends asking each vendor for two reference sites of similar size and visiting one, because the literature on large implementations finds that what a product does at a peer hospital predicts adoption better than what it does in a demonstration room (Kaplan & Harris-Salamone, 2009). Finally, the team notes that scores came from seven people who all work on the same unit, and a second evaluator from another hospital department could check them for shared blind spots, a point the sociotechnical view of health IT makes about who gets to define success (Sittig & Singh, 2010).

References

Cresswell, K. M., Bates, D. W., & Sheikh, A. (2013). Ten key considerations for the successful implementation and adoption of large-scale health information technology. Journal of the American Medical Informatics Association, 20(e1), e9-e13. https://doi.org/10.1136/amiajnl-2013-001684

Kaplan, B., & Harris-Salamone, K. D. (2009). Health IT success and failure: Recommendations from literature and an AMIA workshop. Journal of the American Medical Informatics Association, 16(3), 291-299. https://doi.org/10.1197/jamia.M2997

Sittig, D. F., & Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality and Safety in Health Care, 19(Suppl 3), i68-i74. https://doi.org/10.1136/qshc.2010.042085