HI580 Unit 10 Post-Implementation Evaluation Example

Baselines First, Then the Results: A Unit 10 Post-Implementation Evaluation at 30, 90 and 180 Days on a Labor Unit

[Student Name]

Purdue University Global

HI580: Information Systems Design and Implementation

Unit 10 Post-Implementation Evaluation

[Instructor Name]

[Date]

Brackenridge Medical Center, its unit, staff and every measured value are composite; the sources cited are real.

Measures Fixed Before Go-Live

Six measures were chosen during planning, each tied to a requirement or a workaround from the current-state map:

1. Minutes of re-entry per twelve-hour shift, tied to the bedside worksheet.

2. Delay between a tracing segment and its charted interpretation, tied to REQ-11.

3. Share of interpretations entered from the patient's room, tied to REQ-08.

4. Time for risk management to retrieve an archived tracing, tied to REQ-28.

5. Newborn record creation delay after birth, tied to REQ-19.

6. A ten-item staff usability survey, the same instrument at every round.

A seventh, bedside sign-in time, was added during planning after the interface review raised it as a risk.

Methods

The same procedures were used at baseline and at every follow-up. Re-entry minutes came from direct observation of four nurses across day and night shifts, using the same observer and timing sheet. Entry delay and bedside share came from audit-log extracts comparing each interpretation's signed time with its tracing segment time and the workstation location. Retrieval time was measured by asking the risk manager to retrieve five randomly chosen tracings from the prior year. Newborn delay came from comparing the newborn record's creation time with the shared clock's birth time. The survey was sent to all fifty-four labor nurses at each round.

Results

Each measure is given at baseline and at 30, 90 and 180 days.

1. Re-entry minutes per shift: 40 at baseline; 12 at 30 days; 4 at 90 days; 3 at 180 days.

2. Median delay from tracing to charted interpretation: 52 minutes; 18; 9; 7.

3. Share of interpretations entered from the room: 0 percent; 61 percent; 84 percent; 91 percent.

4. Time to retrieve an archived tracing: 3 days; 40 minutes; 25 minutes; 20 minutes.

5. Median newborn record delay: 14 minutes; 6; 3; 2.

6. Usability survey mean, on a 1 to 5 scale: 2.4; 2.9; 3.6; 3.8.

7. Median bedside sign-in time: 9 seconds at the old central station; 31 seconds; 28 seconds; 27 seconds.

Six of seven measures improved, and the one that worsened is the one nurses mention first when asked about the new system.

Use Versus Uptime

The system was available 99.7 percent of the time over 180 days, with one planned upgrade and one twenty-minute interface outage, during which queued interpretations posted correctly. Uptime answers whether the system runs. It does not answer whether nurses use it as designed. The share of interpretations entered from the room, rising from 61 percent at 30 days to 91 percent at 180, is the better measure of adoption, because it shows nurses abandoning the old habit of charting later. Implementation research distinguishes adoption, the extent to which staff take up a system in practice, from its technical availability, and treats the first as the harder and more telling outcome (Cresswell & Sheikh, 2013).

The Measure That Worsened

Bedside sign-in time rose from about nine seconds to nearly thirty, because room workstations require a full sign-in with a password, while the old central station stayed signed in for the charge nurse. Nurses sign in about twenty times a shift, which adds roughly six minutes, offsetting part of the re-entry saving. The evaluation does not bury this. It recommends badge tap sign-in for room workstations, already approved by information security, and projects that sign-in time would fall to about five seconds.

Interpreting the 30-Day Results

The 30-day figures show a unit still learning. Re-entry had not disappeared, because some nurses kept writing on paper out of habit, and the usability score was barely above baseline. By 90 days most measures had reached a plateau. Evaluating only at 30 days would have underestimated the system's benefit; evaluating only at 180 days would have hidden the difficult first month that staff remember. Reporting all three time points lets leaders see both.

Sociotechnical Lessons

The results reflect changes in several parts of the system together, not the software alone. Room workstations, a facilities change, made bedside charting possible. The shared clock, an infrastructure change owned by biomedical engineering, made entry delay measurable at all. Newborn registration moved to a different role. A sociotechnical view predicts that outcomes depend on how these parts fit (Sittig & Singh, 2010), and the evaluation suggests the fit improved most between 30 and 90 days as workflow, people and technology settled together.

What Staff Said

The survey's open comments matched the numbers. At 30 days, the most frequent comments concerned sign-in time and a category list that required scrolling on the smaller room screens; the list was reordered at week five. By 90 days, comments shifted toward handoff, with several nurses writing that marked segments made the night report shorter. At 180 days, the most common comment was a request to extend the room workstations to the triage bays, which still use the central station. One nurse wrote that she no longer carries a pen for the worksheet, which is perhaps the plainest summary of the change.

Costs and Benefits in Brief

The project's largest measurable benefit is nurse time: about 37 fewer minutes of re-entry per shift, partly offset by about six more minutes of sign-in. Across the unit's staffing, that returns roughly 70 nursing hours a month to patient care. The retrieval improvement has a different kind of value. Risk management's last request under the old system took three days and an outside vendor fee; the new archive produces a tracing in minutes.

Recommendations

1. Install badge tap sign-in for room workstations within the next quarter and remeasure sign-in time.

2. Continue the quarterly audit of bedside entry share, with a target of 95 percent.

3. Repeat the usability survey at one year.

4. Share the retrieval improvement with risk management and legal, since retrieval fell from days to minutes.

Limits

Observation covered four nurses per round, so re-entry minutes are estimates. Survey response rates fell from 78 percent at 30 days to 55 percent at 180, and later respondents may be the most satisfied. The evaluation cannot attribute improvements to the system alone, since staffing and census also changed over six months. Health IT evaluations commonly share these limits, and stating them openly is part of reporting results credibly (Kaplan & Harris-Salamone, 2009).

References

Cresswell, K., & Sheikh, A. (2013). Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review. International Journal of Medical Informatics, 82(5), e73-e86. https://doi.org/10.1016/j.ijmedinf.2012.10.007

Kaplan, B., & Harris-Salamone, K. D. (2009). Health IT success and failure: Recommendations from literature and an AMIA workshop. Journal of the American Medical Informatics Association, 16(3), 291-299. https://doi.org/10.1197/jamia.M2997

Sittig, D. F., & Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality and Safety in Health Care, 19(Suppl 3), i68-i74. https://doi.org/10.1136/qshc.2010.042085