PS497 Unit 8 Program Evaluation Plan Example

How the Program Will Know if It Worked: A PS497 Unit 8 Evaluation Plan

[Student Name]

Purdue University Global

PS497: Bachelor's Capstone in Addiction

Unit 8 Evaluation Plan

[Instructor Name]

[Date]

The writer, the program, the clients and every figure are composite and illustrative; the articles cited are real.

Questions

The evaluation asks four questions. Do clients use the program? Does it reduce substance use and return to intensive treatment in the six months after intensive outpatient treatment, compared with recent graduates who did not have it? Do clients and staff find it acceptable? And what does it cost in staff time?

Measures

Engagement: the share of weekly check-ins completed, counselor response times, text exchanges per client and monthly group attendance. McKay (2009) identified dropout as a central weakness of continuing care, so engagement is measured first and weekly.

Substance use: days of use reported in the weekly check-in, following the approach of Gustafson et al. (2014), who used reported risky drinking days as a main outcome, along with results of monthly breath or urine tests collected at the clinic.

Return to intensive treatment: readmission to intensive outpatient or higher levels of care within six months, drawn from program records.

Retention in any care: whether the client remains connected to any program service at three and six months. Campbell et al. (2014) used treatment retention as a key outcome in their trial of web-delivered treatment, and it matters here because staying connected is the program's main purpose.

Acceptability: a short survey for clients at three and six months and a staff survey at the end of the pilot.

Staff time: counselors log minutes spent on check-ins and texts each week.

Comparison Group

A randomized trial is not possible in a small community program. The plan compares the pilot group with the sixty clients who completed intensive outpatient treatment in the previous year and got the usual follow-up: one group a month and reminder calls. Program records provide their readmission and retention rates, though not weekly use data.

Without last year's graduates as a yardstick, a good pilot result would be impossible to read.

Data Sources and Privacy

Check-in data come from the secure platform; tests, readmissions and retention come from the treatment record. All analyses use de-identified data, and no individual results leave the program. Enrollment includes consent for de-identified data to be used in this evaluation.

Analysis

The evaluation will report engagement trends by week, the share of clients readmitted and retained at six months in each group, and average days of use for the pilot group by month. With twenty clients, differences will be described, not tested for statistical significance, and results will be presented with their uncertainty.

Decision Rules Written in Advance

If fewer than half of enrolled clients complete at least two-thirds of their check-ins by month three, the team will review the design before enrolling more. If the pilot group's six-month readmission rate is no better than the comparison group's, the program will not expand without changes. If engagement is high and readmission is lower, the clinical director will seek funding to offer the program to all graduates. If staff time exceeds the hours budgeted by more than a quarter, the workflow will be revised.

Listening to Clients and Staff

Numbers will not explain why the program works or fails, so the plan adds two kinds of listening. At three months, I will interview five clients, chosen to include both frequent and infrequent users, about what helped, what felt intrusive and what they would change. When the pilot closes, both counselors will meet with the clinical director and me to discuss workload, difficult exchanges and the triage rules. These conversations will be summarized alongside the numbers, so that a low completion rate, for example, can be read in light of what clients say about why they stopped checking in.

What Counts as Success

Before enrollment, the team agreed on what success would look like: at least two-thirds of check-ins completed across the pilot, a six-month readmission rate noticeably lower than last year's graduates, most clients rating the program as helpful, and staff time within budget. Writing these targets down now keeps the team from deciding after the fact that whatever happened was good enough.

Fidelity

The plan also checks whether the program runs as designed: whether counselors reply within one business day, send planned outreach messages and follow the triage rules. A supervisor reviews a sample monthly.

Threats to Interpretation

Several factors could distort how the pilot appears. The clients who say yes may simply be the more motivated ones. Last year's graduates may differ from this year's. Self-reported use may be underreported. The evaluation will report how many eligible clients declined and why, and compare the two groups on age, substance and severity where records allow.

Timeline and Reporting

Enrollment runs for three months, and follow-up for six months after each client's enrollment. The team reviews engagement monthly and the full results at the end of month nine. A short report will go to the clinical director, the counselors and a client advisory group.

References

Campbell, A. N. C., Nunes, E. V., Matthews, A. G., Stitzer, M., Miele, G. M., Polsky, D., Turrigiano, E., Walters, S., McClure, E. A., Kyle, T. L., Wahle, A., Van Veldhuisen, P., Goldman, B., Babcock, D., Stabile, P. Q., Winhusen, T., & Ghitza, U. E. (2014). Internet-delivered treatment for substance abuse: A multisite randomized controlled trial. American Journal of Psychiatry, 171(6), 683-690. https://doi.org/10.1176/appi.ajp.2014.13081055

Gustafson, D. H., McTavish, F. M., Chih, M.-Y., Atwood, A. K., Johnson, R. A., Boyle, M. G., Levy, M. S., Driscoll, H., Chisholm, S. M., Dillenburg, L., Isham, A., & Shah, D. (2014). A smartphone application to support recovery from alcoholism: A randomized clinical trial. JAMA Psychiatry, 71(5), 566-572. https://doi.org/10.1001/jamapsychiatry.2013.4642

McKay, J. R. (2009). Continuing care research: What we have learned and where we are going. Journal of Substance Abuse Treatment, 36(2), 131-145. https://doi.org/10.1016/j.jsat.2008.10.004