HR410 Unit 9 Evaluation Design Example

A Baseline Locked Before the First Session: A Unit 9 Evaluation Design for Other-Coverage Training

[Student Name]

Purdue University Global

HR410: Talent Development and Learning

Unit 9 Evaluation Design

[Instructor Name]

[Date]

Ridgemont Health Plan, its sites and every figure are composite and illustrative; the sources cited are real.

What the Evaluation Must Answer

Did the program for recent hires reduce other-coverage errors, and if so, by how much and at what cost? The answer has to separate the effect of training from the effects of the screen fix and the handle-time exception, which change at the same time.

The Design

Baseline. The quality analyst records each recent hire's audited error rate for the three months before the program. The figures are filed with the training manager and cannot be revised.

Staggered rollout. The north site's recent hires are trained first. The south site's are trained thirty working days later. For those thirty days, the south site serves as a comparison group.

Common changes at both sites. The screen default is removed and the handle-time exception begins at both sites on the same day, before the north site trains. Any change in errors at the south site during the thirty days reflects those common changes. Any additional change at the north site reflects training.

Shadish et al. (2002) describe this kind of design, in which a comparison group receives the treatment later, as a way to strengthen causal inference when random assignment is not possible.

The Data Collection Table

The four levels from Kirkpatrick and Kirkpatrick (2006) run down the side. For each: the measure, source, timing, owner and comparison.

Reaction. A five-item questionnaire on how useful the session was for real calls, not on enjoyment. Source: learners. Timing: end of session 3. Owner: trainer. Comparison: none.

Learning. The six-case skill check from Unit 3, given before and after, using two parallel case sets so remembered answers cannot inflate the second score. Source: learners. Timing: before session 1 and after session 3. Owner: trainer. Comparison: before against after.

Behavior. Audited other-coverage updates at thirty and ninety days. The quality analyst samples them without knowing which site trained first. Source: the membership system. Owner: quality analyst. Comparison: north against south during the thirty-day window, and each site against its baseline.

Results. Errors per hundred updates, reprocessing cost at $85 per error and complaints to the state insurance department. Source: quality and compliance records. Timing: quarterly. Owner: training manager. Comparison: against baseline and the 1.0 standard.

Of the four levels, behavior counts for most here, since it alone shows whether skill reached live calls, and the analyst's blindness to site keeps it honest.

The Threat the Design Cannot Remove

The two sites differ in call volume by season. The south site handles more calls from members in a state whose open enrollment period changes the case mix in some months. If those months fall in the thirty-day window, the comparison could be distorted. The design records case mix at both sites and reports results with and without the months most affected.

Measuring Learning Outcomes Properly

Kraiger et al. (1993) argue that learning outcomes should be measured with methods suited to their kind. A procedural skill should be measured by performance, not by a multiple-choice test about the procedure. The skill check meets that standard.

What the Evaluation Costs

Quality analyst time for the full thirty-day audit and blind sampling, about 60 hours. Trainer time for the skill checks, about 8 hours. Compliance data extraction, about 4 hours. At loaded rates, about $3,100, roughly a third of the $9,800 training estimate in Unit 3.

How Results Will Be Reported

The training manager will report to the member services director at thirty and ninety days. Each report gives the four levels in order, with the comparison for each, and states plainly which result supports the program and which does not. Results for individual representatives are shared only with their team leads and are not used in performance ratings during the evaluation period, so representatives have no reason to hide errors from the audit.

What Would Change the Design

If the south site's training has to move earlier for operational reasons, the comparison window shrinks, and the report will rely more on the baseline comparison at each site. If fewer than ten split-period cases arise at either site during the window, the analyst will add scripted cases worked on the live system under audit, so the behavior measure has enough cases to compare.

What Counts as Success, Stated in Advance

The program will be judged successful if, at ninety days, recent hires' audited error rate is at or below 2.0 per hundred, the north site's rate falls by at least half against the south site's during the thirty-day window, and skill check scores rise from about 57 percent to at least 90 percent. If the north and south sites improve equally during the window, the conclusion will be that the screen fix and scoring change, not training, drove the improvement.

References

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler.

Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311-328. https://doi.org/10.1037/0021-9010.78.2.311

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.