GB542 Unit 9 Learning Evaluation Report Example

A 2.7 Percent Return, Reported Plainly: A Unit 9 Learning Evaluation of a Continuous-Process Technician Program

[Student Name]

Purdue University Global

GB542: Strategic Talent Development and Organizational Learning

Unit 9 Learning Evaluation Report

[Instructor Name]

[Date]

The company, the program and every measurement and dollar figure are composite and illustrative; the evaluation frameworks and the sources cited are real.

Summary

The first cohort of 11 batch operators completed the 36-week program one week before the continuous line started and staffed two of its four crews. The other two crews were staffed by operators waiting for the second cohort, working with the equipment vendor's commissioning technicians. That arrangement created a natural comparison group. On every level the program worked. In money, it paid for itself in just under twelve months, with a benefit-cost ratio of 1.03. The report recommends running the second cohort as designed, with two changes described at the end.

Method

The evaluation follows Kirkpatrick's four levels of reaction, learning, behavior and results (Kirkpatrick & Kirkpatrick, 2016), with the fifth level, return on investment, calculated according to the Phillips method (Phillips & Phillips, 2016). That method requires three steps beyond measuring results: isolating the program's effect from other influences, converting the isolated effect to money and counting fully loaded costs. It also sets conservative rules, including counting only first-year benefits for a program of this length and reporting intangible benefits separately rather than converting them.

Isolating the effect. Trained and waiting crews were not identical before the program; the trained crews' members had slightly better records on the batch line. A simple comparison of the two groups after training would credit the program with that pre-existing gap. The evaluation therefore uses differences in differences, a standard approach to this problem (Angrist & Pischke, 2009). It compares each group's change from the commissioning period, when vendor technicians led every shift on the new line, to the first eight months of normal operation, and subtracts the waiting crews' change from the trained crews' change.

Results by Level

Level 1: reaction. The eleven participants rated the program 4.4 out of 5 on average. The simulator scored highest for realism, at 4.7, and the classroom phase lowest, at 3.9, with comments that some instrumentation content was too general.

Level 2: learning. On the ten-scenario simulator test, the average score rose from 41 percent before the program to 86 percent after. Every participant passed the 75 percent standard.

Level 3: behavior. Auditors reviewed every alarm event in the first three months of operation. Trained crews followed the response standard, acknowledging within ninety seconds and taking the correct first action, in 88 percent of events. Waiting crews did so in 52 percent.

Level 4: results. Off-specification output, as a share of production, and unplanned downtime, in hours per crew per month, were measured in both periods.

*Off-specification output.* During commissioning, trained crews ran at 4.9 percent and waiting crews at 5.3 percent. In operation, trained crews ran at 2.6 percent and waiting crews at 4.1 percent. The raw gap after training is 1.5 points. But trained crews improved by 2.3 points and waiting crews by 1.2 points, so the program's isolated effect is 1.1 points.

*Unplanned downtime.* During commissioning, trained crews logged 9.8 hours a month and waiting crews 10.4. In operation, trained crews logged 5.2 and waiting crews 8.3, a raw gap of 3.1 hours. Trained crews improved by 4.6 hours and waiting crews by 2.1, so the isolated effect is 2.5 hours a month per crew.

About a quarter of the apparent improvement in off-specification output belonged to the crews, not the program, and the report would have overstated its own case without subtracting it.

How the Levels Connect

The four levels tell a consistent story, and that agreement makes it more believable that training, and not some other change, produced the gains. Participants learned the responses, as the test scores show, and used them on the line, as the alarm audits show. The behavior gap between trained and waiting crews, 88 against 52 percent, is large enough to account for most of the difference in off-specification output, since a delayed or wrong response to a viscosity alarm is the most common way a continuous line sends poor material downstream. If the test scores had risen but alarm behavior had not changed, the results would be much harder to attribute to training.

Converting Results to Money

Off-specification output. Each crew produces about 3.6 million pounds a year. A 1.1-point reduction saves 39,600 pounds per crew, or 79,200 pounds for the two trained crews. Off-specification adhesive is downgraded or scrapped at an average loss of $3.10 a pound, including disposal, so the benefit is $245,520.

Downtime. The line produces about 1,800 pounds an hour at a contribution of about $1.75 a pound, so an hour of unplanned downtime costs about $3,155.20 in lost contribution. Two crews avoiding 2.5 hours a month for twelve months avoid 60 hours, a benefit of $189,312.

Total first-year benefit: $434,832. The eight months of observed data are extended to twelve at the same monthly rates, which assumes the waiting crews do not close the gap before the second cohort finishes; since the second cohort will finish in month nine, the report notes this as a limit.

Costs

The fully loaded cost of the first cohort comes from the Unit 5 budget. Wages paid to participants while in the program are the largest item at $215,424. Coaches add $54,912, the vendor's simulator and its support $68,000, and instructors $16,675. Half of the design work, $42,000, is charged to this cohort, and testing, evaluation and materials add $26,369. Together these reach $423,380.

Return on Investment

Net benefit: $434,832 less $423,380 is $11,452.

ROI: net benefit divided by costs, times 100, is 2.7 percent.

Benefit-cost ratio: benefits divided by costs is 1.03.

Payback: monthly benefits average $36,236, so costs are recovered in about 11.7 months.

Intangible Benefits

Several benefits were observed but not converted to money, following the method's rules: fewer near-miss reports on trained crews, higher confidence reported by supervisors in trained technicians and the fact that all eleven participants remain with the company. Each would raise the return if it could be credibly valued.

Limits

The groups are small, two crews in each, so a single unusual month could move the averages. The commissioning period was short, about six weeks, which makes the baseline less stable than a longer one would be. The dollar values for off-specification output and downtime come from finance's standard costs and may change as the line's volume grows. Each limit could shift the size of the effect, yet none plausibly flips its sign; the 2.7 percent figure is best treated as a central estimate that could reasonably land a few points higher or lower.

Recommendations

Run the second cohort as designed. Revise the classroom phase to focus instrumentation content on the line's actual instruments, since participants rated it lowest. Extend coaching on the waiting crews' shifts during the second cohort's training, since the behavior gap shows untrained crews are where alarms are most often mishandled. Report this evaluation to leadership as it stands: a small positive return, measured conservatively, is more useful for future decisions than a larger number that would not survive questioning.

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press.

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Phillips, J. J., & Phillips, P. P. (2016). Handbook of training evaluation and measurement methods (4th ed.). Routledge.