PS451 Unit 10 Selection Program Evaluation Example

What the New Hiring System Is Worth: A PS451 Unit 10 Selection Program Evaluation

[Student Name]

Purdue University Global

PS451: Selection and Assessment in Organizations

Unit 10 Selection Program Evaluation

[Instructor Name]

[Date]

The writer, the tree care company, the hiring figures and every estimate are composite and illustrative; the sources cited are real.

What Changed

Over twelve months, the company replaced a foreman's handshake and an untested strength test with a system built from a job analysis. Climber applicants now complete a movement-based physical assessment, a six-question structured interview and a scored work sample on the training tree. Entry-level applicants take a mechanical reasoning test, a short chipper and drop zone work sample and a paid half-day ride-along, then are placed in one of three tracks. Ratings use five anchored dimensions after frame-of-reference training, and crews train together on rescue and drop zone communication.

The general manager asked a simple question: was it worth it? This evaluation answers in four parts, because the SIOP Principles treat validity, fairness and ongoing review as conditions for any selection system, not optional extras (Society for Industrial and Organizational Psychology, 2018).

Validity

The first year produced thirty-eight hires: eleven climbers and twenty-seven entry-level workers. For the entry-level group, the combined score from the mechanical test and work sample correlated .29 with foremen's six-month ratings on the new anchored form. For climbers, the sample is too small for a stable estimate; the combined score correlated .41 with ratings, but with eleven people that figure could easily be off by .25.

Sackett et al. (2022) reexamined the meta-analytic validity estimates that had guided selection research for decades and concluded that many had been inflated by corrections for range restriction that did not fit the data. Their revised estimates were lower for most predictors, and structured interviews and job-specific tools such as work samples ranked near the top. I therefore use conservative values in the utility analysis below rather than our own small-sample figures or older published estimates.

The honest question is not whether the new system is valid in some general sense, but how much better it predicts than the handshake it replaced.

Fairness

The physical assessment's impact ratio for women rose from about .31 under the old strength test to about .78, with five of eight women passing. Every other component showed impact ratios above .80 for sex, and none of the components showed a ratio below .80 for race or ethnicity, although our applicant pool is small enough that these figures need to be watched rather than celebrated. The rescue station accounts for most of the remaining gap on the physical assessment and is under review.

Applicant Reactions and Early Retention

In the post-process survey, applicants rated the work samples and interview as job related and fair; the personality scale we piloted and dropped had been the only component rated poorly. Ninety-day quits among entry-level hires fell from seventeen of thirty-one to eight of thirty-one. Nine applicants withdrew after the ride-along, which is a cost of a half day's pay each and a benefit of not training people who would have left.

Utility Analysis

Schmidt et al. (1979) showed how to express the value of a selection procedure in dollars. Each hire's yearly gain is the validity improvement times what a one-SD difference in job performance is worth in dollars, times the mean standard score of those hired on the predictor. They demonstrated that even modest validity gains could produce large productivity gains when applied across many hires.

The assumptions for our analysis are these. Old validity: the handshake and strength test are assumed to correlate about .15 with performance, a generous figure for an unstructured judgment. New validity: .35 for the combined system, below both our entry-level local estimate's upper range and the published values for structured interviews and work samples, in line with Sackett and colleagues' cautions. The validity gain is therefore .20.

The dollar value of a standard deviation of performance is estimated at 40 percent of average pay, a common rule of thumb; with average pay of about $40,000, that is $16,000. We hired 38 of about 95 applicants who completed assessment, a selection ratio of 0.40. With top-down selection on a normally distributed score, the average standardized score of those hired is about 0.97.

The gain per hire per year is then .20 × $16,000 × 0.97, about $3,090. With average tenure of about a year and a half, 38 hires produce roughly 57 person-years of work, for a gain of about $176,000. Assessment costs about $180 per applicant in staff time, materials and ride-along pay, or $17,100 for 95 applicants. The net estimate is about $159,000 for one year's hiring.

How Sensitive Is the Figure?

Every input above is an assumption, and the figure moves with each. If the validity gain is only .10, half my estimate, the net value falls to about $71,000. If the dollar value of performance is lower, or tenure shorter, it falls further. Top-down selection is also an idealization; we do not always hire the highest scorers, because track placement and availability matter. I would describe the defensible conclusion this way: across a reasonable range of assumptions, the system returns several times its cost, and the conclusion does not depend on the most optimistic numbers.

The utility model also leaves out the outcomes the company cares about most. One avoided fall, or one avoided struck-by injury, carries medical, legal and human costs far beyond a year's productivity gain. The utility estimate should be read as a floor.

Ongoing Review

The SIOP Principles call for selection procedures to be monitored and revisited as jobs and applicant pools change. The plan for year two includes four steps: a validity study on applicants rather than early hires, once at least eighty have been assessed; a station-by-station adverse impact review every six months; an annual refresh of interview questions; and a repeat of the job analysis if the company adds crane or utility work that changes the climber role.

Conclusion

The new system is better supported, fairer and better received than the one it replaced, and early retention improved sharply. A conservative utility analysis based on Schmidt and colleagues' model, using validity values tempered by Sackett and colleagues' revisions, puts the first year's net value at about $159,000, and at about $71,000 even if the validity gain is half what I assumed. Combined with the safety outcomes the model does not count, the evidence supports keeping and refining the program.

References

Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068. https://doi.org/10.1037/apl0000994

Schmidt, F. L., Hunter, J. E., McKenzie, R. C., & Muldrow, T. W. (1979). Impact of valid selection procedures on work-force productivity. Journal of Applied Psychology, 64(6), 609-626. https://doi.org/10.1037/0021-9010.64.6.609

Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures (5th ed.). Industrial and Organizational Psychology, 11(S1), 1-97. https://doi.org/10.1017/iop.2018.195