PS451 Unit 3 Measurement Paper Example

An Alpha of .84 and a Validity of .18: A PS451 Unit 3 Measurement Paper

[Student Name]

Purdue University Global

PS451: Selection and Assessment in Organizations

Unit 3 Measurement Paper

[Instructor Name]

[Date]

The writer, the tree care company, the vendor tests and every figure are composite and illustrative; the sources cited are real.

The Decision

Ground workers at our Chattanooga company drag brush, feed the chipper, run the ropes that lower cut limbs and keep the drop zone clear. Last year, seventeen of thirty-one new ground workers left within ninety days, and one was struck by a falling limb after a warning call went unanswered. The general manager wanted a test that would pick safer, steadier ground workers, and a vendor offered two: a forty-item mechanical reasoning test about pulleys, levers and loads, and a twenty-item safety-orientation scale built from conscientiousness items. The brochure reported that the combination predicted safety ratings at .45.

Before buying, I wanted to know two things about each test: whether it measures consistently, and whether its scores support the use we would make of them.

Reliability

Reliability is about consistency. A test that gives very different scores to the same person on different days, or whose items do not hang together, cannot support any decision. The vendor reported coefficient alpha of .84 for the mechanical test and .71 for the safety scale.

Cortina (1993) explains what alpha does and does not show. The figure reflects how tightly the items track each other, yet it climbs as items are added, which lets a lengthy test reach a respectable value from items that agree only loosely. A strong alpha is also no proof of a single underlying ability; a test that mixes two or three separate clusters of items can post an impressive figure anyway. And alpha says nothing about stability over time.

Read that way, the mechanical test's .84 across forty items is reasonable but not remarkable, and the safety scale's .71 across twenty items is adequate for comparing groups but thin for deciding about individuals. To check stability, I gave the mechanical test to twenty-five current ground workers twice, two weeks apart. Scores correlated .79 between sessions, which is acceptable.

Validity

Validity is about whether scores support the inferences drawn from them, and it is the more important question. The vendor's .45 came from a single study at a utility contractor with a different job and a sample of about sixty. I could not find any other evidence for the combination.

So I ran a small local study. Forty-six current ground workers took both tests, and their foremen, who did not see the scores, rated each worker's safety behavior over the previous six months on a short behavioral checklist. The mechanical test correlated .18 with the safety ratings and the safety scale correlated .22. Together they correlated about .26. Those are modest values, far below the brochure's figure.

The vendor's number came from someone else's job; ours came from our own crews, and it was a third of the size.

Why the Values Are Modest

Several things limit a local study like this. Forty-six workers is a small sample, and the estimate could easily be off by .15 in either direction. Current employees have already survived hiring and the first ninety days, so the range of scores is narrower than it would be among applicants, which lowers correlations. Foremen's ratings are themselves imperfect; foremen who like a worker may rate his safety behavior generously. Each of these factors pushes the observed values down, so the true relationships may be somewhat larger. None of them would turn .26 into .45.

What the Research Says

Barrick and Mount (1991) pooled the personality and performance research for five groups of occupations, one of them skilled and semiskilled trades like ours. Only conscientiousness held up everywhere, for every occupational group and every kind of performance measure, and even it predicted modestly. That supports including a conscientiousness-based scale, with modest expectations, which is what our local data show.

Christian et al. (2009) analyzed the person and situation factors behind safety. What workers knew about safe practice and how much they cared to follow it predicted safe behavior best, with their sense of how seriously the workplace took safety close behind. For accidents and injuries, though, the group's safety climate showed the strongest association. Personality traits mattered, but less directly.

That finding changes the decision. Even a well-chosen test can only influence who enters a crew. Whether that person stays out of the drop zone depends heavily on what the crew treats as normal. The ground worker struck last year was new, but the climber had dropped the limb without waiting for anyone to answer his warning, and two experienced ground workers nearby had assumed the new man heard it.

The Mechanical Test's Other Purpose

The mechanical test's weak link to safety ratings does not mean it is useless. Ground workers rig lowering lines and run friction devices that rely on the same principles the test covers. When I correlated test scores with foremen's ratings of rigging skill specifically, the relationship was .31. The test seems better suited to predicting rigging competence than general safety, which makes sense given what it measures.

Recommendation

I recommended buying the mechanical reasoning test but not the safety scale as a hiring screen. The mechanical test will be one part of a ground worker composite, alongside a short work sample on the chipper and drop zone, and its use will be framed around rigging ability. The safety scale's modest validity and adverse reactions from applicants, several of whom found its items intrusive, do not justify its cost when used to screen. We will instead use a version of it as a development tool after hire, to start conversations about safety habits.

The larger investment goes to climate: drop zone rules enforced by every foreman, a standard set of verbal calls between climbers and the ground and a monthly near-miss review on each crew. Christian and colleagues' findings suggest that this is where most of the injury reduction lies.

Ongoing Evaluation

As new hires accumulate, I will repeat the validity analysis on applicants rather than incumbents, which avoids the restricted range problem, and track both rigging ratings and near-miss reports. Scores will also be checked for group differences, as the previous unit's work on adverse impact requires.

Conclusion

The vendor's tests had acceptable reliability, but their validity for our ground workers was far lower than advertised. Cortina's analysis of alpha shows why internal consistency is not enough, Barrick and Mount support modest expectations for conscientiousness, and Christian and colleagues show that crew safety climate matters more for injuries than any trait. Keeping the mechanical test for rigging, dropping the safety scale as a screen and investing in climate fits the evidence.

References

Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1-26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x

Christian, M. S., Bradley, J. C., Wallace, J. C., & Burke, M. J. (2009). Workplace safety: A meta-analysis of the roles of person and situation factors. Journal of Applied Psychology, 94(5), 1103-1127. https://doi.org/10.1037/a0016172

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. https://doi.org/10.1037/0021-9010.78.1.98