PS390 Unit 3 Reliability and Validity Paper Example

Two Interviewers, One Candidate, Two Different Scores: A PS390 Unit 3 Reliability and Validity Paper

[Student Name]

Purdue University Global

PS390: Introduction to Industrial/Organizational Psychology

Unit 3 Reliability and Validity Paper

[Instructor Name]

[Date]

The writer, the call center, the candidates and every figure are composite and illustrative; the sources cited are real.

Why Measurement Matters in Hiring

Our Charlotte utility call center replaced its unstructured interview with a six-question structured interview and a work sample in January. A new process is only useful if it measures something consistently and if the scores support the decisions we make with them, in our case whom to hire. Those two qualities are reliability and validity, and the Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014) set out how professionals should establish both for any test used in decisions about people.

Consistency

A measure is reliable when it gives the same reading again, across raters, occasions or items, as long as the person measured has not changed. A measure that gives one candidate a 4 with one interviewer and a 2 with another tells us more about the interviewers than the candidate.

For an interview, the most important kind is interrater reliability, the agreement between two people rating the same answers. Conway et al. (1995) conducted a meta-analysis of interview reliability and found that structured interviews produced substantially higher interrater reliability than unstructured ones, and that reliability was highest when interviews used standardized questions, rating scales with clear anchors and trained interviewers.

What the First Month Showed

In January, two supervisors independently rated each of twenty-four candidates on all six questions. For five questions, their ratings agreed closely; the correlation between their scores was in the high .70s or above. For the sixth, which asked candidates to describe picking up a new skill under time pressure, agreement was much lower. Looking at the ratings, one supervisor gave high scores to candidates who described learning anything quickly, while the other gave high scores only to those who described learning a computer system under time pressure.

The question was fine; the anchors were vague enough that each rater wrote a different question in her head.

We rewrote the anchors to describe what a strong answer includes, such as learning a new system or set of rules, practicing it and checking accuracy, and held a thirty-minute calibration session in which both supervisors rated three recorded answers and discussed differences. In February, agreement on that question rose to match the others, and we added a short calibration session to every new interviewer's training.

Why Structure Helps Consistency

Campion et al. (1997) described many separate components of interview structure, and several of them work mainly by reducing the room for raters to disagree. Asking every candidate the same questions removes differences that come from different conversations. Scoring each reply the moment it ends, against its own described examples, stops one dazzling or clumsy answer from tinting all the others. Using more than one interviewer averages out individual quirks. And training interviewers on what each scale point looks like gives them a shared standard. Our sixth question had every component except the last one in practice: the anchors existed but were too vague to give the raters a shared picture of a strong answer. Calibration supplied it.

Checking the Work Sample

The work sample needed its own check. Two trainers scored the same twelve recorded simulated calls using the checklist. They agreed on most items, such as whether the candidate verified the account number and explained the charge correctly, but disagreed on "showed empathy," an item that meant different things to each of them. We replaced it with two observable items: the candidate acknowledged the caller's concern in words, and the candidate did not interrupt while the caller explained. Agreement on those items was high from the first round. Observable behaviors are easier to rate consistently than impressions, a lesson that applied to both parts of the process.

Consistency Across Items and Time

A third kind of consistency checks whether questions meant to tap one quality move together. Our six questions measure different requirements, so we do not expect them all to correlate highly, and internal consistency is less useful here. For the work sample, we check whether the same recorded call scored a week apart gets the same score from the same rater, a form of consistency over time.

Validity

Validity is not a property of a test alone. In the Standards' terms, validity is about whether the evidence justifies what we do with a score, here deciding whom to hire. Our use is predicting who will succeed as a representative, so we need evidence that higher scores go with better job outcomes.

Several kinds of evidence contribute. Content evidence comes from our job analysis: each question and the work sample are tied to a requirement that high performers and supervisors identified. Criterion-related evidence comes from setting scores beside what later happened to each hire, such as whether they stayed three months and how their early calls were rated. And evidence about consequences asks whether the process treats groups fairly and leads to good decisions.

Plan for Criterion-Related Evidence

We will wait until at least sixty new hires have completed their first quarter, then correlate interview and work sample scores with quality scores and compare retention for candidates who scored high and low. We will check whether the work sample adds prediction beyond the interview. Because we hire only candidates who score above a cutoff, our sample will be restricted in range, which tends to make correlations look smaller than they would be in the full applicant pool, and we will note that limitation. We will repeat the analysis after the second year of hiring, when the larger sample will give steadier estimates, and share both sets of results with the supervisors who do the interviewing so they can see what their ratings predict.

Reliability Limits Validity

A measure cannot predict outcomes better than it can measure consistently. If two interviewers disagree about a candidate, at least one of them is wrong about the candidate, and that noise weakens any relationship with job performance. Improving the sixth question's agreement was therefore a step toward validity as well.

Conclusion

Two supervisors disagreed on one interview question until clearer anchors and a calibration session brought them together, consistent with Conway and colleagues' finding that standardized, anchored, trained interviews are most reliable. The Standards define validity as evidence for a specific use, and the call center's plan gathers content, criterion-related and fairness evidence for hiring decisions over its first year.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Campion, M. A., Palmer, D. K., & Campion, J. E. (1997). A review of structure in the selection interview. Personnel Psychology, 50(3), 655-702. https://doi.org/10.1111/j.1744-6570.1997.tb00709.x

Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565-579. https://doi.org/10.1037/0021-9010.80.5.565