GB793 Unit 5 Levels of Analysis Exercise Example

From What I Experience to What the Branch Has: A Unit 5 Levels of Analysis Exercise on a Bank's Service Climate Study

[Student Name]

Purdue University Global

GB793: Advanced Topics in Management

Unit 5 Levels of Analysis Exercise

[Instructor Name]

[Date]

Westover Savings, its branches and every statistic are composite and illustrative; the methods and studies cited are real.

The Study's Claim

The bank's internal study claims that branches with a stronger climate for service earn higher customer satisfaction. It surveyed 702 employees across the 58 branches with a twelve-item service climate scale, averaged their responses by branch and related those averages to customer satisfaction ratings collected by an outside firm. It reported a positive and significant relationship and recommended that branch managers be evaluated partly on their climate scores.

Service climate, as Schneider et al. (1998) defined it, refers to employees' shared perceptions of the practices, procedures and behaviors that are rewarded and supported with regard to customer service. The word shared matters: climate is a property of a unit, not of an individual. The audit asks whether the study treated it that way at every step.

Row One: Level of Theory

The theory is stated at the branch level. The claim is that branches, not individual employees, differ in climate, and that those differences relate to branch-level customer satisfaction. This is the right level for the construct, since customers experience a branch, not a single employee's private perception. Verdict: aligned.

Row Two: Level of Measurement

For a branch-level construct measured through individual responses, the items should direct each respondent to report on the unit, not on their own experience. Chan (1998) called this a referent-shift consensus model: the item's referent shifts from "I" to "we" or "employees in this branch," and agreement among respondents is then evidence that a shared property exists. Of the twelve items in Westover's scale, nine ask what the respondent personally experiences, for example whether the respondent receives the training needed to serve customers well. Only three ask about the branch, for example whether employees in this branch are recognized for excellent service.

An item about one's own training measures an individual's experience, which may vary within a branch for legitimate reasons, such as tenure. Averaging such items produces a branch mean of individual experiences, which is a different construct from shared climate. Verdict: misaligned for nine of twelve items.

Row Three: Level of Analysis

The study's analysis did not use branch averages after all. The customer satisfaction score for each branch was assigned to every employee in that branch, and the relationship was estimated across 702 individuals, as if each were an independent observation. But employees in the same branch share the same customer score, so the 702 observations contain only 58 independent values of the outcome. Treating them as 702 overstates the evidence: standard errors are too small, and the relationship appears more certain than it is. The appropriate analysis would relate the 58 branch means to the 58 customer scores, or use a multilevel model that recognizes employees nested within branches. Verdict: misaligned.

The study's theory was about 58 branches, but its analysis counted 702 people, and the extra 644 observations added certainty the data did not contain.

The Agreement Statistics

Before averaging individual responses to form a branch score, a study should show that employees within a branch agree and that branches differ. Westover's data yield three statistics.

The median rwg across branches is .71. This index compares the observed variance of responses within a branch with the variance expected if responses were random. LeBreton and Senter (2008) suggest that values around .70 indicate moderate agreement. A median of .71 means that, in a typical branch, employees agree somewhat, but in many branches agreement is weaker than that.

The ICC(1) is .09. This statistic estimates the proportion of variance in individual responses that is due to branch membership. A value of .09 means that 9 percent of the differences among employees' answers reflect which branch they work in. That is within the range commonly reported for unit-level constructs, and it shows that branches do differ, though most of the variation is within branches.

The ICC(2) is .54. This statistic estimates how reliably the branch means distinguish one branch from another, given the number of respondents per branch. With about twelve respondents per branch, a value of .54 means branch means are only moderately reliable. Correlations between branch means and customer scores will therefore be attenuated, understating the true branch-level relationship.

What the Audit Implies

The three problems pull in different directions. The individual-level analysis overstates the precision of the result. The low reliability of branch means understates its size. And the wording of most items means the measure captures averaged individual experiences rather than shared climate. The study's positive finding is therefore difficult to interpret: some relationship between employee perceptions and customer satisfaction likely exists, but the study cannot say how strong it is, how certain it is or whether it reflects climate as a shared property.

Recommended Revisions

Three revisions would align the study. Reword the nine individual-referent items so that each refers to the branch, following the referent-shift approach, and confirm agreement with rwg and ICC statistics on the revised scale. Analyze at the branch level, using 58 branch means or a multilevel model, so that the evidence reflects the actual number of independent units. And increase the number of respondents per branch where possible, or report the ICC(2) alongside any correlation, so that readers can judge how much attenuation to expect. Evaluating managers on climate scores should wait until the measure has been revised and its branch-level reliability shown.

References

Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83(2), 234-246. https://doi.org/10.1037/0021-9010.83.2.234

LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852. https://doi.org/10.1177/1094428106296642

Schneider, B., White, S. S., & Paul, M. C. (1998). Linking service climate and customer perceptions of service quality: Tests of a causal model. Journal of Applied Psychology, 83(2), 150-163. https://doi.org/10.1037/0021-9010.83.2.150