GB740 Unit 9 Change Evaluation Plan Example

Baseline First, Lottery Second: A Unit 9 Change Evaluation Plan for a Civil Engineering Firm

[Student Name]

Purdue University Global

GB740: Organizational Development

Unit 9 Change Evaluation Plan

[Instructor Name]

[Date]

Tallis Engineering, its offices and every figure are composite and illustrative; the methods sources cited are real.

What Success Means, Stated in Advance

The intervention aims to shrink the margin lost on work that borrows staff across office lines. Success is defined before rollout so that it cannot be redefined afterward to fit whatever happens. The primary outcome is fade on multi-office projects, measured as the difference between the margin in the winning proposal and the estimated margin at completion six months after the project starts. The six-month estimate is used because most projects take more than a year to close, and waiting for final margins would delay the evaluation beyond the point where it could inform decisions. For every project that closes during the evaluation period, the six-month estimate will be checked against the final margin, so that the reliability of the earlier measure is known.

The success threshold is a reduction of at least two percentage points in median fade on multi-office projects in changed offices, compared with offices that have not yet changed, within twelve months of each office's start. Two points would recover about a quarter of the gap between multi-office and single-office fade, and on the firm's multi-office fee volume of roughly $38 million a year it would be worth about $760,000.

The Baseline

Twenty-four months of project history, covering 2024 and 2025, supply the baseline. For each office, the plan records median fade on multi-office and single-office projects, the number of each and the share of hours worked across offices. The baseline confirms the Unit 5 finding at office level and shows how much fade varies from one office to another and from one quarter to the next, which tells the evaluation how large a change must be to stand out from ordinary variation.

The Rollout as a Design

Unit 7 scheduled the twelve offices to begin in three groups, half a year apart. The evaluation turns that schedule into a stepped design. Offices are grouped into three size bands, and within each band a lottery decides which wave each office joins, with the exception that the five acquired offices with earnout terms join the last wave, for reasons set out in Unit 7. Hussey and Hughes (2007) set out the logic of such stepped designs, where groups switch to the new condition one step at a time in an order fixed by chance, so that every cluster eventually receives the intervention and, at each step, those still waiting serve as comparisons.

The lottery matters because it prevents the executive team from choosing the most cooperative offices for the first wave. If offices were chosen, any improvement in the first wave could reflect the offices' readiness rather than the intervention. Random order within size bands means that, apart from chance and the earnout exception, early and late offices should be similar.

Secondary Measures

Five secondary measures check that any gain arrives by the route the diagnosis predicted. The share of hours worked across offices should rise, since shared credit makes lending staff less costly. Days to fill a staffing request, recorded by the forum, should fall. Project manager turnover, measured over twelve months in each office, should fall as coordination becomes paid work. Role ambiguity, measured with the same adapted scale from Rizzo et al. (1970) used in the assessment, will be repeated twelve months after each office's start. And unfilled requests by specialty, a measure added after the stakeholder analysis, will show whether scarce specialists are the bottleneck the senior engineers' concerns suggested.

If fade falls but none of these secondary measures move, the improvement probably comes from something other than the intervention. If they move but fade does not, the mechanism works but is not enough.

How Results Will Be Analyzed

Fade for each multi-office project will be compared between offices that have changed and those that have not, at each step, adjusting for project fee, client type and calendar period, since the whole firm may improve or worsen over time for reasons unrelated to the intervention. Results will be reported as differences in median fade with intervals. With only twelve offices, the analysis will have limited power to detect small effects, and the plan states this rather than hoping for significance.

With twelve offices, the evaluation can detect a large effect or rule out a large effect, but it cannot settle a small one, and the plan says so before the data arrive.

Rival Explanations the Design Cannot Exclude

Four rival explanations remain. Attention: offices may improve simply because they know they are being watched, a reaction that would appear in early waves and fade later. Comparing each wave's first and second six months will show whether gains persist. Contamination: offices still waiting may adopt parts of the new arrangements informally, for example by trading favors less, once they see the forum working. That would shrink the measured difference. Market change: if public infrastructure funding rises or falls during the evaluation, project margins may move for reasons unrelated to the intervention, though this should affect changed and waiting offices alike. The earnout exception: because the acquired offices all join last, any difference between them and other offices will be mixed with the effect of the last wave's timing.

The design reduces some threats through the lottery and the comparison at each step, as Shadish et al. (2002) explain for designs with staggered starts, but it does not remove them. Each will be discussed when results are reported.

Reporting

Results will be shared with the executive team and the project manager council every six months, in the same plain format used for the Unit 5 findings. A decision point is set at eighteen months, after the second wave has run for a year: if the threshold is not met in either of the first two waves, the executive team will review the interventions before the third wave begins.

References

Hussey, M. A., & Hughes, J. P. (2007). Design and analysis of stepped wedge cluster randomized trials. Contemporary Clinical Trials, 28(2), 182-191. https://doi.org/10.1016/j.cct.2006.05.007

Rizzo, J. R., House, R. J., & Lirtzman, S. I. (1970). Role conflict and ambiguity in complex organizations. Administrative Science Quarterly, 15(2), 150-163. https://doi.org/10.2307/2391486

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.