GB891 Unit 5 Pilot Test Summary Example

Three Problems Found Before Any Real Data: A Unit 5 Pilot Test Summary for a Bakery Safety Study

[Student Name]

Purdue University Global

GB891: Doctoral Project II: Project Design and Implementation

Unit 5 Pilot Test Summary

[Instructor Name]

[Date]

Corliss Baking Company, its pilot plant and every figure are composite and illustrative; the methods sources cited are real.

What the Pilot Was For

A pilot study is not a small version of the main study, and its results say nothing about whether the intervention works. Leon et al. (2011) argue that pilot studies should be used to test feasibility, the procedures, instruments and logistics a full study will depend on, and that effect sizes estimated from a pilot are too imprecise to guide anything. This pilot was designed accordingly. It asked three questions: do associates understand the pulse items as intended, can observers agree on the observation definitions, and can associates use the near-miss form?

Methods

The pilot ran for three weeks on two crews at the Jasper plant, the company's fifth bakery, which is excluded from the main study. The research assistant ran three weekly pulse cycles on both crews, drawing four members per crew each week. The two observers conducted nine joint observation sessions across the five stations, recording independently. And the assistant conducted cognitive interviews with seven associates, in which each person talked through their reasoning while answering the pulse items and the survey and then restated each question as they understood it, an approach Willis (2005) describes for finding where respondents' understanding departs from the designer's intent.

Finding One: The First Pulse Item

The first pulse item, in version 1.0, asked: "How many times this week did your supervisor talk about safety?" Scores were high on both crews, averaging more than four occasions a week. The cognitive interviews explained why. Five of the seven associates counted the mandatory stretch-and-flex routine at the start of every shift, which the supervisor leads and which includes a sentence about working safely. By that count, every supervisor talked about safety at least five times a week, and the item could not distinguish supervisors who raised safety at other times from those who did not.

Revision. Version 1.1 asks: "Outside the start-of-shift meeting, how many times this week did your supervisor bring up safety with you or your crew?" In the third pilot week, with the revised item, average scores fell to about one occasion a week and varied between associates in a way that matched what the observers had seen.

Finding Two: Agreement on Jam Clearing

Observers agreed well at four stations, with kappa values in the range Landis and Koch (1977) describe as substantial or better. At the jam-clearing station on the packaging line, they agreed in only 61 percent of intervals, and kappa was low. The disagreement came from the definition. Version 1.0 described safe jam clearing as stopping the line and using the clearing tool. Observers disagreed about intervals in which an associate stopped the line but reached in briefly before picking up the tool, and about intervals in which the line had already stopped for another reason.

Revision. Version 1.1 narrows the definition to hand position at the moment the line stops: an interval is safe if, when the line comes to a stop, the associate's hands are outside the guard and the tool is in hand or within reach. In six further joint intervals at the end of the pilot, the observers agreed in all six, which is encouraging but too few to establish reliability; agreement will be checked again in the first week of the main study.

Finding Three: The Near-Miss Form

Corliss's near-miss form was accessed by scanning a QR code posted at each station. Version 1.0 asked for the associate's badge number, name, shift, station, a description and a suggested fix. Associates at the pilot plant took about four minutes to complete it, and three of the seven interviewed said they would not use it because the badge number made them identifiable. The form defeated the purpose the framework assigns to near-miss reporting, which depends on associates believing reports are welcome and safe.

Revision. Version 1.1 has five fields, station, shift, what happened, what could have happened and a suggestion, and no identifier. It takes under two minutes on a phone.

The near-miss form asked for the one piece of information that made associates unwilling to fill it in.

What the Pilot Could Not Test

The pilot could not test whether the feedback sheets change supervisors' behavior, because three weeks with two crews is far too short and too small, and because testing the intervention was never its purpose. It could not test how the procedures hold up under high turnover, because the pilot crews happened to be stable during those three weeks. And it could not test the Haitian Creole materials, since no pilot associates spoke that language. These limits are carried into the main study as risks to be watched, not as problems solved.

Changes Carried Into the Protocol

All three revisions are recorded in the protocol's change log with the date and reason, and the version numbers on every instrument have been updated to 1.1 before the main study begins.

References

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310

Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626-629. https://doi.org/10.1016/j.jpsychires.2010.10.008

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE.