NU504 Unit 6 seminar reflection example

One Significant Result in a Table of Many: A Seminar Reflection on Multiple Outcomes in a Randomized Trial of Earplugs and Eye Masks

[Student Name]

Purdue University Global

NU504: Scientific and Analytic Approaches to Advanced Evidence-Based Practice

Unit 6 Seminar

[Instructor Name]

[Date]

Written alternative to the live seminar. The trial discussed is published; the posted questions are the instructor's.

Question 1: What Does the Results Section Report?

The trial randomized 64 ICU patients to sleep with or without earplugs and an eye mask and recorded polysomnography on the first night (Demoule et al., 2017). The primary outcome was the proportion of total sleep time spent in N3, deep sleep. It was 21 percent with the aids and 11 percent without, p = 0.09, which did not reach the prespecified threshold. The results section then reports a series of secondary sleep descriptors and clinical outcomes. Two comparisons reached significance. Prolonged awakenings were fewer with the aids, 21 against 31, p = 0.02. And among patients who wore the earplugs all night, N3 duration was longer than in the control group, 74 minutes against 31, p = 0.039. Nine of the 30 patients in the intervention group, 30 percent, did not wear the earplugs all night, and no clinical outcome differed between groups.

The inventory matters because the abstract's conclusion, that earplugs and eye mask reduce long awakenings and increase deep sleep when they are well tolerated, rests on the two significant comparisons. A reader who sees only the conclusion does not learn that the primary outcome was not met.

Question 2: How Many Significant Findings Would Chance Alone Produce?

If a trial runs a number of independent tests at the 0.05 level and the intervention does nothing, the probability of at least one significant result is 1 minus 0.95 raised to the number of tests. With ten tests, that is about 40 percent; with twelve, about 46 percent. So a single significant secondary result, in a table of many, is roughly what a trial of an ineffective intervention would be expected to produce nearly half the time.

Two cautions apply. Sleep outcomes are correlated with one another, since a patient with more deep sleep usually has fewer long awakenings, so the tests are not independent and the true chance of a false positive is lower than those figures. The comparison among full-night wearers is a different problem: it compares patients who kept the earplugs in with all controls, which is no longer a randomized comparison. Patients who tolerate earplugs all night may simply be the ones who were going to sleep better anyway.

Question 3: What Could the Authors Have Done?

Three remedies are available, and none requires blaming the authors, who prespecified their primary outcome and reported it honestly. First, label the secondary outcomes as exploratory and say so in the conclusion. Second, adjust for multiple testing, or explain why not; adjustment is not always the right choice, but the decision should be stated (Bender & Lange, 2001). Third, report every secondary outcome with an effect estimate and interval, so a reader can judge size and precision rather than counting p values. The per-protocol comparison should be presented as what it is, a comparison among patients who tolerated the intervention, alongside the randomized result.

The Same Problem in the Unit 5 Trial

The seminar questions sent me back to the trial I critiqued last unit. That trial also named two primary outcomes, delirium and sleep perception, and it reported its cognitive result as a combined outcome of delirium or mild confusion (Van Rompaey et al., 2012). Two primary outcomes double the chances of a significant result under the null, and combining categories adds a further choice an analyst can make after seeing the data. I am not suggesting either choice was made to find significance; the protocol was registered. The point is that the same habits of reading apply to both trials. Polit and Beck (2021) describe a transparent results section as one that reports every outcome with its estimate, not only those that crossed a threshold, and both trials fall somewhat short of that.

Reflection

A month ago, I would have put "earplugs and eye masks increase deep sleep, p = 0.039" into my evidence table without a second thought. The exercise showed me how far that sentence is from what the trial found: a primary outcome that was not met, a secondary result among many, and a comparison that had given up randomization. The trial is still useful. It shows that about a third of ICU patients may not tolerate earplugs all night, which is a practical finding for any unit planning to use them, and it points in a plausible direction on deep sleep.

The rule I will carry into the synthesis is simple: a secondary finding is reported as hypothesis-generating unless it was prespecified and the analysis accounted for multiple comparisons. That rule will change how I describe at least two rows of my evidence table.

References

Bender, R., & Lange, S. (2001). Adjusting for multiple testing: When and how? Journal of Clinical Epidemiology, 54(4), 343-349. https://doi.org/10.1016/S0895-4356(00)00314-0

Demoule, A., Carreira, S., Lavault, S., Pallanca, O., Morawiec, E., Mayaux, J., Arnulf, I., & Similowski, T. (2017). Impact of earplugs and eye mask on sleep in critically ill patients: A prospective randomized study. Critical Care, 21(1), 284. https://doi.org/10.1186/s13054-017-1865-0

Polit, D. F., & Beck, C. T. (2021). Nursing research: Generating and assessing evidence for nursing practice (11th ed.). Wolters Kluwer.

Van Rompaey, B., Elseviers, M. M., Van Drom, W., Fromont, V., & Jorens, P. G. (2012). The effect of earplugs during the night on the onset of delirium and sleep perception: A randomized controlled trial in intensive care patients. Critical Care, 16(3), R73. https://doi.org/10.1186/cc11330