HD520 Unit 9 Limitations Analysis Example

Five Decisions and the Claims Each One Limits: A Unit 9 Limitations Analysis for a Refugee Health Orientation Assessment

[Student Name]

Purdue University Global

HD520: Health Education Program Assessment

Unit 9 Limitations Analysis

[Instructor Name]

[Date]

The agency, the assessment and its results are composite; the sources on observer bias, quasi-experimental design, cognitive interviewing and reporting limitations cited are real.

Approach

Limitations sections often list weaknesses any study could have, such as small samples or self-report, without saying what they change. Reviews of published research find limitations are frequently acknowledged only briefly and rarely tied to the conclusions they weaken (Ioannidis, 2007). This analysis takes a different approach. Each row below starts from a decision made during the assessment, names the limitation that decision produced and states the specific claim it restricts. A paragraph under each row estimates whether the limitation more likely inflates or deflates the main result, a 21-point difference in task completion between early starters and waiting adults, and by roughly how much.

The rows are ordered by how much each threatens that main claim.

Row 1: Cohorts Formed by Scheduling

Decision: compare early starters with adults who waited for a seat, rather than randomize. Limitation: who waited depended on when they arrived, and arrival timing tracked country of origin. Claim restricted: that the 21-point difference reflects the class alone.

Most Swahili-speaking families arrived in a spring surge and waited; they made up 22 percent of waiting adults and 9 percent of early starters. The analysis adjusted for language, but language stands in for other differences, such as years in camps or the quality of prior schooling, that were not measured. Nonequivalent comparison groups leave this kind of selection as the main rival explanation (Shadish et al., 2002). If those unmeasured differences slowed learning among waiting adults, the estimate is inflated. The adjusted and unadjusted estimates differed by only one point, which suggests the measured imbalance had little effect, but the unmeasured part cannot be judged from these data.

Every limitation in this table began as a reasonable decision, and naming the decision first shows a reader what would have to change to remove it, instead of leaving the weakness as a fact of nature.

Row 2: A Simulated Call

Decision: measure ability with a scripted call to an assessor playing a scheduler. Limitation: the simulation omits hold times, voicemail, phone menus and after-hours lines, all common at real clinics. Claim restricted: that graduates can arrange care at a real clinic.

The task probably overstates real-world performance in both groups, since a patient caller in a quiet agency office meets none of the obstacles of a busy phone line. Its effect on the difference between groups depends on whether the class taught the script itself. Educators reported practicing similar calls in class, so early starters may have rehearsed the task's exact form, which would inflate the difference. The interview finding that some task completers missed real appointments supports this concern.

Row 3: One Agency, One Clinic

Decision: run the assessment at a single agency with a single partner clinic. Limitation: Calder City's interpreter line answers within a minute and the partner clinic accepts direct scheduling by phone. Claim restricted: that the class would produce similar results elsewhere.

This limitation does not bias the estimate for Calder City, but it limits transfer. Cities where interpreter lines are slow or clinics require online portals would test different skills. The report will describe the local services in enough detail for other agencies to judge similarity. Agencies serving newer language groups, where interpreters are scarce, may see smaller gains.

Row 4: Translation Checks for Two Languages of Three

Decision: complete cognitive interviewing in Dari and Arabic before baseline and schedule Swahili later, because the Swahili interpreter was hired late. Limitation: whether Swahili speakers understood the scenario card as intended was not tested. Claim restricted: any statement about Swahili speakers' scores.

Cognitive interviews often reveal that translated items are understood differently from the original (Willis, 2005). Because Swahili speakers were concentrated in the waiting group, a confusing card could have lowered their scores and inflated the main difference. This is also the one place where sample size binds: with 9 early starters and 19 waiting adults, no estimate for Swahili speakers alone is meaningful, and the report draws none.

Row 5: Assessors Knew Each Participant's Cohort

Decision: use the same small assessor team for all calls to keep scoring consistent. Limitation: assessors knew which adults had taken the class. Claim restricted: that scoring was neutral between groups.

In a systematic review of trials scored by both blinded and nonblinded assessors, nonblinded assessment exaggerated odds ratios by about a third on average (Hróbjartsson et al., 2012). The checklist here is fairly concrete, such as whether a date was repeated back, which limits room for judgment. A blinded second rater scored a fifth of calls from recordings, and agreement on completion was high, with a kappa of 0.81. The bias is therefore probably small but, if present, would inflate the difference.

Overall Direction

Four of the five limitations would, if they operate, make the class look more effective than it is; none clearly works the other way. Combined with the attrition analysis in Unit 8, which put a pessimistic estimate near 11 points, the true difference is more likely in the lower half of the reported interval than the upper. The report therefore advises the agency to treat the result as evidence of a real but possibly modest benefit.

What a Repeat Assessment Would Change

Each limitation points to a change. Assessors scoring from recordings without knowing the cohort would address Row 5. Completing cognitive interviews in every language before baseline would address Row 4. Adding real appointment tracking for all consenting adults within 42 days would test Row 2 directly. Row 1 cannot be removed without randomizing, which the agency rejected, but recording arrival circumstances and schooling would allow better adjustment. A second agency in another city would speak to Row 3.

Limitations Considered and Left Out

Two entries common in limitations sections were considered and left out. Self-report was not the main outcome, so social desirability does not threaten the main claim, though it does apply to the confidence item, which is reported only as secondary. Overall sample size was set by the roster, and its consequence, an interval wide enough to include small effects, is already stated in the results; repeating it here would tell a reader nothing new. Leaving these out keeps the table to limitations that change what the report can say.

References

Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2012). Observer bias in randomised clinical trials with binary outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. BMJ, 344, e1119. https://doi.org/10.1136/bmj.e1119

Ioannidis, J. P. A. (2007). Limitations are not properly acknowledged in the scientific literature. Journal of Clinical Epidemiology, 60(4), 324-329. https://doi.org/10.1016/j.jclinepi.2006.09.011

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.