NU524 Unit 6 seminar reflection example

Two Peers Chose the Answer I Had Not Keyed: A Seminar Reflection on Reading My Own Stems With the Key in Mind

[Student Name]

Purdue University Global

NU524: Curriculum Development Process, Assessment, and Evaluation Strategies

Unit 6 Seminar

[Instructor Name]

[Date]

Peers are described by role only; the items are the writer's own exercise items.

The Format

Each writer's items were read aloud. Peers answered silently on paper, and only then was the key revealed and debated. I had reviewed my five items twice against the item-writing rules before the seminar and was fairly confident in them.

The Heart Failure Item

My item described a man in his early seventies, living with heart failure, who by phone reported three nights of sleeping in his recliner and a 3.5-pound weight gain since Monday, and asked for the nurse's best response. I had keyed "arrange for him to be seen today." Two of four peers chose the option telling him to take one more furosemide dose that night.

My first reaction was that they were wrong: a nurse cannot change a dose without an order. Then one of them, a heart failure clinic nurse, explained her choice. Many heart failure patients are given a written flexible diuretic plan that tells them to take an extra dose when their weight rises by a set amount, and in her clinic the first response to this call would be to ask whether he has one and follow it. The stem said nothing about whether he did. My item was not wrong about heart failure; it was silent on a fact that decides the answer, and I had not noticed because I already knew which answer I meant.

Revision: I added "He has no written plan for adjusting his diuretic" to the stem. With that sentence, the keyed answer is clearly best, and the distractor still tempts students who remember that diuretics treat fluid overload.

Two Smaller Flaws

Peers caught two more. On the COPD item, two of my options described interpretations ("his saturation of 99 percent as reassurance") while the other two described actions, so the options were not parallel and a test-wise student could eliminate the odd pair. I rewrote all four as findings to address. On the pulmonary embolism item, "anxiety about discharge" was the only option that was not a physiological explanation, and every peer eliminated it at once; it was measuring nothing. I replaced it with pneumothorax, which is plausible after surgery and requires the student to weigh the timing and findings.

Why I Could Not See Them

Downing (2002) describes flaws in locally written items as a common source of construct-irrelevant variance: items that are harder or easier for reasons unrelated to what they are meant to measure. Two of my flaws fit that description exactly. The heart failure item was harder for students who knew about flexible diuretic plans, the more knowledgeable students, which is the reverse of what a good item does. The implausible distractor made the pulmonary embolism item easier for test-wise students regardless of their knowledge. Guidelines for item writing, such as keeping options homogeneous and making every distractor plausible, are easy to state and hard to check in one's own work (Haladyna et al., 2002).

The deeper lesson is about perspective. When I read my own stem, I read it with the key already in my mind, so I supplied the missing facts without noticing. Peers who did not know the key read the stem as a student would.

What the Other Three Items Showed

My acute coronary syndrome, chest tube and revised COPD items drew unanimous agreement with the key, and I learned something from that too. On the chest tube item, all four peers said they had eliminated "clamp the chest tube" immediately because it is a well-known danger, which raises the question of whether that distractor would tempt any student at all. The group agreed to keep it, because weaker students do sometimes choose it on real exams, but to watch its selection rate in the item analysis. On the acute coronary syndrome item, one peer said the stem would be stronger if it mentioned the patient's cardiac history, so that students could not dismiss the chest pressure as pneumonia-related; I added a line noting prior stents.

What I Will Do Differently

I will not treat my own review as enough. Before any item goes on an exam, I will ask at least one colleague to answer it without the key, and I will ask specifically what facts she assumed. I will also add a step to my review: for each stem, list every fact the keyed answer depends on and check that each one is actually in the stem. Nursing evaluation texts recommend peer review of test items for exactly this reason (Oermann & Gaberson, 2019), and the seminar showed me why.

References

Downing, S. M. (2002). Threats to the validity of locally developed multiple-choice tests in medical education: Construct-irrelevant variance and construct underrepresentation. Advances in Health Sciences Education, 7(3), 235-241. https://doi.org/10.1023/A:1021112514626

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5

Oermann, M. H., & Gaberson, K. B. (2019). Evaluation and testing in nursing education (6th ed.). Springer Publishing.