GB532 Unit 7 Seminar Reflection Example

Three of Five Readers Stalled on the Same Item: A Unit 7 Seminar Reflection on Pretesting a Propane Survey

[Student Name]

Purdue University Global

GB532: Marketing Research

Unit 7 Seminar Reflection

[Instructor Name]

[Date]

The propane distributor, the pretest participants and the seminar discussion are composite; the methods and sources cited are real.

The Seminar's Topic and My Test

The seminar was about pretesting: finding problems in a questionnaire before it reaches respondents. Before the session, I ran my own small test. Five propane customers, recruited from my family and a neighbor's road association, read each item of my draft aloud and told me what they thought it was asking, a simple version of what Willis (2005) calls cognitive interviewing, in which respondents think aloud so that the researcher can hear how they interpret each question.

What Broke

Three of the five could not answer item 9, which asked what year their first heat pump was installed. Two had bought homes that already had heat pumps, and one had a unit installed by a contractor while away at work. All three tried to guess and then asked whether "not sure" would count against them. The item assumed that the person answering had arranged the installation.

Two other problems appeared. Item 10, which asked what share of the home's heat came from each source in tenths, made two readers stop and count on their fingers; one said, "I have no idea if it's six-tenths or seven-tenths." And item 17, describing the backup-heat plan, confused a reader who said that paying for actual use was already how propane worked. The concept's key difference, a smaller fixed annual fee, was buried in the middle sentence.

The items that failed were the ones I had worked hardest on, because I had written them for the decision rather than for the person holding the pen.

What Classmates Made of It

A classmate who runs customer surveys for a utility said item 9's failure was common: questions about installation dates assume an owner who remembers buying the equipment. The classmate suggested asking first whether the equipment was already in the home at purchase, then offering age ranges rather than a year.

Another classmate questioned item 10 more deeply. Asking for tenths, that classmate argued, gives an appearance of precision that respondents cannot supply, and the decision needs only to know whether a home relies mostly on the heat pump, mostly on propane or about equally on both. Groves et al. (2009) note that respondents asked for more precision than they can recall often give rounded or invented answers, which adds error rather than detail.

Our instructor pointed out that item 17's problem was a concept-test problem rather than a wording problem: a description of a new offer should lead with what is different from the current one. Otherwise respondents rate what they already have.

Which Repairs Reached the Next Version

Item 9 now asks two questions: "Was a heat pump already in the home when you moved in?" and then, "About how old is your oldest heat pump?" with ranges of under two years, two to five years, more than five years, and not sure.

Item 10 now asks: "During a typical winter month, which best describes how your home is heated?" with options for mostly the heat pump, mostly propane, about equally, and other combinations.

Item 17 now opens with the difference: "This plan would lower the fixed annual fee for your tank and monitoring and guarantee delivery during cold snaps." The pricing sentence follows.

I did not adopt one suggestion. A classmate proposed dropping the plan concept entirely and asking only about current behavior, arguing that hypothetical questions are unreliable. I agree they are weak evidence of what people will do, but the decision needs some measure of interest, and the research proposal already plans a pilot to test behavior. Dillman et al. (2014) suggest that concept questions are useful when their limits are acknowledged, which the report will do.

What I Learned

I had treated pretesting as a final check for typos. The seminar showed me it is the first time the questionnaire meets the people it is for, and that the items written most carefully for the decision maker can be the hardest for the respondent. Five readers found three problems that 653 respondents would otherwise have answered badly, and the cost was an evening of reading aloud at a kitchen table with a notepad.

Next Time

I will pretest earlier, before I become attached to any item's wording, and I will include at least one participant whose situation differs from the one I have in mind, such as someone who bought a home with equipment already installed.

References

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.

Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.