MT362 Unit 8 AI Pilot Test Design Example

Decide Before You Design: An MT362 Unit 8 Pilot Test Design for a Churn Model and a Simpler Rule

[Student Name]

Purdue University Global

MT362: Artificial Intelligence Applications for the Marketing Professional

Unit 8 Pilot Test Design

[Instructor Name]

[Date]

The chain, population and thresholds are composite; the sizing figures were computed with the stated script, and the sources cited are real.

What Each Result Would Decide

The decision comes before the design. If neither the rule arm nor the model arm beats control at the adjusted significance level, Fernbrook keeps its current cancel-page offer and drops the vendor. If only the rule arm beats control, the rule rolls out without a model. The model survives only by beating control and producing enough retained margin to pay for its own discounts and its fee. Writing these outcomes down before enrollment prevents a disappointing result from being reinterpreted afterward. Lemmens and Gupta (2020) argue that a retention program should be judged by the profit it adds after the cost of its incentives, so the model's test is not whether it lowers cancellations but whether it lowers them enough to pay for its discounts and fee.

The Population

Of about 71,400 active Repeat Delivery subscribers, 58,200 are eligible after removing those who opted out of all marketing email, subscribers enrolled for less than sixty days and staff accounts. Text offers within any arm go only to the roughly 41 percent with written consent, as the Unit 4 review required. Anyone who cancels citing a pet's death during the pilot leaves the analysis entirely rather than being counted as a failure, since no arm is meant to change that outcome.

Three Arms

Arm A, the control, keeps the current $10 offer on the cancel page. The comparison group keeps present practice rather than nothing, because the question is whether a change beats what the chain already does. Arm B adds the schedule calculator prompt and a skip offer whenever a subscriber edits a schedule. Arm C adds model-timed offers from the Unit 5 ladder on top of everything in arm B. Subscribers are randomized individually, stratified by state, California or Oregon, and by tenure band, so each arm carries the same mix.

Measures

The primary measure is cancellation within 90 days of enrollment. Guardrails are discount cost per subscriber, unsubscribe rate, complaint rate and the age-band detection ratio from the Unit 7 bias review. Kohavi et al. (2020) recommend choosing one primary metric in advance and treating others as guardrails that can stop an experiment but cannot declare a winner, and the design follows that advice.

The Sizing Script

A twelve-line script in the appendix computes arm size from four inputs and prints each input beside the result. The baseline is 16.5 percent of subscribers canceling within 90 days, taken from the past year. The minimum effect worth detecting is two percentage points, the smallest reduction that would cover the vendor's fee at the estimated margin. Power is 80 percent. Because two arms are each compared with control, the 5 percent error rate is split between the two comparisons, a Bonferroni adjustment, giving 0.025 per comparison. The printout shows 6,220 subscribers per arm. For comparison, it also shows that a three-point minimum effect would need only about 2,690 per arm, which the design rejects because a two-point effect is still worth money.

The pilot is sized to detect the smallest effect that would pay for the model, not the effect the vendor promised.

How Value Will Be Computed

The same calculation will be applied to every arm once the pilot ends. Each subscriber kept relative to control is valued at an estimated $120 in margin, a figure finance will confirm before the results are read. Discounts actually redeemed in each arm are subtracted, and for the model arm the vendor's fee is spread across the subscribers it would cover. The estimate of $120 is labeled as an assumption, and the analysis will show how the ranking of arms changes if it is halved. Fixing this method now prevents a favorable way of counting from being chosen after the numbers are known.

Who Runs It

The analytics lead owns randomization and the sizing script, lifecycle marketing owns the offers in each arm, and the director of lifecycle marketing owns the day-45 harm review. The vendor receives scores for arm C only and has no access to outcomes until the pilot closes, so it cannot adjust its model partway through.

Timing From the Calendar

Enrollment runs through February 2027, and follow-up continues through May. The window avoids holiday ordering in December and the spring flea and tick season, which lifts orders every year and would blur the comparison.

The Stopping Rule

The design allows one interim look, at day 45, and only for harm. The model arm halts if complaints exceed 0.5 percent of recipients, if unsubscribes reach double the control rate or if the age-band detection ratio falls below 0.8. Each threshold was agreed with the director of lifecycle marketing before enrollment. The design forbids ending the pilot early because the model arm looks ahead, since early peeks at a winner inflate the chance of a false result. Ascarza (2018) found that targeting retention offers by predicted risk can underperform, which is one more reason not to stop early on a promising start.

What Ninety Days Cannot Show

The pilot cannot show whether saved subscribers stay for a year, whether discounts change what customers expect to pay later, or whether results hold outside California and Oregon. Each limitation becomes a question for the adoption proposal in Unit 10.

References

Ascarza, E. (2018). Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80-98. https://doi.org/10.1509/jmr.16.0163

Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.

Lemmens, A., & Gupta, S. (2020). Managing churn to maximize profits. Marketing Science, 39(5), 956-973. https://doi.org/10.1287/mksc.2020.1229