MT362 Unit 7 AI Model Bias Review Example

Four Stand-Ins for Age: An MT362 Unit 7 Bias Review of a Churn Model That Missed Older Cancelers

[Student Name]

Purdue University Global

MT362: Artificial Intelligence Applications for the Marketing Professional

Unit 7 Bias Review

[Instructor Name]

[Date]

The chain, model and results are composite; the frameworks, regulation and research cited are real.

Defining the Harm First

A bias review measures nothing until it says what unequal treatment would look like. The obvious test, comparing how often the model flags each age group, is the wrong one here. Older subscribers cancel less often than younger ones, so a model that flags them less often may simply be accurate. The harm this review cares about is different: a loyal 70-year-old subscriber who is about to leave and receives no save offer, because the model did not see the risk. The review therefore measures detection among actual cancelers, the share of subscribers who did cancel within 90 days whom the model had flagged beforehand. Schwartz et al. (2022) stress that bias must be defined in the context of a system's use, and in this context a missed canceler is the outcome the chain never intended.

Why the Audit Needs the Excluded Field

Birth year was removed from the model by design, as the Unit 4 review required. But removing it made age invisible to the model and to anyone checking the model. Fairness cannot be tested on an attribute nobody can see. The review therefore joins birth years back for audit use only, under a written restriction that keeps them out of every production system. About 27,100 subscribers, a little over a third, gave a birth year before the field was dropped in 2021, and their backtest scores form the audit sample. Because those who gave birth years may differ from those who did not, results are reported for this sample only.

Detection by Age Band

Among subscribers in the audit sample who canceled within 90 days, the model had flagged 61 percent of those under 45, 54 percent of those 45 to 64 and 38 percent of those 65 and over. The ratio between the oldest and youngest bands is 0.62.

The review borrows the four-fifths ratio as a screening threshold: a ratio below 0.8 calls for investigation. The figure comes from federal hiring rules, which use it to flag possible adverse impact in hiring (Uniform Guidelines on Employee Selection Procedures, 1978). It is employment law's rule of thumb, not a marketing rule, and the review labels it that way. At 0.62, the model fails the screen clearly.

Removing age from the model did not remove age from the model; four other fields carried it back in.

Four Stand-Ins for Age

Proxies are traced one at a time, each judged on two questions: how closely does the feature track age, and does it predict cancellation for reasons other than age?

Tenure over ten years correlates with age at 0.41. It also predicts cancellation on its own merits, since long-standing subscribers have built a habit that resists small annoyances. It stays.

Email addresses at legacy internet-provider domains correlate with age at 0.37 and add almost nothing to prediction once tenure is included. They go.

Paper catalog opt-in correlates with age at 0.44, the strongest of the four, and also adds almost nothing once tenure is included. It goes.

Phone orders correlate with age at 0.33 and predict cancellation in their own right, because phone orderers are more exposed to delivery problems that the app would have flagged. They stay, with a note to retest.

Lambrecht and Tucker (2019) showed that an ad-delivery algorithm produced unequal outcomes by gender without any intent to discriminate, through the economics of the system itself, and this model's problem has the same character: no one chose to miss older cancelers.

Two Features Out, One Point Lost

The backtest was rerun without the catalog flag and the email-domain feature, with the flagging threshold reset so that the model flags the same total number of subscribers. Detection among cancelers 65 and over rose to 51 percent, while detection among those under 45 fell to 58 percent, a ratio of 0.88. The middle band moved from 54 to 55 percent. Overall detection fell from 55 to 54 percent. The fix costs one point of overall detection, which at roughly 4,200 cancellations a month means about 42 fewer subscribers flagged correctly, against a model that no longer systematically misses its oldest customers.

Fixes Considered and Rejected

Two other fixes were considered before the proxies were removed, and the review explains why each was set aside. The first was a separate flagging threshold for older subscribers, lowering the score needed to flag anyone 65 or over until detection matched. That would close the gap on paper, but it would require using birth year in production, which the Unit 4 review ruled out, and it would apply only to the third of subscribers whose birth year is known. The second was retraining the model with older cancelers weighted more heavily. That might work, but it would depend on the vendor's willingness to change its training process, and the chain cannot inspect that process directly. Removing two features the chain can see, and measuring the effect with a test it can repeat, was the fix within the chain's own control.

What the Fix Does Not Settle

The review does not claim the model is now fair in every sense. Detection still differs by seven points between the youngest and oldest bands, and the ratio of 0.88 passes a screen borrowed from another field rather than a standard written for retention marketing. Flag rates also still differ, because older subscribers cancel less, and the review accepts that difference as accurate. What the fix does settle is the specific harm the review defined: loyal older subscribers who were about to leave are now much less likely to go unnoticed.

Frameworks, With Dates

The review follows two federal frameworks, named with their dates so readers can check them: NIST Special Publication 1270, published in March 2022, on identifying and managing bias in artificial intelligence (Schwartz et al., 2022), and the AI Risk Management Framework 1.0, published in January 2023 (Tabassi, 2023). The framework's emphasis on measuring and monitoring risks over time is the reason the audit repeats.

A Test That Repeats

The review asks for the same audit each quarter and whenever the vendor retrains the model, owned by the analytics lead. A ratio below 0.8 in any audit pauses model-chosen offers until the cause is found. The age-band ratio also becomes a guardrail in the Unit 8 pilot.

Limits

The audit sample is the third of subscribers who gave a birth year before 2021, and they may not represent the rest. The correlations measure association, not cause. And the review addresses only age; other attributes, such as disability, cannot be tested with the data the chain holds.

References

Lambrecht, A., & Tucker, C. (2019). Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management Science, 65(7), 2966-2981. https://doi.org/10.1287/mnsc.2018.3093

Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022). Towards a standard for identifying and managing bias in artificial intelligence (NIST Special Publication 1270). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.1270

Tabassi, E. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1

Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. pt. 1607 (1978).