GF582 Unit 7 Scorecard Default Rate Hypothesis Test Example

Does the New Scorecard Lower Defaults? A Unit 7 Hypothesis Test Report With Its Assumptions Beside It

[Student Name]

Purdue University Global

GF582: Statistical Methods for Decision Making

Unit 7 Hypothesis Test Report

[Instructor Name]

[Date]

The lender, the vendor and every count are composite and illustrative; the articles and textbook cited are real.

Claim Box

The vendor's claim. Loans approved using its scorecard default at a lower rate than loans approved under the lender's previous rules.

Null hypothesis. The one-year default rate under the new model equals the rate under the old model.

Alternative hypothesis. The default rate under the new model is lower.

Direction. One-sided, fixed before the data were examined, because the claim itself is directional and the lender would adopt the model only if it lowered defaults.

Significance level. 5 percent.

Test Box

Old model. 127 defaults among 2,400 loans approved in the year before the switch: 5.29 percent.

New model. 90 defaults among 2,150 loans approved in the year after: 4.19 percent.

Difference. 1.11 percentage points.

Pooled default rate under the null. 217 defaults among 4,550 loans: 4.77 percent.

Standard error of the difference. Taking 0.0477 times 0.9523, multiplying by the sum of 1 over 2,400 and 1 over 2,150, and then the root gives about 0.633 percentage points.

Test statistic. 1.11 divided by 0.633, about 1.75.

P-value, one-sided. About 0.040.

Decision. Reject the null at the 5 percent level. The data are consistent with the vendor's claim.

Assumptions Table

Independence of loans. Each loan's outcome is assumed independent of the others. Partly met: loans to businesses in the same industry or region tend to default together in a downturn, which would make the true standard error larger than the formula assumes.

Large samples. Both groups have well over ten defaults and ten non-defaults, so the normal approximation to the difference in proportions is reasonable. Met.

Comparable groups. The test assumes the two cohorts differ only in the scorecard used. Questionable: the old-model loans were made in a year when equipment prices and interest rates were higher, and the new-model loans in a calmer year. Any change in the economy between the two years affects defaults independently of the model.

Same definition of default. Both cohorts use 90 days past due within twelve months of funding. Met.

Complete follow-up. Every loan in both cohorts has been observed for a full twelve months. Met.

The test says the drop in defaults is unlikely to be chance, but it cannot say the scorecard caused it, because the two cohorts were approved in different years and the economy changed between them, and that before-and-after design is the weakest link in the evidence.

A Second Look With Two-Sided Testing

Had the lender chosen a two-sided test, asking whether the model changed defaults in either direction, the p-value would be about 0.080, and the result would not be significant at 5 percent. That is why the direction must be fixed before the data are seen. Choosing it afterward, to reach significance, would make the 5 percent level meaningless (Anderson et al., 2020).

How Big Is the Effect?

Statistical significance does not measure importance, a point the American Statistical Association has stressed in its guidance on p-values (Wasserstein & Lazar, 2016). Using the unpooled standard error of about 0.629 points, the two-sided 95 percent range for the gap runs from about negative 0.13 to 2.34 points. Because that range reaches below zero, the improvement could be negligible or more than two points. The next unit's decision brief turns that range into dollars.

Power of the Test

With these sample sizes, the test had only about an even chance of detecting a true one-point reduction in defaults at the 5 percent level. A result that barely clears significance with modest power should be read cautiously: a second pilot of the same size could easily fail to reach significance even if the model works.

What the Lender Can Say to the Vendor

The lender can tell the vendor that its pilot results are consistent with the claim but not conclusive, and that renewal will depend on a cleaner comparison. A vendor confident in its model should welcome that test.

What Would Strengthen the Evidence

A randomized comparison, scoring half of new applications with the old rules and half with the new model over the same months, would remove the timing problem. Research on credit scoring stresses that model comparisons are most reliable when both models are applied to the same population at the same time (Hand & Henley, 1997). A shorter alternative is to apply both models to the same historical applications and compare how each would have ranked the loans that later defaulted.

Reporting the Result

The credit committee should see the p-value, the interval and the design caveat together on one page, so that no single number is read alone.

Conclusion

The data support the vendor's claim at the 5 percent level on a one-sided test, with a p-value of about 0.040. The evidence is real but modest, and the before-and-after design leaves open the possibility that a better economy, not the model, produced part of the improvement.

References

Anderson, D. R., Sweeney, D. J., Williams, T. A., Camm, J. D., & Cochran, J. J. (2020). Statistics for business and economics (14th ed.). Cengage.

Hand, D. J., & Henley, W. E. (1997). Statistical classification methods in consumer credit scoring: A review. Journal of the Royal Statistical Society: Series A (Statistics in Society), 160(3), 523-541. https://doi.org/10.1111/j.1467-985X.1997.00078.x

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108