| Course | MT438 Analytics in the Digital Supply Chain |
|---|---|
| Unit | Unit 9 |
| Paper type | Predictive model evaluation |
| Length | About 1,160 words, 7 pages double-spaced |
| Format | APA 7 student paper |
| School | Purdue University Global |
| Program | BS in Business Administration |
| Updated | October 2026 |
Free sample paper for MT438 Unit 9
Predicting Lateness From Lateness: An MT438 Unit 9 Evaluation of a Late-Load Prediction Model
[Student Name]
Purdue University Global
MT438: Analytics in the Digital Supply Chain
Unit 9 Predictive Model Evaluation
[Instructor Name]
[Date]
Oakhurst Lawn and Garden, its carriers, the model and every figure are composite; the sources cited are real.
What the Model Is For
Oakhurst Lawn and Garden's analyst built the model to help dispatchers decide which outbound loads to the home-improvement account to watch. Each morning it scores every load leaving that day with a probability of missing the retailer's delivery window. Dispatchers would then call carriers on the highest-scoring loads and, where a delay looks likely, rebook the appointment before the 24-hour penalty point. That purpose sets the standard for judging the model: it must work with information available at dispatch, and its alerts must be accurate enough that dispatchers keep acting on them.
Training Data and Inputs
The model is a gradient-boosted classifier trained on 24,618 loads shipped from 2023 through 2025, of which 2,683, 10.9 percent, missed their window. It uses eleven inputs: carrier, origin plant, destination distribution center, scheduled pickup time, miles, day of week, month, weather alerts on the route, load weight, the carrier's late rate over the prior ninety days and the number of check calls logged for the load.
Known at Dispatch, or Later?
Each input was dated by when it becomes known. Ten of the eleven are known at dispatch. The eleventh is not. Check calls are logged by dispatchers when they phone a carrier about a load, and most check calls happen after a load is already running behind. A model that uses check calls is learning to predict lateness partly from evidence that lateness has begun. Kaufman et al. (2012) call this leakage: information in the training data that would not be available when the prediction is actually made, which makes a model look better in testing than it can be in use.
A load gets check calls because it is late, so a model that counts them is partly reading the answer.
Three Layers of Performance
The analyst reported an area under the curve of 0.84 from five-fold cross-validation on the 2023 to 2025 loads. This evaluation reports three figures, each on data further from the model's training. With check calls included, cross-validated area under the curve is 0.84. With check calls removed and the model retrained, it falls to 0.74. On an untouched holdout of 3,904 loads shipped from February through May 2026, the model without check calls scores 0.69. Area under the curve measures how well a model ranks late loads above on-time ones across every possible threshold (Fawcett, 2006). The last figure is the one a dispatcher would experience.
The Confusion Matrix
The analyst proposed flagging loads with a predicted probability of 0.30 or higher. On the 2026 holdout, 539 loads were late, 13.8 percent. At the 0.30 threshold the model flags 598 loads. Of these, 262 were late and 336 were on time. It missed 277 late loads and correctly passed 3,029 on-time ones.
In dispatcher terms, precision is 43.8 percent: fewer than half of the loads the model flags actually run late, so a dispatcher would make roughly one unnecessary call for every useful one. Recall is 48.6 percent: the model catches just under half of the late loads, and the other 277 arrive late without warning.
Beaten by a Constant
The model's overall accuracy on 2026 loads is 84.3 percent. A rule that simply calls every load on time scores 86.2 percent, because 86.2 percent of loads were on time. That comparison does not mean the model is useless; it means accuracy is the wrong yardstick when the event of interest is rare. Saito and Rehmsmeier (2015) show that for imbalanced outcomes like this one, precision and recall describe a classifier's usefulness far better than accuracy or even area under the curve, and this evaluation leads with them for that reason.
Calibration
A dispatcher who sees "30 percent" should be able to read it as a three-in-ten chance. On 2026 loads, the model's predicted probabilities average 10.2 percent, while the actual late rate is 13.8 percent. The model is systematically too optimistic, so its scores understate risk, and a threshold chosen on training data flags fewer loads than it should.
Where the Training Data Stops
Two changes in 2026 explain much of the gap between training and holdout performance. Moreno-Torres et al. (2012) call this kind of problem dataset shift: the conditions that produced the training data no longer hold.
First, a regional carrier added in January hauls 742 of the 3,904 holdout loads, 19 percent, and runs late on 159 of them, 21.4 percent. Because it has no history, its ninety-day late rate input is blank and the model treats it as an average carrier.
Second, the retailer narrowed its delivery window from four hours to two in February. About 118 of the 539 late loads in 2026, 22 percent, arrived inside the old window but outside the new one. Lateness itself changed definition, and the model was trained on the old one.
Findings and Recommendation
The model should not be used as built. It should be approved for limited use after four changes. Remove the check-call input permanently. Retrain on loads labeled under the two-hour window, recomputing 2023 to 2025 labels against two hours where arrival times allow. Give new carriers a provisional late rate drawn from their first thirty loads, and flag their loads for manual review until then. Recalibrate predicted probabilities on 2026 data so that scores match observed rates.
After retraining, the model should run alongside dispatchers for eight weeks without driving any calls, so that its precision and recall can be measured on live loads before anyone relies on it. If precision stays below 40 percent, the alerts will cost more dispatcher time than they save, and the threshold should be raised before use.
Choosing the Threshold
The 0.30 threshold was chosen by the analyst because it maximized a balanced score on the training data. A dispatcher's costs are not balanced. A missed late load costs a $1,000 fine when rebooking would have worked, while an unnecessary call costs perhaps five minutes of a dispatcher's time. On those costs, a lower threshold that flags more loads would pay for itself even at lower precision, provided the dispatch desk can absorb the calls. The desk handles about forty loads to this account on a busy spring day and has said it can make about ten extra calls. The threshold should therefore be set after retraining so that it flags roughly ten loads on a typical peak day, and its precision and recall at that setting should be the figures reported to management. Choosing the threshold from the desk's capacity and the cost of each kind of error, rather than from a statistical default, ties the model to the decision it exists to support.
References
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861-874. https://doi.org/10.1016/j.patrec.2005.10.010
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), 1-21. https://doi.org/10.1145/2382577.2382579
Moreno-Torres, J. G., Raeder, T., Alaiz-RodrÃguez, R., Chawla, N. V., & Herrera, F. (2012). A unifying view on dataset shift in classification. Pattern Recognition, 45(1), 521-530. https://doi.org/10.1016/j.patcog.2011.06.019
Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), Article e0118432. https://doi.org/10.1371/journal.pone.0118432
MT438 Unit 9 instructions, in plain terms
The MT438 Unit 9 assignment presents a predictive model, or asks students to choose one, and calls for an evaluation of whether it can be trusted for a decision. Start with what the model is for, since purpose sets the standard. List the inputs with the moment each becomes known, because anything recorded after the prediction is needed can leak the answer. Report performance on data progressively further from training, ideally on a period the model never saw. Translate the confusion matrix into the terms of the people who will use it. Compare the model with a simple rule, especially when the event is rare. Check whether predicted probabilities match observed rates. Ask what has changed since the training data ended. Finish with a decision: use, limited use after changes or no use, with the conditions spelled out. A short section on threshold choice often strengthens the recommendation.
How this MT438 Unit 9 example is built
The Oakhurst evaluation begins with the dispatchers' decision and what it requires. Training data and eleven inputs come next, then a section dating each input, which exposes the check-call leak; Kaufman and colleagues define it, and a highlighted line puts it in plain words. Three layers of performance follow, 0.84, 0.74 and 0.69, with Fawcett on what the curve measures. The confusion matrix is read in dispatcher terms: about one useful call per wasted one and 277 late loads missed. A section shows accuracy losing to a constant rule, citing Saito and Rehmsmeier. Calibration finds predictions averaging 10.2 percent against 13.8 actual. Moreno-Torres and colleagues frame two shifts, the new carrier and the narrower window. Findings approve limited use after four changes and a silent trial, and a final section sets the threshold from the dispatch desk's capacity.
Reading the MT438 Unit 9 grading rubric
Purpose-driven judgment earns most of the points in an MT438 model evaluation. Graders look first for whether the student checked when each input becomes available; a leak found and explained is often the single most rewarded finding. Performance reported only on training data, or by cross-validation alone, tends to lose marks, while results on a later holdout show the student knows where a model is actually used. Precision and recall translated into operational terms are credited above bare statistics. Comparing against a simple baseline shows the student understands why accuracy misleads with rare events. Calibration and dataset shift are advanced points that set strong papers apart. A recommendation with specific conditions is rewarded over a general verdict. Figures should be consistent across sections, since graders often recompute precision or recall. Purdue Global expects machine learning and analytics sources cited in APA format.
MT438 Unit 9 help from the desk
Many Unit 9 evaluations simply repeat the model builder's reported accuracy and call it good. Another common problem is never asking when each input is recorded, so a leaking field goes unnoticed. Students often report accuracy for a rare event without comparing it to the rule of always predicting the common outcome. Confusion matrices appear without interpretation, leaving the reader to work out what precision means for the people using the model. Changes since training, like a new supplier or a redefined outcome, are frequently ignored. Recommendations tend to be all or nothing. Before submitting, compute what a constant prediction would score on your data and place it beside the model's figure. If the model does not beat it on the measure that matters for the decision, your evaluation needs to say why it might still be useful, or why it is not.
Get an MT438 Unit 9 example written to your instructions
Pass along the model description, output or results that come with the Unit 9 case, plus the prompt and rubric. We produce a composite evaluation that dates every input, tests on data outside training, translates the error rates into operational terms and states where the training data ends. Opening custom sample at no cost, usually within 24-48h. The paper above is an original model document written by our desk, not a submitted student paper and not an official Purdue University Global document.
More MT438 papers and BS in Business Administration samples
- MT438 Unit 1: Supply Chain Data Inventory Post
- MT438 Unit 2: Order Data Quality Audit
- MT438 Unit 3: Supply Chain Metrics Exercise
- MT438 Unit 4: Operations Dashboard Critique
- MT438 Unit 5: Grass Seed Forecast Accuracy
- MT438 Unit 6: Production Cut Seminar Reflection
- MT438 Unit 7: Shipment Tracking Technology Brief
- MT438 Unit 8: Fill Rate Root Cause Analysis
- MT438 Unit 10: Analytics Recommendation Memo
- MT437 Unit 9: Automation Business Case
- MT436 Unit 9: Supplier Scorecard
- MT435 Unit 9: Lean Waste Review
- MT434 Unit 9: Reverse Logistics Plan
MT438 Unit 9 questions, answered
What is data leakage in a predictive model?
Leakage happens when a model trains on information that would not be available at the moment the prediction is needed, often a field recorded after the outcome began. The model looks impressive in testing because it is partly reading the answer. Checking when each input becomes known, relative to the decision the model supports, is the most reliable way to find it.
Why is accuracy a poor measure for rare events?
When only a small share of cases are positive, a model that always predicts the common outcome scores high accuracy while catching nothing. Precision, recall and area under the curve describe how well the model finds the rare cases. For a course evaluation, compare the model against that always-negative baseline to show the reader why accuracy alone misleads.
How do I evaluate a model if I cannot rerun it?
Work from what the case supplies: its inputs, reported results and training period. Check whether any input could leak, whether results come from data outside training, and whether conditions have changed since the training period ended. Many cases are built so that the flaws are visible in the description, and naming them clearly earns credit without any software.
Where can I find a free MT438 Unit 9 sample paper?
Read the whole MT438 Unit 9 evaluation here; it tests a composite late-load prediction model for leakage, holdout performance, calibration and data shift.
Why does the MT438 Unit 9 sample remove check calls from the model?
Dispatchers log check calls mostly after a load is already delayed, so the input carries evidence of lateness that would not exist when the morning prediction is made.