Beyond the Notebook: Why Forecasting ML Stalls Before It Reaches the Planner

Meet Arjun, the data scientist.

Arjun optimised what he was given: weighted error on a test set. The nine points were real. But most of them came from slow-moving tail items, where a better forecast changes no order, and the model had been trained on monthly category data when Leila plans weekly, by SKU and location. It was accurate about the wrong thing at the wrong grain.

There was a quieter flaw. The training history was sales, not demand. Wherever the company had run out of stock, the sales line went flat, and the model dutifully learned to under-forecast exactly the items the business runs out of most often. A forecast that reproduces your past stockouts is not a forecast. It is a memory.

Meet Naveen, the supply chain director.

Naveen asks the question that should have come before the build: if this model were right, what would we do differently? Nobody could say. So the team went back and did what should come first. They set the model against a seasonal naive forecast and a simple exponential smoothing baseline, over rolling windows so that no future data leaked into the past. The model won, but by far less than the notebook claimed, and not everywhere.

Then they pushed both forecasts through an inventory simulation. That is where the value appeared: concentrated in a small set of high-value, volatile items, as lower safety stock and fewer expedited shipments. Finally they tested the model where it would be tested in real life, in a promotion and a supplier outage. The model that looked best in calm months was not the one that held up in a shock.

Only now could Naveen say what the model was worth, in inventory and service, in terms Finance would sign.

Meet Sunita, the model owner.

Eight months in, a competitor cuts its price and a new sales channel opens. Nothing crashes and no error appears. The model simply gets steadily, silently worse, because the market it learned no longer exists. Unlike ordinary software, a model rots.

Sunita’s job did not exist at go-live. Now it does. She tracks accuracy against the baseline every month, owns the retraining triggers, and can tell a planner in plain terms why the model moved a number. That last duty matters most. When the model lifted one region by 40 percent, it also showed which drivers did it: price, weather, a distributor’s stock. Explanations of that kind, whether feature importance or SHAP values, are not a nicety. A planner will not defend a number she cannot explain to her own sales director.

Meet Leila, the demand planner.

Back to Leila, and to why she reached for her spreadsheet. Early in the rollout the model gave her three poor forecasts in a row, and nobody could say why. After the third, she stopped looking. That is not stubbornness. Her standing rests on knowing her numbers cold, and a black box that is sometimes wrong and cannot explain itself threatens it.

What changed was how the project treated her. She now sits in the design sessions and challenges which drivers go into the model. Her role has been rewritten, from preparing the number to interrogating it, managing the exceptions and deciding when to override. Her overrides are tracked, not policed.

The scorecard changed too. It stopped reporting accuracy alone and began reporting forecast value added: does each step, the model and then Leila’s adjustments, beat a simple baseline? The metric, popularised by the forecasting practitioner Michael Gilliland, ends the argument between human and machine, because it asks only whether a step helped. Where Leila’s overrides add value, they stay. Where they do not, the data says so, kindly.