Cleaning Historical Demand Data Before It Reaches Your Forecasting Model

163 Views

Demand planning teams invest heavily in forecasting engines, from statistical baselines to machine learning. However, far less attention goes to the data those engines consume.

Yet forecast accuracy is largely determined upstream, in the quality of the historical demand series fed into the model. So, a capable algorithm trained on unclean history will reproduce the distortions in that history with more confidence, not less.

Industry research has long argued that data preparation is where much of the achievable accuracy is won or lost. Thus, before any model runs, three routine checks — a transparent baseline, outlier treatment, and driver screening — remove the most common sources of error.

Start with a baseline you can explain

Before comparing candidate models, you can establish a reference forecast simple enough to audit. A weighted average is the usual choice: recent periods carry more weight than older ones, reflecting the fact that last quarter usually says more about next quarter than the same period two years ago.

The calculation is carried out using a weighted average of past demand, with weights assigned by recency. This baseline serves two purposes. Firstly, it gives planners a sanity check, since any advanced model should beat it. If it does not, the added complexity is not paying for itself. It also exposes structural problems in the series. Then, if a plain weighted average swings sharply from period to period, the underlying data is noisier than a more elaborate model will admit.

This is the logic behind forecast-value-added analysis: every step in the process, including the model itself, should be measured against the naive baseline it is meant to improve upon.

Treat spikes before they train the model

Historical demand is rarely clean. As is well known, promotions, one-off bulk orders, stockouts, and data-entry errors all leave marks that do not represent underlying demand. Left in place, they pull the average up or down and teach the model to expect events that will not recur on schedule.

The mean is sensitive to these points. This means that a single large order can lift a monthly average well above the level most periods actually saw. So, applying a median provides a more robust approach. Since it reports the middle value rather than the arithmetic center, it is far less affected by extreme observations, making it a useful reference for the typical demand an SKU sees in a normal period.

A practical approach is to compare the mean and median for each series. Where they diverge, the gap flags a skewed distribution worth investigating, often a promotional spike or an outlier that should be tagged, adjusted, or modeled separately rather than absorbed silently into the baseline. This preserves the genuine demand signal while stripping out noise that the forecast should not learn.

Screen candidate drivers before adding them

Modern demand models increasingly ingest external variables: price, weather, macroeconomic indicators, or the sales history of a related product. More inputs are not automatically better. A variable with no real relationship to demand adds noise and can degrade accuracy, particularly when history is limited.

correlation check is a fast first filter. Measuring the strength and direction of the relationship between a candidate driver and historical demand shows whether the variable carries a usable signal before it is built into a model. A weak coefficient is grounds for leaving the variable out or investigating further; a strong one warrants a closer look.

Here, two cautions apply. Correlation does not establish causation, since a driver may move with demand because both respond to a third factor, such as seasonality. And a relationship may be lagged, so testing the driver against demand several periods ahead often reveals more

than a same-period comparison. Used as a screening tool rather than proof, correlation keeps model inputs disciplined.

Make cleaning a step, not an afterthought

The value of these checks comes from running them consistently. Research on demand planning maturity places data management and governance ahead of algorithm selection. Therefore, organizations that formalize data preparation tend to see more durable gains in accuracy than those that chase model sophistication alone.

In practice, that means a documented routine before each planning cycle: build the weighted baseline, compare mean and median to isolate outliers, screen new drivers with a correlation pass, and record what was changed and why. The steps are simple enough to run in a spreadsheet and important enough to protect whatever sits downstream, whether that is a statistical package or a machine learning pipeline.

Forecasting technology will keep advancing. The demand history that it depends on will keep arriving imperfectly. Teams that treat data cleaning as a defined, repeatable stage, rather than a preface to the real work, get more from every model they run.