The Demand Planning Process

• Measuring Performance

dp lanning
Global Supply Chain
&
eployment
Business Forecasting 2006
 Measuring Performance
• Successfully measuring and evaluating the absolute performance and impact of the
forecast is complex due to the interactive influence of internal (operational) and
external (customer related) variables
– Example: If the forecast for a given product is 100 units but due to a manufacturing problem,
only 50 are available to sell, the customer must either wait for product to become available or
seek an acceptable alternate. If the customer waits, the forecast will be undersold. If the
customer purchases an alternate from the portfolio, the error is doubled: One product is
undersold, another is oversold. In either case, the resultant forecast error is not due to the
predictive quality of the forecast but rather, a condition outside the forecasting process
producing the error
• The Forecast will ALWAYS be wrong
• The Forecast will ALWAYS be changing
• Use performance indicators to understand the contributing factors driving the
forecast error to minimize the intensity of the error in the future

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

The Need for Different Metrics

• How accurate is the forecast?
– The answer to that question is dependent on who is asking the question

Sales
“Were is my product
- I forecasted it?” Planning
“The forecast stinks,
what’s going on?” Senior
Management
Marketing
“Why is everyone
“What is the
complaining about the
forecast trend?”
Production forecast, what’s going
“The forecast on?”
keeps changing,
Financ what’s going on?”
e
“Are we going to
make our numbers?”

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance

The Need for Different Metrics

• Because the forecast impacts different areas of
the business, a suite of measurements that can
answer performance questions from each of
these perspectives is required
– Base level accuracy metrics
– Comparative forecast analysis
– Forecast stability

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Base Level Accuracy Metrics

• Base level accuracy metrics calculate the predictive
quality of the forecast, providing a trended view of
performance over time. Each metric describes a
particular dynamic of performance and each is best
suited for communicating performance at a discreet
level, although they may have applicability across levels
– Relative accuracy
– Weighted mean absolute percentage error (WMAPE)
– Forecast tolerance
– Forecast bias

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance – Base Level Accuracy Metrics

Relative Accuracy
• Relative accuracy compares the total predicted quantity for a
period against the actual yield, illustrating the relationship
between the two numbers
– Most meaningful at the lowest forecasted level
– To calculate relative accuracy, capture the forecast and the actual orders
for a period and divide the smaller number by the larger

Units
Forecast 1,000 800
Orders 800 1,000
Relative Accuracy 80.0% 80.0%

– The result is an absolute percentage that can be interpreted against a scale

with a range of zero to 100%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance – Base Level Accuracy Metrics

Relative Accuracy
• Relative accuracy can have value at an aggregated level
– Similar products bundled into a single portfolio
• Common applications
• Shared components and physical characteristics
• Uniform cost and pricing structures
• Provide value to the sales, marketing, and financial teams
• The disadvantage in using relative accuracy at an aggregated level is that
its does not account for product mix performance
Portfolio A
Relative
Product Forecast Orders
Accuracy
ATP1001 10,000 9,905 99.1%
ATP1002 100 20 20.0%
ATP1003 200 100 50.0%
ATP1004 50 200 25.0%
ATP1005 10 30 33.3%
Total 10,360 10,255 99.0%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance – Base Level Accuracy Metrics

Weighted Mean Absolute Percentage Error (WMAPE)

• WMAPE is the measurement used for reporting aggregated accuracy results

• WMAPE is the global reported accuracy measurement
• WMAPE solves the problem of using Relative Accuracy for product groups by
accounting for product mix characteristics
– To calculate WMAPE determine the absolute error for each product within the portfolio,
convert each error value to a percentage of the total forecast, and sum the percentages to
determine the WMAPE for the group. To convert the error into accuracy, subtract it from one
– The effect of considering the error contribution of each product on the accuracy results is
apparent when we compare the relative accuracy results to the WMAPE accuracy

Portfolio A
Relative Absolute Percent of
Product Forecast Orders
Accuracy Error Total Forecast
ATP1001 10,000 9,905 99.1% 95 0.9%
ATP1002 100 20 20.0% 80 0.8%
ATP1003 200 100 50.0% 100 1.0%
ATP1004 50 200 25.0% 150 1.4%
ATP1005 10 30 33.3% 20 0.2%
Total 10,360 10,255 99.0% WMAPE 4.3%
Accuracy 95.7%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance – Base Level Accuracy Metrics

Forecast Tolerance
• Forecast tolerance is another approach to measuring
aggregated forecast performance
• Forecast tolerance quantifies the number of products
within a group that meet or exceed a predetermined
forecast accuracy target or range
– Extremely stringent
– Often misused and misunderstood
– Limited in value for improving forecast performance
– Should be used to focus efforts on a specific segment

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance – Base Level Accuracy Metrics

Forecast Tolerance
• To determine the tolerance result for a segment of products, first calculate
the relative accuracy for each product, then determine whether the
relative results meets or exceeds the established threshold, total the sum of
products meeting their targets, and divide the result by the total number
of products within the portfolio
Portfolio A
Forecast Relative Meets
Product Forecast Orders
Target Accuracy Target?
ATP1001 95% 10,000 9,905 99.1% Y
ATP1002 85% 100 20 20.0%
ATP1003 85% 200 100 50.0%
ATP1004 85% 50 200 25.0%
ATP1005 85% 10 30 33.3%
ATP1006 90% 400 375 93.8% Y
ATP1007 90% 250 300 83.3%
ATP1008 95% 1,000 980 98.0% Y
ATP1009 95% 500 480 96.0% Y
ATP1010 90% 700 850 82.4%
Total 4
Total Products 10
Accuracy 40.0%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance – Base Level Accuracy Metrics

Forecast Accuracy
Portfolio A

120%
96.3% 94.6% 97.0% 96.9% 97.4% 96.5%
100% 88.6%
92.7% 92.0% 91.4% 93.8%
86.6%

80% 89.4% 89.9% 88.8% 89.9% 89.0%

84.2% 86.8% 83.4% 83.2%
86.6% 85.2%
81.2%

60% 52.0% 47.0% 51.0% 49.0% 48.0% 47.0%

52.0%
44.0% 44.0% 43.0%
41.0% 39.0%
40%

20%

0%
J an` Feb Mar A pr May J un J ul A ug Sep Oct Nov Dec

Relative A ccuracy WMA PE Tolerance

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance – Base Level Accuracy Metrics

Forecast Bias
• Forecast bias is a continued prejudice in the forecast, the tendency to
always over-forecast or under-forecast
– Over-forecasting (positive forecast bias) contributes to an increased inventory
investment, reducing operational cash flow
– Under-forecasting (negative forecast bias) reduces inventory investment, increases
the likelihood of poor customer service, creates lost profit risk and increases
expenses
– To calculate forecast bias capture the forecast and the actual orders for a period and
subtract the orders from the forecast, then divide the result by the larger of the
forecast or orders. This will generate a positive or negative percentage that can be
plotted on a scale relative to zero

Relative
Product Forecast Orders Bias
Accuracy
ATP1003 200 100 50.0% 50.0%
ATP1004 50 200 25.0% -75.0%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance – Base Level Accuracy Metrics

Forecast Bias
Portfolio A

100.00%
80.00%
60.00%
40.00%
11.4% 7.3% 8.0% 8.6%
20.00% 3.7% 5.4% 3.0% 3.1% 2.6% 3.5%
0.00%
-20.00% -6.2%
-13.4%
-40.00%
-60.00%
-80.00%
-100.00%
J an Feb Mar A pr May J un J ul A ug Sep Oct Nov Dec

Bias

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Comparative Forecast Analysis

• The base level accuracy metrics provide the first layer of information
surrounding forecast performance
• Is the issued forecast the best possible forecast to use?
• To answer this question, a comparison against other potential forecasts is
necessary
– Comparative forecast analysis is an activity that could continue endlessly
– Everyone will have a different opinion on what the best forecast is
– To avoid analysis paralysis and provide a comparison that effectively determines
whether the forecast used is providing the best possible value, a comparison
between the issued forecast and a maximum of two alternate projections is preferred
• The system generated Statistical Forecast
• A Naïve Forecast model (Simple Moving Average)

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Comparative Forecast Analysis

• The Comparative Forecast Analysis requires that three separate forecasts are
created and archived
– The Consensus or Final Forecast
– The system generated Statistical Forecast
– A calculated Naïve Forecast, a simple moving average
• Base level accuracy metrics are generated for each forecast, trended, and compared
against one another
– This provides a tiered comparison of the issued forecast against both a complex and simple
alternative
Comparative Forecast Accuracy
Jan Feb Mar Apr May Jun
Relative 96.3% 94.6% 86.6% 97.0% 88.6% 92.7%
Consensus WMAPE 84.2% 86.8% 83.4% 89.4% 81.2% 83.2%
Forecast Bias 3.7% 5.4% -13.4% 3.0% 11.4% 7.3%
Tolerance 52.0% 47.0% 41.0% 51.0% 44.0% 39.0%
Relative 98.0% 98.3% 87.2% 96.4% 98.4% 97.2%
Statistical WMAPE 74.4% 82.7% 84.0% 86.1% 88.5% 86.0%
Forecast Bias 2.0% 1.7% -12.8% -3.6% 1.6% 2.8%
Tolerance 43.0% 32.0% 43.0% 42.0% 48.0% 36.0%
Relative 92.8% 92.0% 87.6% 99.5% 91.6% 94.5%
Simple
WMAPE 79.3% 84.7% 85.0% 88.8% 85.8% 83.4%
Moving
Bias 7.2% 8.0% -12.4% 0.5% 8.4% 5.5%
Average
Tolerance 48.0% 51.0% 48.0% 48.0% 49.0% 56.0%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
Measuring Performance

Comparative Forecast Analysis

• If either alternative method consistently outperforms
the issued forecast, then the changes made in the
forecasting process are not adding value and the
forecast methodology should be reviewed and adjusted

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Forecast Stability
• Forecast stability is the measure of change occurring in the forecast over a
period
– Monitoring the degree in forecast variability is import to the manufacturing and
distribution functions that use the forecast in running their operations
– High forecast variability increase the operational costs in these functions as they
adjust to meet market demands
• Manufacturing line change-over
• Maintenance
• Over-time expense
• Premium freight and handling costs
– The change in the forecast over the mid-term is also important in gauging the
effectiveness of the financial budgeting process and the ability to accurately reflect
trends and step-level market changes
• Standard cost
• Manufacturing and purchase price variance
• Resource availability
• Capital expenditures

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Forecast Stability
• To review forecast stability in the short-term and mid-term, two measures of
stability are used
– The first measures the change in the forecast across a frozen short-term period matched to the
reaction time defined by manufacturing and purchasing lead-times
Portfolio A
Forecast
Month
Jan
Forecast Frozen Prior Year Nov 8,683,667
Prior Year Dec 8,781,970
Last Change Jan 8,763,799
Average 8,743,145
Stdev 52,305
Variability 0.6%

– The second measures the change in the annual aggregate volume for the current and following
year

Forecast Variance
Month
Total Monthly To Budget
Budget Month Jan 106,205,197
Feb 105,137,078 -1.0% -1.0%
Mar 100,944,036 -4.0% -5.0%
Apr 111,433,472 9.4% 4.7%
May 107,355,025 -3.7% 1.1%
Jun 107,049,865 -0.3% 0.8%

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process
 Measuring Performance

Forecast Stability
• Understanding the stability of the forecast over the short-term and mid-term
provides an awareness of potential increases in operational expenses due to forecast
volatility

dp lanning
Global Supply Chain
&
eployment
Business Forecasting The Demand Planning Process