Module 4 Forecast Uncertainty and Evaluation.

Appendix

Notation
Actual series: yt , forecast for period t: yt
Pf
Pf
Average of yt , y = f1 t=1 yt , average of forecast series yt , y = f1 t=1 yt ,
q P
f
Standard error of yt : sy = f1 t=1 (yt y)2 , standard error of yt : sy =
q P
f
1
yt y)2
t=1 (
f
Forecast error: F Et = yt yt
Pf
Bias: BIAS = f1 t=1 F Et =

1
f

Standard forecast error SE =

q P
f
1

Mean squared error M SE =

Pf

yt
t=1 (

t=1 (F Et

1
f

Pf

t=1

Root mean squared error RM SE =

Mean absolute error M AE =

1
f

yt ) =

Pf

1
f

Pf

t=1

yt

1
f

BIAS)2

F Et2

t=1

M SE =

q P
f
1

t=1

F Et2

|F Et |

Mean absolute percentage error M AP E =

1
f

Pf

t=1

| FyEt t |

Proof that M SE = SE 2 + BIAS 2

SE 2

f
1X
(F Et BIAS)2
f t=1

f
1X
(F Et2 2BIAS F Et + BIAS 2 )
f t=1

f
f
f
1X
1X
1X
F Et2
2BIAS F Et +
BIAS 2
f t=1
f t=1
f t=1

Pf

t=1

yt

= M SE 2BIAS

f
1X
F Et + BIAS 2
f t=1

= M SE BIAS 2

Proof that M SE = BIAS 2 + (sy sy)2 + 2(1 r)sy sy

Since we already know that M SE = SE 2 + BIAS 2 , we only need to show that
SE 2 = (sy sy)2 + 2(1 r)sy sy
(sy sy)2 + 2(1 r)sy sy

=
=

s2y 2sy sy + s2y + 2sy sy rsy sy

s2y + s2y 2rsy sy

By definition of the correlation coefficient r =

Then,
s2y + s2y rsy sy

Cov(y
yt )
t ,
sy sy

1
f

Pf
t=1

(yt
y )(
yt y
)
sy sy

=
=

f
f
f
1X
1X
1X
(yt y)2 +
(
yt y)2 2
(yt y)(
yt y) =
f t=1
f t=1
f t=1

f
o
1 Xn
(yt y)2 + (
yt y)2 2(yt y)(
yt y)
f t=1

f
o2
1 Xn
(yt y) (
yt y)
f t=1

f
o2
1 Xn
(
y y) (
yt yt )
f t=1

f
1X
(BIAS F Et )2 = SE 2
f t=1

Therefore bias proportion plus variance proportion plus covariance proportion sum uip to unity.

Example of incorrect forecast ranking based on the

U1 statistic
This section provides an example of two forecasts, where the first forecast is
clearly superior to the second, but the U1 statistic based on the first forecast is

larger than for the second. In other words, if one makes his judgement solely
based on U1 statistic, she will incorrectly prefer the inferior forecast.
Suppose we know that the actual series yt are distributed with zero mean
and variance 2 . Consider the following forecasts for this test period:
1. Set yt = 0 for t = 1..f . Then
q P
f
1
U1

q P
f
1

yt2 + f1 t=1 yt2

q P
f
1
2
t=1 (yt 0)
f
q P
q P
=
f
f
1
1
2
2
t=1 0 +
t=1 yt
f
f
q P
f
1
2
t=1 yt
f
q P
=1
f
1
2
t=1 yt
f
f

yt )2
q P

t=1 (yt

t=1

2. Independently draw yt from N (0, 2 ) every t = 1..f . This forecast is a

random variable. Then by law of large numbers and using that yt is
independent of yt
q P
f
1
t )2
t=1 (yt y
f
q P
U1 = q P
f
f
1
t2 + f1 t=1 yt2
t=1 y
f
q P
Pf
Pf
f
1
1
2
t yt + f1 t=1 yt2
t=1 yt 2 f
t=1 y
f
q P
q P
=
f
f
1
1
2+
2
y

t=1 t
t=1 yt
f
f

2 2 0 + 2
2

=
<1

2
2
2
+
Clearly, the second forecast is worse
the first. The root
squared error
qthan
q mean
Pf
Pf
1
1
2
of the first forecast is RM SE1 = f t=1 (yt yt ) = f t=1 yt 2 . For
q P

f
the second forecast RM SE2 = f1 t=1 (yt yt )2 2 > RM SE1 (see the
numerator of the second U statistic). At the same time the first forecast recieves
a worse ranking based on the U1 statistic.