REGRESSION IN TIME SERIES

Some common trends are

i. No trend, which is modeled as

Yt = β0

t
Figure 1: No Trend

ii. Linear trend, which is modeled as

Yt = β0 + β1t (simple regression)

There is a straight line long-run growth or decline over time.

β1 > 0 β1 < 0

t t
Figure 2: Linear Trend

iii. Quadratic trend, which is modeled as

Yt = β0 + β1t + β1t 2

There is a quadratic long-run growth over time either be

growth at an increasing or decreasing rate.
 t t

t t
Figure 3: Quadratic Trend

LINEAR TREND-SIMPLE REGRESSION ANALYSIS

If there is a relationship between two variables, we can express

the relationship in terms of a mathematical equation using the
population regression line

Yi = β0 + β1 X i + εi

Where β0 and β are parameters whose values are unknown and

1

hence must be estimated. The sample regression line

Yˆi = b0 + b1 X i (1)

Note that εi is unobservable, but it can be estimated by

ei = Yi −Yˆi

Least Square Estimates of the Slope and Intercept

The parameters of sample regression can be estimated by the
method of ordinary least squares (OLS). OLS estimates are
obtained by minimizing the sum of squared errors (SSE):
n n n

SSE = ∑ ei2 = ∑ (Yi − Yˆi ) 2 = ∑[Yi − (b0 + b1 X i )]2

i =1 i =1 i =1

Taking the partial derivatives of SSE with respect to a and b and

setting them equal to zero, we get
n
− 2∑ (Yi − b0 − b1 X i ) = 0
i =1
n

And − 2∑ X i (Yi − b0 − b1 X i ) = 0
i =1

These equations can be written as

n n
b0 n + b1 ∑ X i =∑Yi
i =1 i =1
n n n
b0 ∑ X i + b1 ∑ X =∑ X i Yi i
2
(2)
i =1 i =1 i =1

Finally, by rearrange terms and solving these simultaneous

equations for a and b, we arrive at the following formulas:

n∑i =1 XY − ∑i =1 X ∑i =1 Y
n n n

b1 =
n∑i =1 X 2 −
n
(∑ X ) n
i =1
2

b0 = Y − b1 X

Or Equation (1) can be written in matrix form as

  n ∑ X  b0  ∑ Y 
 2   =  
X ∑∑ X  b1  ∑ X  Y
Estimated Regression Coefficient by Matrix
To obtain the estimated regression coefficients from equation (1)
by matrix methods, we premultiply both sides by the inverse of
X' X

From equation 1

Y =Xb

(X' X) −1 X' Y = (X' X) −1 X' X b

We then find, since (X' X) −1 X' X = I and Ib = b , we get

b = (X' X) −1 X' Y
where
  y1 
y 
 b0   2
. 
1 x1 
1 x 

b=  
 2

. . 
, Y=   and X =
. . 


. 
 b1 
. . 
 
.  
1 x n 


 
 y n 
The Standard Deviation About the Regression Line
The standard error of the regression is defined as

∑ (Yi − Yˆi ) 2 ∑
n n 2
e
i =1 i
se = i =1
=
n−2 n−2

Is used to measure of the average variation of the observed

values of Y around the regression line (the forecast values, Yˆ ).

Testing the Slope Parameter or Coefficient

The slope (b) of the equation is a measure of the change in Yˆi
given a one unit change in X i . A positive slope indicates a
positive relationship between X i and Yˆi . A negative slope
indicates a negative relationship. A slope equal to zero indicates
no relationship. One way to evaluate the regression model is to
test the hypothesis that the population slope ( β) equal to zero
(indicating no linear relationship):

H0 : β = 0 (there is no linear relationship)

H0 : β ≠ 0 (there is linear relationship)

The t-test statistic is computed in the following manner

 b−β
t=
sb
Where
s e2
sb =
∑
n
i =1
(X i − X )2

The null hypothesis would be rejected if the value of test statistics

(t) falls in the critical region by the lower and upper α / 2
percentage points of the t distribution with n – 2 degrees of
freedom. That is, rejected H 0 if

t > tα/ 2 ,v =n −2

The Coefficient of Determination

The coefficient of determination, or R 2 is measure of the common
variance between two sets of variables can be calculated from the
following formula:

R2 =
∑(Yˆ i −Y )2
∑(Y i −Y )2

This value is the proportion of the total variation in Y that is

explained by X in the regression line. It values cannot exceed the
total variation, the maximum value of R 2 is 1 that is 100%. The
correlation coefficient (r) is the square root of the R 2 .

ANALYSIS OF VARIANCE
Analysis of variance (ANOVA) focuses on the ability of a
regression line to explain or account for the variation in Y.
Formally, the total deviation in Y can be regarded as the sum:

(Yi −Y ) = (Yˆi −Y ) + (Yi −Yˆi )

Which holds for all i. It therefore follows that:

∑(Y i −Y ) = ∑(Yˆi −Y ) + ∑(Yi −Yˆi )

Furthermore, it can be shown that this same equality holds when
these deviations are squared. Thus:

∑(Y −Y ) 2 = ∑(Yˆi −Y ) +∑(Yi −Yˆi ) 2

2
i

Total Explained + Unexplained

Variation variation variation

Where variation is defined as the sum of squared deviations. It is

occasionally convenient to rewrite the above equation as

∑(Y −Y ) 2 = b 2 ∑( X i − X ) + ∑(Yi −Yˆi ) 2

2
i

Total Variation + Unexplained

Variation Explained by X variation

The ANOVA table incorporates the F test of the regression

equation’s significance. The F test may be viewed as a test that

H0 : β = 0 (there is no linear relationship)

H0 : β ≠ 0 (there is linear relationship)

The F-test statistic is computed using the ANOVA table.

Table 1: Analysis –of-Variance Table for Linear Regression

Source of Variation Degree of Variance F
Variation Freedom
Explained ∑ ˆ
(Y −Y ) 1
i
2
∑ (Yˆ −Y )
∑(Yˆi −Y )
i
2 2

by s2
regression
Unexplaine ∑(Yi −Yˆi ) 2 n–2
s2 =
∑(Yi − Yˆi ) 2
d variation n −2
(Error)
Total ∑(Yi −Y ) 2 n–1

The null hypothesis would be rejected if the value of test statistics

(F) falls in the critical region by the upper ( α ) percentage points
 of the F distribution with v1 = n − 2 and v2 = n −1 degrees of freedom.
That is, rejected H 0 if
F > f α,v ,v 1 2

CRITERIA FOR MODEL SELECTION

1. RMSE

2. MAE

3. R2 =
∑(Yˆ i −Y )2
∑(Y i −Y )2

4. AIC and SBC

Akaike’s Information Criterion(AIC) and Schwarz’ Bayesian

Criterion (SBC) are given by
AIC p = n ln SSE p − n ln n + 2 p

SBC p = n ln SSE p − n ln n + [ln n] p

p = the number of parameters

The best model is chosen if the model has small value of AIC or
SBC.

TRANSFORMATIONS FOR NONLINEAR RELATION

All the models presented describe deterministic trend that are

linear in the parameters ( β0 , β1 , β2 ) . Sometimes, however useful
models are not linear in parameters. For instance, the model
y t = β 0 ( β1t )ε t

Is not linear in the parameters. To apply the techniques of

estimation and prediction we must transform this model to one
that is linear in the parameters by taking logarithms on both
sides, yields
ln y t = ln β0 + t ln( β1 ) + ln( εt )

If we let Yt = ln yt , a = ln β0 , b = ln β1 and et = ln ε t , the transform version

of the model becomes
Yt = a + b.t + et

Thus, the model is the linear trend.

The other not linear models are

1. yt = β 0 e β t ε t
1

2. y t = β0 ( β1−t )ε t

β1
3. Yt = β o + + εt
t

L
4. Yt =
1 + exp( β0 + β1t + ε t )

Multiple Regression

Multiple regression analysis is one of the most widely used for all
statistical methods. Is used to determine the relationship between
response function (y) and predictor variables (x’s). The regression
model

y t = β0 + β1 X 1 + β2 X 2 +... + βp −1 X p −1 +εt

Is called a first-order model with p-1 predictor variables.

The parameters of the model can be calculated using the least

square estimates

ˆ = ( X T X)
β −1
(X T Y )
 ˆ
ˆ = Xβ
Y
where

 y1 
 
 y2  1 t1

t12 . . . t1n 

.  1 t 2 t 22 . . . t 2n 
. . .
Yˆ =  
.
X= 
.  . .
. .
.
.
.
.
  
1 t n t n . . . t nn 
2 

. 
y 
 n
Residuals
ei = Yi −Yˆi

In matrix notation, we have

ˆ
ˆ = Y − Xβ
e = Y −Y

Regression Coefficient

Tests on Individual Regression Coefficient

H0 : βj = 0
H1 : β j ≠ 0

The test statistic

βj
t0 =
σˆ C jj
 C = (X T X) −1
and σ̂ =
2 SSE
= ∑(Y
i −Yˆi ) 2
(p the number of
n−p n−p
parameters)
Hypothesis null is rejected if the absolute value of the test
statistic t >tα
0 . / 2 , n −p

Test for Significance of Regression

The test for significance of regression is a test to determine

whether there is a linear relationship between the response
variable y and a subset of the predictor or regressor variables
x1 , x 2 ,..., x k . The hypotheses are

H 0 : β1 = β2 = ... = β k = 0
H 1 : at least one β j ≠ 0

The test for significance of regression is usually summarized in an

analysis of variance (ANOVA) table.
Table 2: Analysis –of-Variance Table for Polynomial Regression
Source of Variation Degree of Mean F
Variation Freedom Square
Explained SSR = ∑(Y −Y )
ˆ k i
2
SSR
∑(Yˆi −Y )
2

by k
s2
regression
Unexplaine SSE = ∑(Yi −Yˆi ) 2 n – p s 2 = σ̂ 2 =
SSE
d variation n− p

(Error)
Total SST = ∑(Yi − Y ) 2 n – 1
The null hypothesis would be rejected if the value of test statistics
(F) falls in the critical region by the upper ( α ) percentage points
of the F distribution with v1 = k and v 2 = n − p degrees of freedom.
That is, rejected H 0 if
F > f α, v , v
1 2

Exercise
1. Table 1 shows the quarterly average exchange rate for the
Japanese yen (yen per dollar) over the 19 periods (1985.4-
1990.2).

Year Data Year Data Year Data

1 207.18 8 146.97 15 137.96
2 187.81 9 135.65 16 142.33
3 169.89 10 127.99 17 143.14
4 155.84 11 125.72 18 147.99
5 160.46 12 133.70 19 155.40
6 153.22 13 125.16 20 X
7 142.68 14 128.55 21 X

i. Fit the data using the model

y t = β0 + β1 X 1 + β2 X 2 +... + βp −1 X p −1 +εt

With let X 1 = y t −1 , X 2 = y t −2 ,..., X p = y t −p . Using p = 1, 2, 3

ii. Using the analysis-of-variance (ANOVA) approach and t- test to test

whether or not the slope βk = 0 . Take α = 5% as your level of significance.

iii. Find R 2 , RMSE, MAE, AIC and BSC and interpret the results?

iv. Using the best model, forecast the data from 1990.3-1990.4.

2. Table 2 represents the U.S. apparel employment (in

hundreds of thousands) for period 1969-1990.
 Table 2: The U.S. apparel employment (in hundreds of
thousands)
Yea Data Year Data Year Data
r
196 14.29 1977 13.39 198 11.61
9 13.81 1978 13.59 5 11.41
197 13.58 1979 13.25 198 11.33
0 13.88 1980 12.90 6 11.33
197 14.26 1981 12.71 198 11.30
1 13.67 1982 11.89 7 10.87
197 12.60 1983 11.93 198
2 13.44 1984 12.30 8
197 198
3 9
197 199
4 0
197 199
5 1
197 199
6 2

i. Fit the data using the nonlinear function

(a) yt = β 0 e β t ε t 1

(b) y t = β 0 ( β1t )ε t
β1
(c) yt = β o + + εt
t
1
(d) yt =
1 + exp( β0 + β1t + ε t )
(e) y t = β 0 + β1 t + β 2 t + ... + β k t + ε t
2 k
, for k = 1 , 2, 3, 5.

ii. Using the analysis-of-variance (ANOVA) approach and t- test to test

whether or not the slope βk = 0 . Take α = 5% as your level of significance.

iii. Find R 2 , RMSE, MAE, AIC and BSC and interpret the results?

iv. Using the best model, forecast the U.S. apparel employment data
from 1991-1992.
 Task 2

For regression model, the error terms εt are assumed to be a

random variable with

If the model is appropriate for the data, the observed residuals

et should then reflect the properties assumed for the εt .

Using the your data, test whether the et follows the properties assumed
for the εt .

Example:

Table 1: Production of cultured yellow tail in Japan

Yea 196 196 1967 1970 1973 1976 1979 1982 1984
r 1 4
Ton 190 950 2120 4330 8030 1016 1549 1463 1545
s 0 0 0 0 0 00 00 00 00

(a) Find the equation of the regression line to predict production of cultured
yellow tail in Japan

(b) Using the analysis-of-variance (ANOVA) approach and t- test to test whether
or not the slope β = 0 . Take α = 5% as your level of significance.
(c) Find R 2 , RMSE, MAE, AIC and BSC and interpret the results?

Task 1

The table following shows the values for Microsoft Corporation Revenue.

Time Revenue Time Revenue Time Revenue Time Revenue

1 26 8 37 15 162 22 300
2 37 9 67 16 170 23 310
3 41 10 81 17 176 24 336
4 37 11 98 18 209 25 369
5 35 12 100 19 197 26 460
6 37 13 103 20 220 27 486
7 41 14 156 21 235 28 526

a. Using the data from time 1 to 20, fit the data using the nonlinear function

yt = β 0 e β t ε t
1
a. Using the analysis-of-variance (ANOVA) approach and t- test approach to
test whether or not the slope β1 = 0 using α = 5%.

b. Find R 2 and RMSE.

d. Test whether the errors have mean, µ = E[ε t ] = 0 , variance,
σ 2 = V [ε t ] = σ a2 is constant, normal distributed, εi and ε j are
uncorrelated for all i, j; i ≠ j .

e. Using the estimate model from (a) forecast the data from time 21-28.

Normality of the Error

(i) Jarque-Bera proposed normality of the error based on χ2 . The

test is defined as

H0 : The series follow normal distribution

H 1 : The series does not follow normal distribution

(n − k )  2 1 2 
χ2 =  S + ( K − 3) 
6  4 

Where n = number of observations,

k = number of parameters mode
K = kurtosis
S = Skewness
 The hipotesis nol is rejected if χ2 > χα
2
, v =n −1 .

(ii) Kolomogorov-Smirnov test

(iii) Anderson Darling test

Test for εi and ε j are uncorrelated (independent) for all

i, j ; i ≠ j .

(i)Durbin-Watson Test

H0 : The series are random independents (no-trends

series)
H 1 : The series are positive autocorrelated

∑(et − et −1 ) 2
d= 2

∑et2
1

Reject H 0 if d < d L
Not Rejected if d > d u
Inclusive if d L < d < d u (Other method should be used).

(i)Run Tests
H 0 : The series are random independents (no-trends
series)
H 1 : The series are dependent (has trend)

R = the number of runs above and below the median

µR = m +1
m( m −1)
σR =
2m −1

 n / 2 f o nr e v e n
m= 
 (n − 1) / 2 f o nr o d d
 R − µR
z=
σR

For m ≤ 20 , α = 0.1
Reject H 0 if R ≥ RU or R ≤ RL

For m > 20 , α
Reject H 0 if z ≥ zα / 2