MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1

EARNINGS

EXP

This sequence provides a geometrical interpretation of a multiple regression model with

two explanatory variables.

© Christopher Dougherty 1999–2006 1

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1

EARNINGS

EXP

Specifically, we will look at an earnings function model where hourly earnings, EARNINGS,
depend on years of schooling (highest grade completed), S, and years of work experience,
EXP.
© Christopher Dougherty 1999–2006 2
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1

EARNINGS

EXP

The model has three dimensions, one each for EARNINGS, S, and EXP. The starting point
for investigating the determination of EARNINGS is the intercept, 1.

© Christopher Dougherty 1999–2006 3

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1

EARNINGS

EXP

Literally the intercept gives EARNINGS for those respondents who have no schooling and
no work experience. However, there were no respondents with less than 6 years of
schooling. Hence a literal interpretation of 1 would be unwise.
© Christopher Dougherty 1999–2006 4
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

pure S effect
1 + 2S
1

EARNINGS

EXP

The next term on the right side of the equation gives the effect of variations in S. A one
year increase in S causes EARNINGS to increase by 2 dollars, holding EXP constant.

© Christopher Dougherty 1999–2006 5

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1 + 3EXP
pure EXP effect

1

EARNINGS

EXP

Similarly, the third term gives the effect of variations in EXP. A one year increase in EXP
causes earnings to increase by 3 dollars, holding S constant.

© Christopher Dougherty 1999–2006 6

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1 + 2S + 3EXP
1 + 3EXP combined effect of S
pure EXP effect and EXP

pure S effect
1 + 2S
1

EARNINGS

EXP

Different combinations of S and EXP give rise to values of EARNINGS which lie on the
plane shown in the diagram, defined by the equation EARNINGS = 1 + 2S + 3EXP. This is
the nonstochastic (nonrandom) component of the model.
© Christopher Dougherty 1999–2006 7
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1 + 2S + 3EXP + u
u
1 + 2S + 3EXP
1 + 3EXP combined effect of S
pure EXP effect and EXP

pure S effect
1 + 2S
1

EARNINGS

EXP

The final element of the model is the disturbance term, u. This causes the actual values of
EARNINGS to deviate from the plane. In this observation, u happens to have a positive
value.
© Christopher Dougherty 1999–2006 8
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1 + 2S + 3EXP + u
u
1 + 2S + 3EXP
1 + 3EXP combined effect of S
pure EXP effect and EXP

pure S effect
1 + 2S
1

EARNINGS

EXP

A sample consists of a number of observations generated in this way. Note that the
interpretation of the model does not depend on whether S and EXP are correlated or not.

© Christopher Dougherty 1999–2006 9

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

EARNINGS = 1 + 2S + 3EXP + u

1 + 2S + 3EXP + u
u
1 + 2S + 3EXP
1 + 3EXP combined effect of S
pure EXP effect and EXP

pure S effect
1 + 2S
1

EARNINGS

EXP

However we do assume that the effects of S and EXP on EARNINGS are additive. The
impact of a difference in S on EARNINGS is not affected by the value of EXP, or vice versa.

© Christopher Dougherty 1999–2006 10

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

Yi   1   2 X 2 i   3 X 3 i  ui

Yˆi  b1  b2 X 2 i  b3 X 3 i

The regression coefficients are derived using the same least squares principle used in
simple regression analysis. The fitted value of Y in observation i depends on our choice of
b1, b2, and b3.
© Christopher Dougherty 1999–2006 11
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

Yi   1   2 X 2 i   3 X 3 i  ui

Yˆi  b1  b2 X 2 i  b3 X 3 i

ei  Yi  Yˆi  Yi  b1  b2 X 2 i  b3 X 3 i

The residual ei in observation i is the difference between the actual and fitted values of Y.

© Christopher Dougherty 1999–2006 12

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

RSS   ei2   (Yi  b1  b2 X 2 i  b3 X 3 i ) 2

We define RSS, the sum of the squares of the residuals, and choose b1, b2, and b3 so as to
minimize it.

© Christopher Dougherty 1999–2006 13

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

RSS   ei2   (Yi  b1  b2 X 2 i  b3 X 3 i ) 2

  (Yi 2  b12  b22 X 22i  b32 X 32i  2b1Yi  2b2 X 2 iYi

 2b3 X 3 iYi  2b1b2 X 2 i  2b1b3 X 3 i  2b2 b3 X 2 i X 3 i )

  Yi 2  nb12  b22  X 22i  b32  X 32i  2b1  Yi

 2b2  X 2 iYi  2b3  X 3 iYi  2b1b2  X 2 i
 2b1b3  X 3 i  2b2 b3  X 2 i X 3 i

RSS RSS RSS

0 0 0
b1 b2 b3
First we expand RSS as shown, and then we use the first order conditions for minimizing it.

© Christopher Dougherty 1999–2006 14

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

b1  Y  b2 X 2  b3 X 3

 X  X 2  Yi  Y    X 3 i  X 3 
2
2i

  X 3i  X 3  Yi  Y    X 2 i  X 2  X 3 i  X 3 
b2 
 X  X   X  X 
 2i 2  3i 3  2i 2 3i 3
2 2
   X  X  X  X   2

We thus obtain three equations in three unknowns. Solving for b1, b2, and b3, we obtain the
expressions shown above. (The expression for b3 is the same as that for b2, with the
subscripts 2 and 3 interchanged everywhere.)
© Christopher Dougherty 1999–2006 15
MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

b1  Y  b2 X 2  b3 X 3

 X  X 2  Yi  Y    X 3 i  X 3 
2
2i

  X 3i  X 3  Yi  Y    X 2 i  X 2  X 3 i  X 3 
b2 
 X  X   X  X 
 2i 2  3i 3  2i 2 3i 3
2 2
   X  X  X  X   2

The expression for b1 is a straightforward extension of the expression for it in simple

regression analysis.

© Christopher Dougherty 1999–2006 16

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

b1  Y  b2 X 2  b3 X 3

 X  X 2  Yi  Y    X 3 i  X 3 
2
2i

  X 3i  X 3  Yi  Y    X 2 i  X 2  X 3 i  X 3 
b2 
 X  X   X  X 
 2i 2  3i 3  2i 2 3i 3
2 2
   X  X  X  X   2

However, the expressions for the slope coefficients are considerably more complex than
that for the slope coefficient in simple regression analysis.

© Christopher Dougherty 1999–2006 17

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

b1  Y  b2 X 2  b3 X 3

 X  X 2  Yi  Y    X 3 i  X 3 
2
2i

  X 3i  X 3  Yi  Y    X 2 i  X 2  X 3 i  X 3 
b2 
 X  X   X  X 
 2i 2  3i 3  2i 2 3i 3
2 2
   X  X  X  X   2

For the general case when there are many explanatory variables, ordinary algebra is
inadequate. It is necessary to switch to matrix algebra.

© Christopher Dougherty 1999–2006 18

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

. reg EARNINGS S EXP

Source | SS df MS Number of obs = 540

-------------+------------------------------ F( 2, 537) = 67.54
Model | 22513.6473 2 11256.8237 Prob > F = 0.0000
Residual | 89496.5838 537 166.660305 R-squared = 0.2010
-------------+------------------------------ Adj R-squared = 0.1980
Total | 112010.231 539 207.811189 Root MSE = 12.91

------------------------------------------------------------------------------
EARNINGS | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
S | 2.678125 .2336497 11.46 0.000 2.219146 3.137105
EXP | .5624326 .1285136 4.38 0.000 .3099816 .8148837
_cons | -26.48501 4.27251 -6.20 0.000 -34.87789 -18.09213
------------------------------------------------------------------------------

EAR Nˆ INGS  26.49  2.68 S  0.56 EXP

Here is the regression output for the earnings function using Data Set 21.

© Christopher Dougherty 1999–2006 19

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

. reg EARNINGS S EXP

Source | SS df MS Number of obs = 540

-------------+------------------------------ F( 2, 537) = 67.54
Model | 22513.6473 2 11256.8237 Prob > F = 0.0000
Residual | 89496.5838 537 166.660305 R-squared = 0.2010
-------------+------------------------------ Adj R-squared = 0.1980
Total | 112010.231 539 207.811189 Root MSE = 12.91

------------------------------------------------------------------------------
EARNINGS | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
S | 2.678125 .2336497 11.46 0.000 2.219146 3.137105
EXP | .5624326 .1285136 4.38 0.000 .3099816 .8148837
_cons | -26.48501 4.27251 -6.20 0.000 -34.87789 -18.09213
------------------------------------------------------------------------------

EAR Nˆ INGS  26.49  2.68 S  0.56 EXP

It indicates that earnings increase by $2.68 for every extra year of schooling and by $0.56
for every extra year of work experience.

© Christopher Dougherty 1999–2006 20

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

. reg EARNINGS S EXP

Source | SS df MS Number of obs = 540

-------------+------------------------------ F( 2, 537) = 67.54
Model | 22513.6473 2 11256.8237 Prob > F = 0.0000
Residual | 89496.5838 537 166.660305 R-squared = 0.2010
-------------+------------------------------ Adj R-squared = 0.1980
Total | 112010.231 539 207.811189 Root MSE = 12.91

------------------------------------------------------------------------------
EARNINGS | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
S | 2.678125 .2336497 11.46 0.000 2.219146 3.137105
EXP | .5624326 .1285136 4.38 0.000 .3099816 .8148837
_cons | -26.48501 4.27251 -6.20 0.000 -34.87789 -18.09213
------------------------------------------------------------------------------

EAR Nˆ INGS  26.49  2.68 S  0.56 EXP

Literally, the intercept indicates that an individual who had no schooling or work experience
would have hourly earnings of –$26.49.

© Christopher Dougherty 1999–2006 21

MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE

. reg EARNINGS S EXP

Source | SS df MS Number of obs = 540

-------------+------------------------------ F( 2, 537) = 67.54
Model | 22513.6473 2 11256.8237 Prob > F = 0.0000
Residual | 89496.5838 537 166.660305 R-squared = 0.2010
-------------+------------------------------ Adj R-squared = 0.1980
Total | 112010.231 539 207.811189 Root MSE = 12.91

------------------------------------------------------------------------------
EARNINGS | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
S | 2.678125 .2336497 11.46 0.000 2.219146 3.137105
EXP | .5624326 .1285136 4.38 0.000 .3099816 .8148837
_cons | -26.48501 4.27251 -6.20 0.000 -34.87789 -18.09213
------------------------------------------------------------------------------

EAR Nˆ INGS  26.49  2.68 S  0.56 EXP

Obviously, this is impossible. The lowest value of S in the sample was 6. We have
obtained a nonsense estimate because we have extrapolated too far from the data range.

© Christopher Dougherty 1999–2006 22

Copyright Christopher Dougherty 1999–2006. This slideshow may be freely copied for
personal use.

© Christopher Dougherty 1999–2006 27.08.06