Linear Regression with One

Predictor variable
KNNL Chapter 1
Model Error Distribution Unspecified
{ } { } { }
0 1
0 1
2 2
1,...,
Response on the trial
, parameters (intercept and slope of line)
known constant, value of predictor variable on the trial
Random error term 0 ,
i i i
th
i
th
i
i i i i j
Y X i n
Y i
X i
E
| | c
| |
c c o c o o c c
= + + =

= =
{ } { } { }
{ } { } { }
{ } { } { }
( ) ( )
0 1 0 1 0 1 0 1
2 2 2 2
0 1
0 1 0 1
*
0 1 1 0 1
0
(Based on Rules in Appendix A)
0
, , , 0
Alternative Form:
i i i i i i i
i i i i
i j i i j j i j
i i i i i
i j
E Y E X X E X X
Y X
Y Y X X i j
Y X X X X X
| | c | | c | | | |
o o | | c o c o
o o | | c | | c o c c
| | | c | | c
= =

= + + = + + = + + = +
= + + = =
= + + + + = = =
= + + + = + +
*
0 0 1
X | | | = +
Least Squares Estimation - I
( )
( )
( ) ( )( )
0 1
0 1
0 1
2
2
0 1
1 1
0 1 0 1
0 1
0
1,...,

function of unknown , and observed data
Goal: select values of , that minimize and label them as ,
: 2 1
i i i
i i i
n n
i i i
i i
i i
i
Y X i n
Y X
Q Y X
Q b b
Q
i Y X
| | c
c | |
| |
c | |
| |
| |
|
= =
= + + =
= +

= =
c
=
c

( ) ( )( )
set
0 1
1 1 1
set
2
0 1 0 1
1 1 1 1
1
0
: 2 0
n n n
i i
i i
n n n n
i i i i i i i
i i i i
Y nb b X
Q
ii Y X X X Y b X b X | |
|
= = =
= = = =
= = +
c
= = = +
c


Least Squares Estimation - II
( ) ( ) ( ) ( )
1
2
2
1
1 1 1 1 1
2
1 2
1 1
1
1 1
1
Solving (by multiplying by and by and taking ) :
n
i
i
n n n n n
i i i i i i
i i i i i
n
n n
i
i i n n
i
i i
i i i
i i
i
i X ii n ii i
n X Y X Y b n X X
X
X Y
X Y b X
n n
X Y
b
=
= = = = =
=
= =
= =

| |
| |
= |
|
|
\ .
\ .
| |
| |
|
|
\ . |
=
|
|
|
\ .
=

( )( )
( )
( )
( )
( )
1 1
1 1
2 2
1 1
1
2 1
1
0 1
1 1
1
From :
n n
i i n n
i i
i i i n n
i
i i XY
i i i
n
n
i i
XX XX
i
i
i
n
i
i
i
n n
i
i i i
i i
XX
X Y
X X Y Y
X X
SS
n
Y k Y
SS SS
X X
X
X
n
X X X
i b Y b X Y l Y
n SS
= =
= =
= =
=
=
=
= =


= = = =
| |

|
\ .

| |

|
= = + =
|
\ .

Fitted Values and Residuals
{ }
0 1 0 1
^
0 1
^
0 1
True Regression Function: (Unknown, since , parameters)
Estimated Regression Function (Fitted):
For the observation: 1,...,
Residuals:Differences between ob
th
i
i
E Y X
Y b b X
i Y b b X i n
| | | | = +
= +
= + =
( )
( )
( )
( )
^
0 1
1
1
^
0 1 0 1
1 1 1 1
served and fitted (predicted) values:
1,...,
Properties of Residuals:
0 (From LS eq )
0 (From LS eq )
0
i
i i i i
n
i
i
n
i i
i
n n n n
i
i i i i i i
i i i i
e Y Y Y b b X i n
e i
X e ii
Y e b b X e b e b X e
=
=
= = = =
= = + =
=
=
= + = + =

Estimating Error Variance o
2

{ } { } ( )
{ }
( )
{ }
{ }
( )
( )
2
2
2 2 2
0 1
^
0 1
2
2
^
2
2
1 1
0
unobservable since
We use residual to "estimate"
Obtain the "average" squared residual to estimate :
2 2 2
n n
i
i i
i i
E E E E
Y X
e
e Y Y Y b b X
e Y Y
SSE
s M
n n n
o o c c c c c
c c | |
c
o
= =
= = = =
= +
= = +
| |

|
\ .
= = = =


SE
Normal Error Model
( )
( )
( )
( ) ( )
( )
2
0 1
2
0 1
2
/ 2
0 1 2
0 1
1 1
2
^ ^ ^
0 1
1,..., ~ 0, (independent)
1 1
exp 1,...,
2
2
1
, , 2 exp
2
Goal: Choose values , , th
i i i i
i i
i i
n n
n
i i n
i
i i
Y X i n N
y X
f y f i n
y X
L f
| | c c o
| |
o
to
| |
| | o t o
o
| | o

= =
= + + =

+ | |

= = =
` |
\ .
)

+ | |

= =
` |
\ .
)
[
( )
( )
( )
( )
2
0 1 2
1
2
0 1
0 1
1
^ ^
0 1 0 1
2 2
at maximize (or equivalently ln( )) :
1
ln 2 ln
2 2 2
Note: maximizing wrt , is same as minimizing
,
1 1
1
2 2
n
i i
i
n
i i
i
L l L
y X
n n
l
y X
l
b b
y
l n
| |
t o
o
| |
| |
o
| |
o o
=
=
=
+ | |
| |
=
| |
\ .
\ .
+ | |
|
\ .
= =
c
| |
=
|
c
\ .

( ) ( )
( )
2
^ ^
2
2
0 1 2
^
0 1
2 1 1
2
2
1
2
0
n n
i i i n
set
i i
i i
i
y X e
X
n
s
n n n
| |
| |
o
o
= =
=
| |
| |
+
|
|
+

\ .
\ .
= = = =