F-tables.

CONCLUSION: The t - d i s t r i b u t i o n i s OK when you o n l y

have one independent v a r i a b l e . When you have more, " t h e number

o f independent v a r i a b l e s t h a t i t t o o k t o e x p l a i n t h a t much

v a r i a n c e " must be taken i n t o account.

NOTE: R-squared o n l y i n d i c a t e s t h e p r o p o r t i o n o f v a r i a n c e

explained. I t does n o t t a k e i n t o account how many v a r i a b l e s

were used t o e x p l a i n t h a t much v a r i a n c e .

c ) The F - t e s t DOES t a k e i n t o account t h e number o f independent

v a r i a b l e s needed t o r e n d e r t h e v a r i a n c e e x p l a i n e d :

k -
- k -- k
, where
Fn-k-l S S ~ ~ 2~ ~ ~
1 - R
n - k - 1 n - k - 1

k = # independent v a r i a b l e s i n t h e r e g r e s s i o n e q u a t i o n .

Some p o i n t s w o r t h n o t i n g r e g a r d i n g t h i s F - t e s t :

A1 though p r o p o r t i o n a l , t h e above F - r a t i o should NOT suggest

t o you t h a t t h e s e two numbers a r e e q u i v a l e n t .

S S ~ ~ ~
21 Regression Mean Square =
k

3
31 Mean Square E r r o r = - -3 ~ ~ ~ ~ ~

41 T h i s i s an F - t e s t t o e v a l u a t e R' ( i .e., t o e v a l u a t e whether

a l l independent v a r i a b l e s t o g e t h e r e x p l a i n a p r o p o r t i o n o f

t h e v a r i a n c e i n t h e dependent v a r i a b l e t h a t i s g r e a t e r t h a n

zero). The hypotheses a r e as f o l l o w s :

168
 d. Extending the above illustration, a 95% confidence interval for the
partial correlation between math aptitude and creativity can be found
as follows:
T(rMC.GR) = T(-.5) = -.5493

T(-.5) f 1.96
1 25 - 31 - 2 OR (-.9876, -.1110)

Then, T- 1 (-.9876) = -.756 and T- 1 (-.1110) = -.I105 , leaviug us

with a 95% confidence interval of (-.756, -.1105).

C. Standardized partial regression coefficients

1. You will only be given the for~iiulas for calculating partial regression
slopes in the "trivariate" case (i.e., in the case with two independent
variables). The formulas for standardized partial slopes are as
foll ows:

A
NOTE: Standardized partial regression coefficients (/Ii) are sometimes
referred to as BETA WEIGHTS or as PATH COEFFICIENTS.

We shall not go into detail about how partial regression coefficients

are calculated when there are more than two independent variables, other
than to say that they are assigned values that minimize the sum of
squared deviations from the regression line-hich is only to say that
these coefficients are OLS estimates.

2. In bivariate regression the standardized regression coefficient equals

the correlation coefficient. UNLIKE in bivariate regression, the
 sense, because t h e case o f no c o l l i n e a r i t y e n t a i l s t h a t each

independent v a r i a b l e e x p l a i n s a d i s t i n c t (and t h u s summable) amount

o f the variance i n Y.

3. Now l e t us proceed w i t h two cases i n which $ 0 .

rx1x2

a. I n t h e f i r s t case, t h e d a t a a r e c o n f i g u r e d as f o l l o w s :

Males = 1 m m

Income

Females = 0

Education

You should be a b l e t o guess t h e s i g n s (and t o approximate, r o u g h l y ,

t h e magnitudes) o f b i v a r i a t e c o r r e l a t i o n s by s e e i n g how t h e y a r e

distributed i n a plot. For p r a c t i c e i n d o i n g t h i s , you may want t o

v i s i t t h e f o l l o w i n g website:

http://www.public.iastate.edu/-carlos/40l/1ectures/app1ets/guess
I n any case, t h e d a t a p o i n t s ( l a b e l e d w i t h m's and f ' s ) i n t h e above

f i g u r e appear t o be s l i g h t l y h i g h e r as one l o o k s f r o m l e f t t o r i g h t .

T h i s suggests a modest p o s i t i v e a s s o c i a t i o n between income and

education. Note a l s o t h a t t h e average incomes f o r men and women a r e

n e a r l y i d e n t i c a l i n t h e p l o t , suggesting t h a t income and gender may

be s t a t i s t i c a l l y independent. F i n a l l y , i t should be c l e a r t h a t t h e

average e d u c a t i o n o f women i s much h i g h e r t h a n t h a t o f men. Thus i t

would n o t be unreasonable f o r us t o f i n d t h a t t h e t h r e e v a r i a b l e s '

intercorrelations are ryX = .1 and ryX= 0 and rX = -.9 .

1 2 1 2

N o t i c e what happens when t h e p a r t i a l b e t a between income and

194
e d u c a t i o n i s e s t i m a t e d i n a r e g r e s s i o n i n which gender i s a l s o an

independent v a r i a b l e :

A A
Which i s MUCH ( ! ! ! ) l a r g e r t h a n ryX = .1 = /3 (where /3 i s t h e
1
s t a n d a r d i z e d s l o p e from t h e r e g r e s s i o n o f income on e d u c a t i o n ) .

COMMENTS :

1) Note how t h e p a r t i a l b e t a i s t h e s l o p e between a s t a n d a r d i z e d

dependent and a s t a n d a r d i z e d independent v a r i a b l e a f t e r t h e

independent v a r i a b l e has been a d j u s t e d f o r t h e o t h e r independent

variables i n the equation. (See t h e arrow i n t h e f i g u r e . )

2) The s t r a n g e composition o f t h e sample (namely, t h a t i t o n l y has

p o o r l y educated ma1es and we1 1 educated females) SUPPRESSES t h e

" t r u l y " s t r o n g r e l a t i o n between e d u c a t i o n and l a t e r income. (In a

f o l l o w - u p study, one m i g h t recommend u s i n g a STRATIFIED SAMPLE t o

p r e v e n t t h i s from o c c u r r i n g . )

3) NOTICE a l s o ( f r o m t h e f o r m u l a on p. 176 o f t h e s e L e c t u r e Notes f o r

t h e s t a n d a r d i z e d s l o p e i n t h e t r i v a r i a t e case) t h a t suppression
A
occurs ( i . e . , lall > lryXI ) whenever ryX * * (0.
1 1 rx1x2

I t always i s a c o u n t e r i n t u i t i v e f i n d i n g when t h e c o r r e l a t i o n s

among t h r e e v a r i a b l e s have a n e g a t i v e p r o d u c t . A u s e f u l analogy

here i s t o t h e g r e a t e r l i k e l i h o o d t h a t " t h e enemy o f my enemy i s

my f r i e n d " ( v i z . , when a p o s i t i v e p r o d u c t r e s u l t s from

" m u l t i p l y i n g " two n e g a t i v e a s s o c i a t i o n s and a p o s i t i v e one), t h a n

 Stat 404

Multiple Regression (Cont’d)

A. Hierarchical regression models

1. The following is an F-test for comparing hierarchically related regression models:

(R
2
Y . X1 ... X k − RY2. X ... X )
(k − g )
1 g

Fk − g ,n−k −1 =
(1 − R 2
) , where
(n − k − 1)
Y . X1 ... X k

RY2. X 1 ... X k (or Rc2 ) is the proportion of variance explained in the complete model,
RY2. X ... X (or Rr2 ) is the proportion of variance explained in the reduced model, and
1 g

k and g are the respective degrees of freedom for the complete and reduced models.

Note that Pedhazur uses k1 (instead of k) and k2 (instead of g) here. Also please do not

mistake this F-test for the one mentioned earlier (Lecture notes, p. 168) and below (p. 3)

for evaluating whether the overall variance explained by a regression model is

statistically significant.

2. So, what are hierarchically related regression models?

a. To understand this, one must first understand the distinction between higher-order

versus lower-order variables, and their relation to misspecified regression models.

i. Higher-order variables are (by definition) constructed from lower-order variables.

Later in the course we shall multiply variables together to create new variables

(namely, interaction and polynomial variables). These constructed variables are

of a “higher order” than any of the variables multiplied during their construction.

1
 ii. Lower-order variables that are not themselves constructed from even-lower-order

variables are sometimes referred to as having marginal effects (as distinct from

interaction effects or polynomial effects) on the dependent variable.

iii. A regression model is said to be misspecified if it includes a higher-order variable

but excludes any lower-order variable from which it was constructed.

Misspecified regression models (almost always) yield meaningless results and

should be avoided.

b. Two regression models are said to be hierarchically ordered if three conditions are

met:

i. Both models have the same dependent variable.

ii. The independent variables in one model (namely, the reduced model) are a subset

of the independent variables in the other model (namely, the complete model).

iii. Neither the reduced model nor the complete model is misspecified.

3. The just-introduced F-test is used in evaluating which between hierarchically ordered

regression models provides the more parsimonious fit to one's data.

B. What is adjusted R-squared?

1. Pedhazur raises this question on page 207 under the subheading, “shrinkage.”

2. If a dependent variable were unrelated to a set of independent variables, you would NOT

expect to find R2 = 0. Such a finding would be about as rare as one that R2 = 1.

3. Instead, you would expect that the average proportion of variance explained by the

independent variables would equal the average proportion of variance explained per

2
 residual degree of freedom. This is why the following F-test is used to evaluate the

statistical significance of R2:

RY2. X1... X k
Fk ,n−k −1 = k
(1 − R 2
)
(n − k − 1)
Y . X1 ... X k

Note that F equals 1 when the average proportion of variance explained by the k

independent variables equals the ratio of the proportion of the “left over” (or residual)

variance in Y to the number of residual degrees of freedom.

4. Thus built into this F-test is the expectation that …

RY2. X1... X k
k =1
(1 − R 2
)
(n − k − 1)
Y . X1 ... X k

or that…

RY2. X 1 ... X k 1 − RY2. X 1 ... X k

=
k n − k −1

RY2. X 1 ... X k (n − k − 1) = k − kRY2. X 1 ... X k

RY2. X 1 ... X k (n − k − 1 + k ) = k

k
RY2. X1... X k =
n −1

3
5. Accordingly, if you were to have a sample of n=5 and if with 2 independent variables you

explain half (50%) of the variance in Y, you should NOT get excited.

6. The formula for adjusted R-squared (Pedhazur, p. 208) is as follows:

2
Radj

[
= 1 −  1 − RY2. X1... X k ]
n −1 

n − k −1 

Substituting our expected value of RY2. X 1 ... X k into this expression, we get…

 k  n −1 
2
Radj = 1 −  1 −  
 n − 1  n − k − 1 

 n −1 k  n − k −1
= 1−  −  = 1 − =0
 n − k −1 n − k −1 n − k −1 .

k
2
7. Although Radj = 1 whenever RY2. X 1 .. X k = 1 , note that Radj
2
< 0 whenever RY2. X 1 ... X k < .
n −1

Thus unlike RY2. X 1 ... X k (which can never be negative) Radj

2
may not be interpreted as a

proportion of variance explained.

2
8. Radj is a measure of linear association greater than would be expected by chance (i.e., by

random sampling error).

9. If one returns to the illustration on multiple regression that begins on p. 200 of your

lecture notes, recall that in the regression of HINCOME on PINCOME and REDUC

RH2 . PR = .431 . This can be converted to an adjusted R-squared as follows:

 63 − 1 
2
Radj = 1 −  [1 − .431]  = .412
 63 − 2 − 1 