Lecture 2

Almost all multivariate methods depend on functions

of the elements of a sample covariance matrix, usually
denoted by , or on the elements of a sample correlation
matrix, usually denoted by R.
The most important functions of the elements in a
matrix for multivariate data analysis purposes are (1) its trace,
(2) its determinant, and (3) its eigenvalues and eigenvectors.

Trace and Determinant

In each of the following definitions, let
11 12 L 1p

L 2p
= 21 22

M
M
O
M

L
pp
p1 p2
Note that is a square p p matrix. Also let be a symmetrix matrix.
Variance-covariance matrices and correlation matrices are always symmetric.

Definition 4.1
p

The trace of , denoted by tr (), is defined by tr () ii 11 22 L pp .

i 1

Definition 4.2
p

The determinant of a sqaure matrix , denoted by | |, is defined by | | 1j1 j

j 1

where 1 j (1)1 j |1 j | where 1 j is the matrix obtained from by deleting

its first row and its jth column.

Example
The characteristics equation of a second order (2 2)
matrix A can be written as:

Eigenvalues
Remark

When one expands the determinant expression,

the resulting equation has the form

c1 p c2 p 1 L c p c p 1 0
Thus, it is a polynomial equation in of degree p. The eigenvalues of
are defined as the roots of this polynomial equation.
If is symmetrix matrix, its eigenvalues are real numbers and, hence, they can
be ordered from largest to smallest: 1 2 L p .

Eigenvectors
Let be an eigenvalue of A. Then there exists
a vector v such that

Av v
The vector x is called an eigenvector of A associated
with the eigenvalue . Ordinarily we normalise v so that
it has length one, that is,

v v 1
T

Property of Eigenvectors
When two eigenvalues of (convariance or correlation matrix)
are not equal, their corresponding eigenvectors(vi , v j ) will be
orthogonal to one another, i.e.,

viT v j 0 when i j .

Thus,
2/ 13

is an eigenvector correspoinding to the eigenvalue -5.

-3/ 13
2/ 13

3/ 13

is an eigenvector correspoinding to the eigenvalue 7.

Question: Are they orthogonal?

Eigen/diagonal Decomposition
Let

be a square matrix

Theorem: Exists an eigen decomposition

Columns of U are eigenvectors of S

Diagonal elements of

are eigenvalues of

Diagonal decomposition:
why/how

Let U have the eigenvectors as columns: U v1 ... v p

Then, SU can be written

SU S v1 ... v p 1v1 ... p v p v1

Thus SU=U .
And S=U U1.

... v p
...

Problem
2
What is the eigen decomposition of S=
1

1
2 ?

Example continued

1 / 2 1 / 2 1 0 1 / 2
Then, S=

1 / 2 1 / 2 0 3 1 / 2
U

1/ 2

1/ 2

(U-1= UT )

Symmetric Eigen Decomposition

If

is a symmetric matrix:

Theorem: Exists an eigen decomposition

S QQ

where Q is orthogonal:
Q-1= QT
Columns of Q are normalized eigenvectors
Columns are orthogonal.

Positive Definite and Positive Semidefinite Matrices

If a matrix is symmetric and if all of its eigenvalues are positive, the matrix
is called a positive

definite matrix.

If a symmetric matrix has nonnegative eigenvalues and if at least one of its

eigenvalues is actually equal to zero, then the matrix is called a positive

semidefinite matrix.
If a matrix is either positive definite or positive semidenfinite, the matrix is
defined to be a nonnegative

matrix.

Remark Variace-covariance matrices and correlation matrices are always

nonegative matrices. That is, all of the eigenvalues of a variance-covariance

matrix or a correlation matrix will be nonnegative real numbers.

Vector Subtraction

Expectation of A Random Variable

E. g. Among the population in a town, 20% have an
annual income of 90,000 Yuan, 60% have an annual
income of 60,000 Yuan, and the remaining 20% have
an annual income of 30,000. What is the average
annual income?
90,000 20% 60,000 60% 30,000 20% 60,000

E.g. Supply, Inc. is a supplier for Demand, Inc. 10%

of the time, it takes Supply, Inc. 9 days to fill the
orders of Demand, Inc.; 50% of the time, it takes 10
days; 40% of the time, it takes 11 days. On average,
what is the number of days it takes Supply, Inc. to fill
the orders?
9 10% 10 50% 11 40% 10.3

Expectation
The expected value of a random
variable X, denoted by E(X), is the longrun average value of the random variable
over many repeated trials or occurrences.

E ( X ) xf ( x)
x

E ( X ) xf ( x)dx

Variance (Measures of Dispersion)

Why do we need it?
e.g. The number of working days required
to fill orders from two suppliers.
.5

.5
Dawson Supply, Inc.

.4
RelativeFrequency

R
elativeFrequency

.4

.3

.2

J.C. Clark Distributors

.3

.2

.1

.1

0
9

10

11

Number of Working Days

10

11

12

Number of Working Days

Mean=10.3
Which one would you choose?

13

14

15

Variance
The variance is the average of the squared
deviations (the difference between data value and the
mean). Formally, suppose X has mean E ( X ) . The
variance of X, is defined as:

Var ( X ) E[( X ) 2 ]
It follows that Var ( X ) 0
The variance provides a measure of the spread or
dispersion of the distribution around its mean.
The variance can be affected greatly by extreme
values.

Covariance and Correlation

Covariance: measure the association

between two random variables.
Let X and Y be random variables having a
X let E(X)=
Y
specified joint distribution, and
,
2
E(Y)= 2 ,

Y
X
Var(X)=
, Var(Y)=
. The covariance of X
and Y, is defined
Cov ( X , Y )as
E[( X X )(Y Y )]

X2 Y2
If
,
, then Cov(X,Y) will be
finite.
Cov(X,Y) can be positive, negative, or zero.

Correlation

If 0 X2 0 , Y2
, the correlation
of X and Y, is defined as
( X ,Y )

Cov ( X , Y )
XY

The range of possible values of the

correlation is:
1 ( X ,Y ) 1

Relationship between
independent and uncorrelated

Linear relationship and

correlation
If X i and X j are linear dependent, that is,
X j aX i b (a 0)
Please compute their correlation ij .

 Variance: How much X varies around the expected

value
Covariance is the measure the strength of the linear
relationship between two random variables
if X and Y are independent, then their covariance
is zero (converse is not true).
Correlation is also a measure of linear dependence.
range between 1 and +1

Scatter Plots of Data with

Various Correlation Coefficients
Y

r = -1

r = -.6

X
Y

r = +1

r=0

r = +.3

r=0

Linear Correlation
Linear relationships
Y

Curvilinear relationships
Y

X
Y

X
Y

Linear Correlation
Strong relationships
Y

Weak relationships
Y

X
Y

X
Y

Linear Correlation
No relationship
Y

X
Y

Univariate Normal Distribution

Based on a continuous random variable x, - <

Unimodal and symmetric.
Density function given by

f(x)

1
e
2

where = mean
= standard deviation

2 2

The Multivariate Normal

Distribution
The univariate normal
distribution has a
generalized form in p dimensions:

f(x)
%

1
p 2

where - xi , i =

12

'

x x -1
% %
% %
2

squared generalized
distance from x to
1,,p.
~

This p-dimensional normal density function is

denoted by Np(, ) where
~ ~
1
11 12 L 1p

2
21 22 L 2p

=
, =

M M O M
%
% M

p
p1 p2 L pp

Multivariate Normal Distribution for p=2

(Bivariate Normal)

f(x) =
%

12

'

x x -1
% %
% %
2

squared generalized
distance from x to

where - xi , i = 1, 2.

This 2-dimensional normal density function is

denoted
by N2(,) where
~ ~
1
11 12
= , =

%
% 2
21
22

Example
Consider a bivariate normal distribution with
8
=
6

6 2
and
=
,
2 3
how is the plot given by the equation
1

12

'

x -1 x
%%
%%
2

or
( x - ) ' 1 ( x - ) c2 ?

c1

TheBivariateNormalDistribution
f(x,y)

f(x,y)

f(x,y)

1
2

1
2
y

1
2

ContourPlotsoftheBivariateNormalDistribution
y

1
2

1
2

1
2

ScatterPlotsofdatafromtheBivariateNormalDistribution
y

1
2

1
2

1
2

Properties of the
Multivariate Normal
Distribution
For any multivariate
normal random vector X
~

1.

The density

f(x)
%

1
p 2

has maximum value

=
%

12

'

x x -1
% %
% %
2

at
1

i.e., the mean is equal to the mode!

2.

The density

f(x)
%

1
p 2

12

'

x x -1
% %
% %
2

is symmetric and is centered at , i.e., the

mean is equal to the median.
~

3. Zero covariance implies that the~

corresponding components of X are
independently distributed
~

4. Conditional distributions of the

components of X are (multivariate) normal
~
~

Some Other Important Results Regarding the

Multivariate Normal Distribution
5. If X ~ Np(, ), then any linear
~
~ ~
combination p

a'X =
%%

i=1

'
ai X i ~ N p a,
a 'a
%% %%%

Linear combinations of the components of X are

normally distributed

6. If X ~ Np(,), then any set of q linear

~
~ ~
combinations p

a1i X i
i=1

a 2i X i
'
'
'
A X = i=1
~
N
A,A
A
q

%%
%% %%%

M
p

a
X
qi i

i=1

Furthermore, if d is a vector of constants,

then
~
~
~
~ ~
X + d ~ Np( + d, )

7. If X ~ Np(, ), then all subsets of X are

~
~
~
~
(multivariate) normally distributed, i.e., for
any partition

X1

qx1
___ , = ___
= ,
px1
2
X2
p-q x1
p-q x1

qx1

X =
px1

12

11

qx p-q

qxq

pxp

21
p-q xq

22
p-q x p-q

then X1 ~ Nq( 1, 11), X2 ~ Np-q( 2, 22)

~

~
~

All subsets of the components of X have a

(multivariate) normal distribution

8.

if X1 ~ Nq1( 1, 11) and X2 ~ Nq2( 2, 22) and are

independent, then
0
X1
X% ~ N q1 + q 2
%2

1 11
% , %
2 0 22
%
%

The Sampling Distributions of X

The assumption that X1,,Xn constitute a
random sample
with mean and covariance . _
(1) For a univariate normal distribution, X is
normal with
1 2
population variance

mean and variance

sample size

(2) For the multivariate (p 2) X is normal

~
with
1

mean and covariance matrix

n %
%

(3) What if X is not normal? (Yes, still normal

when n is large, by CLT)

X (CLT)
,
2

n (
)

2
(x )
x ~ N , or
~ N 0 , 1
n
SE

2
SE

58