Chapter 2.

Review of Linear Algebra

Matrix Algebra

Definition
An m × n matrix, A, is a rectangular array of
elements
 a11 a12 a1n 
a 
A   aij    21 a22 a2 n 
 
 
 am1 am 2 amn 

m = # of rows
n = # of columns
dimensions = m × n
Vector
Definition
A vector, v, of dimension n is an n × 1 matrix
rectangular array of elements

 v1 
v 
v  2

 
 
vn 
vectors will be column vectors (they may also be row vectors)
Matrix Operations

Addition
Let A = (aij) and B = (bij) denote two n × m
matrices Then the sum, A + B, is the matrix

 a11  b11 a12  b12 a1n  b1n 

a b  a2 n  b2 n 
A  B   aij  bij    21 21 a22 b22

 
 
 am1  bm1 am 2  bm 2 amn  bmn 

The dimensions of A and B are required to be

both n × m.
Scalar Multiplication

Scalar Multiplication
Let A = (aij) denote an n × m matrix and let c be
any scalar. Then cA is the matrix

 ca11 ca12 ca1n 

 ca ca2 n 
cA   caij    21 ca22

 
 
cam1 cam 2 camn 
Addition for vectors
v3

 v1  w1 
v  w  v2  w2 
 w1 
 v3  w3 
w   w2 
 w3 

 v1 
v  v2 
v2
 v3 

v1
Scalar Multiplication for vectors
v3
 cv1 
cv  cv2 
 cv3 

 v1 
v  v2 
 v3 

v2

v1
Matrix Multiplication

Matrix multiplication
Let A = (aij) denote an n × m matrix and B = (bjl)
denote an m × k matrix
Then the n × k matrix C = (cil) where
m
cil   aij b jl
j 1

is called the product of A and B and is denoted

by A∙B
In the case that A = (aij) is an n × m matrix and B
= v = (vj) is an m × 1 vector m
Then w = A∙v = (wi) where wi   aij v j
j 1

is an n × 1 vector A
w3
v3
 v1 
v  v2 
 v3 

w2
v2  w1 
w   w2   Av
 w3 

w1
Some particular types of matrices

Definition
An n × n identity matrix, I, is the square matrix
1 0 0
0 1 
0
I  In  
 
 
0 0 1

Note:
1. AI = A
2. IA = A.
Inverse
Definition (The inverse of an n × n matrix)
Let A denote the n × n matrix
 a11 a12 a1n 
a 
 
 a a 2n 
A  aij  21 22

 
 
 an1 an 2 ann 
Let B denote an n × n matrix such that
AB = BA = I,
If the matrix B exists then A is called invertible Also B is called the
inverse of A and is denoted by A-1
The transpose of a matrix

Consider the n × m matrix, A

 a11 a12 a1n 
a a2 n 
A   aij    21 a22
 
 
 am1 am 2 amn 
then the m × n matrix,A (also denoted by AT)
 a11 a21 am1 
a am 2 
A   a ji    12 a22
 
 
 am1 am 2 amn 
Symmetric Matrices

• An n × n matrix, A, is said to be symmetric if

A  A
Note: 
 AB   BA
 AB   B A
1 1 1


 A  A 
1 1
Trace and determinant of a square matrix

The trace and the determinant

of a square matrix
Let A denote then n × n matrix
 a11 a12 a1n 
a a2 n 
A   aij    21 a22

 
 
 an1 an 2 ann 

n
Then tr  A   aii
i 1
  a11 a12 a1n 
a a a2 n 
A  det  21 22
 the determinant of A
 
 
n

 an1 an 2 ann  
 aij Aij
j 1

where Aij  cofactor of aij 

 the determinant of the matrix 
 1
i j
 
 after deleting i row and j col. 
th th

 a11 a12 
det    a11a22  a12 a21
a21 a22 
Some properties

1. I  1, tr  I   n

2. AB  A B , tr  AB   tr  BA

1 1
3. A 
A

 A11  
A12   22 11 12 22 A21
A A A A1

4. A   
 A21 A22   A11 A22  A21 A111 A12

 A22 A11 if A12  0 or A21  0
Special types of Matrices

1. Orthogonal matrices
– A matrix is orthogonal if PˊP = PPˊ = I
– In this cases P-1=Pˊ .
– Also the rows (columns) of P have length 1 and
are orthogonal to each other
Ortogonal matrices (continued….)

Suppose P is an orthogonal matrix

then PP  PP  I
Let x and y denote p × 1 vectors.
Let u  Px and v  Py

Then uv   Px   Py   xPPy  xy

and uu   Px   Px   xPPx  xx

Orthogonal transformation preserve length and

angles – Rotations about the origin, Reflections
Ortogonal Matrix example

The following matrix P is orthogonal

1 3
1
3
1
3

1 
P 2
 1
2
0 
1 1  2 6 
 6 6
Special types of matrices (continued)

2. Positive definite matrices

– A symmetric matrix, A, is called positive definite if:
 
x Ax  a11x12    ann xn2  2a12 x1 x2   2a12 xn 1 xn  0
 
for all x  0
– A symmetric matrix, A, is called positive semi definite if:
 
xAx  0
 
for all x  0
Theorem The matrix A is positive definite if
A1  0, A2  0, A3  0,, An  0

where
 a11 a12 a13 
 a11 a12  a a , 
A1  a11  , A2   , A 
 3  12 22 23  a
 12 22 
a a
a13 a23 a33 
 a11 a12  a1n 
a a  a 
and An  A   12 22 2n 

    
 
a1n a2 n  ann 
 Example

 1 .5 .25 .125
 .5 1 .5 .25 
A  A  A4  0.421875  0
 .25 .5 1 .5 
 
.125 .25 .5 1 

 1 .5 .25

A3  det  .5 1 .5  A3  0.5625  0
.25 .5 1 

 1 .5
A2  det   A2  0.75  0, A1  det1  1  0
.5 1 
Special types of matrices (continued….)

3. Idempotent matrices
– A symmetric matrix, E, is called idempotent if:
EE  E
– Idempotent matrices project vectors onto a linear
subspace 
 
EEx   Ex
x


Ex
Quadratic Forms

• A function of n variables f(x1,x2,…..,xn) is called

a quadratic form if
n n

• f ( x1 , x2 ,...., xn )   qij xi x j  xT Qx, where Q(nxn) = [qij]

i 1 j 1
xT
and = (x1,x2,…..,xn).
• Q is assumed to be symmetric.
• Otherwise, Q can be replaced with the symmetric
matrix (Q+QT)/2 without changing value of the
quadratic form.
Definitions

• A matrix Q is positive definite when xTQx > 0

for all x ≠ 0.
 2  1
• Q 
 1 2 
is positive definite.
• A matrix Q is positive semi-definite when xTQx
≥ 0 for all x and there exists an x ≠ 0 such that
xTQx = 0.
 1  1
• Q  is positive semi-definite.
 1 1 
• A matrix Q is negative definite when xTQx < 0
for all x ≠ 0.
 1 1
• Q 
 1 1
is negative definite.
Definitions

• A matrix Q is negative semi-definite when –Q

is positive semi-definite.
• is negative semi-definite.

• A matrix Q is indefinite when xTQx > 0 for

some x and < 0 for other x.
• is indefinite.
 Principal Minor

• Principal minor of order k: A sub-matrix obtained

by deleting any n-k rows and their corresponding
columns from an n x n matrix Q.
1 2 3
• Consider Q  4 5 6
 
7 8 9

• Principal minors of order 1 are diagonal elements

1, 5, and 9. 1 2 1 3 5 6
• Principal minors of order 2 are 4 5, 7 9 and 8 9
     
• Principal minor of order 3 is Q.
• Determinant of a principal minor is called
principal determinant.
• There are 2n-1 principal determinants for an n x n
matrix.
Leading Principal Minor

• The leading principal minor of order k of an n x n

matrix is obtained by deleting the last n-k rows
and their corresponding columns.
1 2 3
• Q  4 5 6
7 8 9

• Leading principal minor of order 1 is 1.

• Leading principal minor of order 2 is 1 2
4 5
 
• Leading principal minor of order 3 is Q itself.
• No. of leading principal determinants of an n x n
matrix is n.
Test

• Jacobi Criterium for quadratic forms:

– If A is regular (|A|≠0)
• If all the leading minors principals are >0, then A is a Positive
Definited Matrix
• If the leading minors principals have alternate sign,
beginning with negative, then A is Negative Definited
(A1<0, A12>0, A123<0, ….)
• Otherwise A is undefined
– If A is singuar (|A|=0)
• If all the minors principals are ≥0, then A is a Positive Semi-
Definited Matrix
• If the odd minors principals are ≤ 0 and the even minors
principals ≥0, then A is Negative Semi-Definited
(A1<0, A12>0, A123<0, ….)
• Otherwise A is undefined
Exercises
 Convex Sets
• The intersection of convex sets is a convex set
(Figure A.4).
• The union of convex sets is not necessarily a
convex set (Figure A.4).
 Convex Sets
• A convex combination of vectors x(1) , x(2),…., x(n) is
a vector x such that
• x = λ1x(1)+λ2x(2)+…+λkx(k)
• λ1+λ2+…+λk = 1
• λi≥0 for i = 1,2,….,k
• Extreme point: A point in convex set that cannot
be expressed as the midpoint of any two points in
the set
• Convex set S = {(x1 ,x2)|0≤x1≤2,0≤x2≤2}
• This set has four extreme points (0,0), (0,2),(2,0) and
(2,2)
• A hyperplane is a convex set.
Convex Function

 Convex function: A function is convex on set

D if and only if for any two points x(1) and x(2)
Є D and 0 ≤ λ ≤ 1,
• f[λx(1) + (1-λ)x(2)] ≤ λf(x(1) ) +(1- λ) x(2)
Properties of Convex Functions
 Property
• The linear approximation of f(x) at any point in
the interval always underestimates the true
function value
f ( x)  f ( x 0 )  f ( x 0 )( x  x 0 ) for all x
Convexity of a Function

 The Hessian matrix of a function f(x1, x2 ,…, xn) is an n

x n symmetric matrix given by
 2f 
H f ( x1 , x2 ,...., xn )      2
f
x1x2 
 Test for convexity of a function: A function f is convex
if the Hessian matrix of f is positive definite or
positive semi-definite for all values of x1, x2 ,…, xn.
 Test for concavity of a function: A function f is convex
if the Hessian matrix of f is negative definite or
negative semi-definite for all values of x1, x2 ,…, xn.
 Example
f ( x1 , x2 , x3 )  3x  2 x  x  2 x1 x2  2 x1 x3  2 x2 x3  6 x1  4 x2  2 x3
2
1
2
2
2
3

 6 x1  2 x2  2 x3  6  • H is a symmetric.
 
f ( x1 , x2 , x3 )   4 x2  2 x1  2 x3  4  • All diagonal elements are positive.
 2x  2x  2x  2 
 3 1 2 
• The leading principal determinants are
 6  2  2
6 2
H f ( x1 , x2 , x3 )   2 4 2  6 0
2 4
 20  0
 2 2 2 
H f  16  0

• H is a positive-definite matrix , which implies f is convex function

Optimization exercises (mathematical programming
without constrains)