Linear Algebra–A Review

EC-424: Mathematical Optimization for Economics

August 3, 2016

1 Vector Space
A vector space over Rn , (henceforth, simply vector space) is a set V , on which
are defined two operators “addition”, which specifies, for each x and y in V ,
an element (x + y) ∈ V ; and “scalar multiplication”, which specifies for each
a ∈ R and x ∈ V , an element ax ∈ V . These operators are required to satisfy
the following axioms ∀x, y, z ∈ V , and ∀a, b ∈ R:

a. x + y ∈ V whenever x and y are in V (closure under addition).

b. x + y = y + x (commutative law for addition).
c. x + (y + z) = (x + y) + z (associative law for addition).

d. ∃0 ∈ V such that ∀x ∈ V x + 0 = x (additive identity).

e. ∀x ∈ V, ∃(−x) ∈ V such that x + (−x) = 0 (additive inverse).
f. a · x ∈ V whenever x is in V (closure under scalar multiplication).

g. a · (x + y) = a · x + a · y (distributive under scalar multiplication).

h. (a + b) · x = a · x + b · x (distributive under scalar multiplication).
i. (a · b) · x = a · (b · x) (associative law for scalar multiplication).
j. ∃1 ∈ R such that ∀x ∈ V 1 · x = x.

Throughout this review, letters a, b, c will denote the scalars, while x, y, z will
denote the elements of V .
Remark. A subset of Rn that satisfies all the axioms listed above is called a
subspace of Rn .
Definition 1.1. The most restrictive structure on V is that of inner product.
An inner product on V is a function h·, ·i: V ×V 7→ R that satisfies the following
conditions ∀x, y ∈ V :

1
 1. Symmetry: hx, yi = hy, xi.
2. Bilinearity: hax + by, zi = ahx, zi + bhy, zi, and hx, ay + bzi = ahx, yi +
bhx, zi.
3. Positivity: hx, xi ≥ 0, hx, xi = 0 iff x = 0.
An inner product space is a pair (V, h·, ·i), where V is a vector space over R,
and h·, ·i is an inner product on V .
Definition 1.2. A norm is a function k·k: V 7→ R that satisfies the following
requirements ∀x, y ∈ V :
1. Positivity: kxk ≥ 0, kxk = 0 iff x = 0.
2. Homogeneity: kaxk = kak · kxk.
3. Triangle inequality: kx + yk ≤ kxk + kyk.
A normed space is a pair (V, k·k), where V is a vector space, and k·k is a norm
on V .
Remark. The distance between two points x1 and x2 in Rn is the Euclidean
distance function, represented by:

n
!1/2
X
dE (x1 , x2 ) = kx1 − x2 k = (xi − yi )2 (1)
i=1

which in turn is based on the Euclidean norm

n
!1/2
X
kxk = x2i (2)
i=1
.
One can think of a norm as a measure of the size of the vector x or more
specifically as a measure of the distance from the point x to the origin 0.
Definition 1.3. A metric on V is a function d: V × V 7→ R that satisfies the
following conditions ∀x, y, z ∈ V :
1. Positivity: d(x, y) ≥ 0, and d(x, y) = 0 iff x = y.
2. Symmetry: d(x, y) = d(y, x).
3. Triangle Inequality: d(x, z) ≤ d(x, y) + d(y, z).
A metric space is a pair (V, d), where V is a vector space, and d is a metric
on V .

2
Definition 1.4. Vectors x1 , x2 , . . . , xk ∈ Rn , are linearly dependent iff there
exist scalars, c1 , c2 , . . . , ck not all zero, such that
c1 x1 + c2 x2 + . . . + ck xk = 0.
Definition 1.5. Vectors x1 , x2 , . . . , xk ∈ Rn , are linearly independent iff for
scalars, c1 , c2 , . . . , ck ,
c1 x1 + c2 x2 + . . . + ck xk = 0,
implies c1 = c2 = . . . = ck = 0.
Definition 1.6. Let x1 , x2 , . . . , xk be a fixed set of vectors in Rn . Let the set
of all linear combinations of x1 , x2 , . . . , xk ,
L[x1 , x2 , . . . , xk ] ≡ {c1 x1 + c2 x2 + . . . + ck xk | c1 , c2 , . . . , ck ∈ R}.
be called the set generated or spanned by x1 , x2 , . . . , xk . Let there be a V ⊆ Rn .
If there exists x1 , x2 , . . . , xk ∈ Rn , such that every x ∈ V can be written as a
linear combination of x1 , x2 , . . . , xk :
x ∈ L[x1 , x2 , . . . , xk ],
we say that x1 , x2 , . . . , xk span V .
Remark. (a) A set of vectors that span Rn must contain at least n vectors.
(b) Let x1 , x2 , . . . , xk be a set of k vectors in Rn . Form the n × k matrix whose
columns are these x0i s:

A = (x1 , x2 , . . . , xk ).

Let b be a vector in Rn . Then, b lies in the space L[x1 , x2 , . . . , xk ], spanned

by x1 , x2 , . . . , xk iff the system Ac = b has a solution, where c is a k × 1
vector.
(c) Let x1 , x2 , . . . , xk be a set of k vectors in Rn . Form the n × k matrix whose
columns are these x0i s:

A = (x1 , x2 , . . . , xk ).

Then, x1 , x2 , . . . , xk span Rn iff the system of equations Ax = b has a

solution x for every b ∈ Rn .
Definition 1.7. Let x1 , x2 , . . . , xk be a fixed set of k vectors in Rn . Let V
be the set L[x1 , x2 , . . . , xk ] spanned by x1 , x2 , . . . , xk . Then if x1 , x2 , . . . , xk are
linearly independent, then x1 , x2 , . . . , xk is called a basis of V . More generally,
let y1 , y2 , . . . , ym be a collection of vectors in V . Then, y1 , y2 , . . . , ym forms a
basis of V if:
(a) y1 , y2 , . . . , ym span V , and

3
(b) y1 , y2 , . . . , ym are linearly independent.
Remark. (a) Every basis of Rn contains n vectors.
(b) Let x1 , x2 , . . . , xn be a set of n vectors in Rn . Form the n × n matrix whose
columns are these x0i s:

A = (x1 , x2 , . . . , xn ).

Then, the following statements are equivalent:

(1) x1 , x2 , . . . , xn are linearly independent.

(2) x1 , x2 , . . . , xn span Rn .
(3) x1 , x2 , . . . , xn form a basis of Rn .
(4) The determinant of A is non-zero.

2 Matrices
A matrix is simply a rectangular array of numbers. So, any table of data is
a matrix. The size of matrix is indicated by the number of its rows and the
number of its columns. A matrix with k rows and n columns is called a k ×n (“k
by n”) matrix. The number in row i and column j is called the (i, j)th entry,
and is often written as aij (for a matrix A). Two matrices are equal if they both
have the same size and if the corresponding entries in the two matrices are equal.

Rank and trace of a Matrix

The rank of a k×n matrix A is defined as the largest number of linearly indepen-
dent row vectors (or the largest number of linearly independent column vectors -
these two numbers always being equal), denoted as r(A). Thus, r(A) ≤ min k, n.

2.1 Special Kinds of Matrices

In this section we list some of the important classes of k × n matrices that arise
in economic analysis.
(i) Square matrix k = n, that is, equal number of rows and columns.

The trace of a square matrix of order n, i.e. for an n × n matrix A, is

defined as
Pnthe sum of all its diagonal elements denoted as tr(A): thus,
tr(A) = i=1 aii .

(ii) Column matrix n = 1, that is, one column. For example,

4
  
a  
 b  and 0
1
c

(iii) Row matrix k = 1, that is, one row. For example,

a b c and 2 1

(iv) Diagonal matrix k = n and aij = 0 ∀i 6= j, that is, a square matrix in

which all non-diagonal entries are 0. For example,
 
  1 0 0
a 0
and 0 2 0
0 b
0 0 3

(v) Scalar matrix is a diagonal matrix with identical diagonal elements.

Thus, k = n, aii = δ, ∀i for some real number δ, and aij = 0 ∀i 6= j.
For example,
 
  −3.3 0 0
a 0
and  0 −3.3 0 
0 a
0 0 −3.3

(vi) Identity matrix (written as In for an identity matrix of order n, or more

often, simply as I, if the order is evident from the context) is a scalar
matrix with every diagonal element equal to 1. Thus k = n, aii = 1, ∀i
and aij = 0 ∀i 6= j. For example,
 
1 0
is an identity matrix of order 2 and
 0 1

1 0 0 0
0 1 0 0
  is an identity matrix of order 4.
0 0 1 0
0 0 0 1

(vii) Null matrix is any k × n matrix with every element of the matrix equal
to 0. Thus,aij = 0, ∀i, j.
(viii) Upper-Triangular matrix aij = 0 if i > j, that is, a matrix (usually
square) in which all entries below the diagonal are 0. For example,
 
  1 2 3
a b
and 0 4 5
0 d
0 0 6

(ix) Lower-Triangular matrix aij = 0 if i < j, that is, a matrix (usually

square) in which all entries above the diagonal are 0. For example,

5
  
  1 0 0
a 0
and 2 4 0
c d
3 5 6

(x) Symmetric matrix AT = A, that is, aij = aji ∀i, j. These matrices are
necessarily square. For example,
 
  1 2 3
a b
and 2 4 5
b d
3 5 6

(xi) Idempotent matrix A square matrix B for which B · B = B, such as

B = I or
 
5 −5
4 −4

(xii) Permutation matrix A square matrix of 0s and 1s in which each row

and each column contains exactly one 1. For example,
 
0 1 0
1 0 0
0 0 1

(xiii) Nonsingular matrix A square matrix is called singular iff its determi-
nant vanishes, i.e. | A |= 0, it is non-singular otherwise. A square matrix
whose rank equals the number of its rows (or columns) is non-singular.
When such a matrix arises as a coefficient matrix in a system of linear
equations, the system has one and only one solution.

2.2 Basic Operations

2.2.1 Addition
One can add matrices of the same size, which is to say with same number of
rows and columns. The (i, j)th entry of the sum matrix is simply the sum of
the (i, j)th entries of the two matrices being added. In symbols:
     
a11 . . . a1n b11 . . . b1n a11 + b11 ... a1n + b1n
 .. .. + .. ..  =  .. .. 
 . a ij .   . b ij .   . a +b ij.  ij
ak1 ... akn bk1 ... bkn ak1 + bk1 ... akn + bkn

2.2.2 Scalar Multiplication

The product of the matrix A and the number r, denoted rA, is the matrix cre-
ated by multiplying each entry of A by r.

6
    
a11 ... a1n ra11 ... ra1n
 .. ..  =  .. .. 
r . aij .   . raij . 
ak1 ... akn rak1 ... rakn

2.2.3 Matrix Multiplication

Not all pairs of matrices can be multiplied together, and the order in which
matrices are multiplied can matter. We can define the matrix product AB iff
number of columns of A = number of rows of B.
For the matrix product to exist, A must be k × m and B must be m × n for
some positive integers k, m and n. To get the (i, j)th entry of AB, multiply the
ith row of A and the jth column of B as follows:

 
b1j

 b2j 

ai1 ai2 ··· aim ·  .  = ai1 b1j + ai2 b2j + · · · + aim bmj .
 .. 
bmj
Pm
In other words, the (i, j)th entry of the product is defined to be h=1 aih bhj .

2.2.4 Transpose
The transpose of a k×n matrix A is the n×k matrix obtained by interchanging
the rows and columns of A. This matrix is often written as AT . The (i, j)th
entry of A becomes the (j, i)th entry of AT .

2.2.5 Inverse
For any given square matrix A of order n, if there exists a matrix B of the same
order such that
A.B = B.A = I
then, such matrix B is called Inverse of matrix A. Inverse of a matrix, if it
exists, is unique. Inverse of a matrix A, if it exists, is written as A−1 .
Remark. A square matrix A is invertible iff it is non-singular, i.e. | A |= 0.

Using the matrices we can express any system of linear equations in a compact
form. Let A denote the coefficient matrix of the system:
 
a11 · · · a1n
A =  ... .. 

a .  ij
ak1 ··· akn

7
    
x1 b1
 ..   .. 
Also, let x =  .  and b =  . 
xn bk
.
The n × 1 matrix x contains variables, and the k × 1 matrix b contains the
parameters from the right-hand side of the system. So any system of equations
can be written as:
     
a11 · · · a1n x1 b1
 .. ..  ·  ..  =  .. 
 . a ij .   .  .
ak1 ··· akn xn bk
or simply as
Ax = b.
If the matrix of coefficients A is a non-singular sqaure matrix, then solution of
the above system of equations is obtained as:
x = A−1 .b.

3 Eigen Values and Eigen Vectors

Let A be a symmetric matrix of order n. Consider the system A.x = λ.x, where
x is n × 1 vector and lambda is a scalar. x = 0 is a solution for this system of
equations for any arbitrary λ, where 0 is a null vector. This is called a trivial
solution. The non-trivial solutions for this system always exist. Let (λ, x) be a
non-trivial solution. Then λ is called an eigen value or characteristic root and
x is called a corresponding eigen vector of this matrix A. There are always n
eigen values (including the repeated roots), and n eigen vectors. Any two of
eigen vectors corresponding to distinct eigen values are mutually orthogonal and
for any pair of repeated roots, a pair of mutually orthogonal eigen vectors can
be found. If a vector v is an eigen vector of matrix A, then for any non-zero
scalar c, c.v is also an eigen vector. Thus all n distinct, and pairwise orthogonal
eigen vectors can be normalized, so than kvi k = 1, ∀i. Then all eigen vectors
are of length 1 and are mutually orthogonal. Such set of vectors is also called
orthonormal. Quite clearly, these form a linearly independent set of vectors.
Let vi , i = 1, 2, . . . n be an orthonormal set of eigen vectors for any symmetric
matrix A of order n. Then viT .vi = 1, ∀i and viT .vj = 0, i 6= j.

Consider a square matrix P whose columns are the eigen vectors of a symmetric
matrix A, thus, P = (v1 v2 . . . vn ). Then it can be easily shown that P T .A.P
(call it matrix Q) is a diagonal matrix with its diagonal elements being the
corresponding eigen values, (i.e. qii = λi , ∀i, and qij = 0 for i 6= j). So,
finding eigen values and eigen vectors can be used to diagonalize a symmetric
matrix which is found to be useful in certain applications. Note that matrix P
is non-singular, hence invertible.

8
4 Quadratic Forms
A quadratic form on Rn is a real valued function of the form
X
Q(x1 , x2 , . . . , xn ) = qij xi xj , (3)
i≤j

in which each term is a monomial of degree two. Each quadratic form Q can be
represented by a square matrix A so that
Q(x) = xT · A · x, where qii = aii ∀i, and qij = (aij + aji ) for i 6= j.
Note that for any given (n×n)square matrix B, we can always find a symmetric
matrix A such that value of the quadratic form is same for any arbitrary vector
x, as follows: Let A = 21 (B + B T ). Then (a) A is a symmetric matrix; and
(b) xT · A · x = xT · B · x, ∀x ∈ Rn . Thus, for the purpose of dealing with the
quadratic forms, any square matrix can be, so to say, “symmetrized”. Thus,
each quadratic form Q can be represented by a symmetric matrix A so that

Q(x) = xT · A · x, where qii = aii ∀i, and qij = 2.aij for i < j, j = 1, 2, . . . n.
Definition 4.1. Let A be an n × n square matrix, then A is:
(a) positive definite if xT Ax > 0 ∀x 6= 0 in Rn ,
(b) positive semidefinite if xT Ax ≥ 0 ∀x 6= 0 in Rn ,

(c) negative definite if xT Ax < 0 ∀x 6= 0 in Rn ,

(d) negative semidefinite if xT Ax ≤ 0 ∀x 6= 0 in Rn ,
(e) indefinite if xT Ax > 0 for some x in Rn , and < 0 for some other x in Rn .

Definition 4.2. Let A be an n × n matrix. A k × k submatrix of A formed by

deleting n − k columns, say columns i1 , i2 , . . . , in−k and the same n − k rows,
rows r1 , r2 , . . . , rn−k , from A is called a kth order principal submatrix of A.
The determinant of a k × k principal submatrix is called a kth order principal
minor of A.

Definition 4.3. Let A be an n × n matrix. The kth order principal submatrix

of A obtained by deleting the last n − k columns from A is called the kth order
leading principal submatrix of A. Its determinant is called the kth order
leading principal minor of A. We shall denote the kth order leading principal
submatrix by Ak and the corresponding leading principal minor by |Ak |.

For the purposes of the propositions given in the remark below, if we wish
to test definiteness of any square matrix B, then consider that it has been first
“symmetrized ” as: A = 21 .(B + B T ). The test of definiteness is applied to this
symmetrized matrix A, and the results hold also for the orginal matrix B.
Remark. Let A be an n × n symmetric matrix. Then,

9
(a) A is positive definite iff all its n leading principal minors are (strictly) pos-
itive.
(b) A is negative definite iff its n leading principal minors alternate in sign as
follows:

|A1 | < 0, |A2 | > 0, |A3 | < 0, etc.

The kth order leading principal minor should have the same sign as (−1)k .
(c) If some kth order leading principal minor of A (or some pair of them) is
nonzero but does not fit either of the above two sign patterns, then A is
indefinite.
(d) A is positive semidefinite iff every leading principal minor of A is ≥ 0; A is
negative semidefinite iff every leading principal minor of odd order is ≤ 0
and every principal minor of even order is ≥ 0.

(e) Let λi , i = 1, 2, . . . n be eigen values of matrix A and let vi , i = 1, 2, . . . n be

corresponding orthonormal eigen vectors. For any non-zero vector x 6= 0,
consider y = P −1 .x, where P is matrix of eigen vectors forming its columns.
Let Q = P T .A.P be the diagonalized matrix with eigen values of A being
its diagonal elements. Now, x = P.y. Thus,
Pn
xT .A.x = y T .P T .A.P.y = y T .Q.y = i=1 λi .yi2 .

In the light of the remarks above, a symmetric matrix A is positive definite

iff all its eigen values are strictly positive and is negative definite, iff all
its eigen values are strictly negative, is positive semidefinite iff all its eigen
values are ≥ 0 with at least one eigen value positive, and it is negative
semidefinite iff all its eigen values are ≤ 0 with at least one eigen value
negative; and finally, it is indefinite, iff at least one eigen value is positive
and at least one eigen value is negative.

Prepared by:
Shobhit Mohan (with some editing inputs from Naresh Kumar Sharma)

10