MATRIX ALGEBRA

21

Figure 2.2 Multiplication of two complex quantities

2.7.3 Example
The addition and multiplication of two quantities, each with its own magnitude
and phase, are common operations in signal processing and system analysis.
The rectangular representation is more convenient for addition (Equation 2.8)
whereas the exponential notation facilitates multiplication (Equation 2.12). The
multiplication of two complex quantities is demonstrated in Figure 2.2 using
complex, polar, and exponential forms. Note the efficient implementation using
exponentials.
2.2

MATRIX ALGEBRA

A matrix is an arrangement of numbers in columns and rows. A review of

fundamental matrix operations follows.

2.2.7

Definitions and Fundamental Operations

The following notation is used to designate the matrix a by its elements:

22

MATHEMATICAL CONCEPTS

where the index i refers to the row number and varies from 1 to M, and k indicates
column number and varies from 1 to N, where M and N are integers. Sometimes
it is more convenient to vary i from 0 to M 1, and k from 0 to N 1. For
example, this is the case when the first entry refers to zero time or zero frequency.
A matrix a is
square if M = N
real when all its elements are real numbers
complex if one or more of its elements are complex numbers
nonnegative if all aj k > 0
positive if all ^ k > 0
Negative and nonpositive matrices are similarly defined.
The trace of a square matrix is the sum of the elements in the main diagonal,
a; j. The identity matrix I is a square matrix where all its elements are zeros,
except for the elements in the main diagonal, which are ones: Ii>k = 1.0 if i = k,
else Ij k = 0. Typical operations with matrices include the following:
addition:
subtraction:
scalar multiplication:
matrix multiplication:
Note that matrix multiplication is a summation of binary products; this type of
expression is frequently encountered in signal processing (Chapter 4).
The transpose aT of the matrix a is obtained by switching columns and rows:

A square matrix a is symmetric if it is identical to its transpose (aT = a or

aj k = ak j). The matrices (aT a) and (a aT) are square and symmetric for any
matrix a.
The Hermitian adjoint aH of a matrix is the transpose of the complex conjugates
of the individual elements. For example, if an element in a is aj k = b + j c, the
corresponding element in the Hermitian adjoint is a k ; = b j c. A square matrix
is Hermitian if it is identical to its Hermitian adjoint; the real symmetric matrix
is a special case.

MATRIX ALGEBRA

The matrix a

23

is the inverse of the square matrix a if and only if

A matrix is said to be orthogonal if aT = a , then

Finally, a matrix is called unitary if the Hermitian adjoint is equal to the inverse,
a.*1^!"1.
The determinant of the square matrix a denoted as |a| is the number whose
computation can be defined in recursive form as

where the minor is the submatrix obtained by suppressing row i and column k. The
determinant of a single element is the value of the element itself. If the determinant
of the matrix is zero, the matrix is singular and noninvertible. Conversely, if
|a| 7^ 0 the matrix is invertible.
The following relations hold:

24

MATHEMATICAL CONCEPTS

A row in a matrix is linearly independent if it cannot be computed as a linear

combination of the other rows. The same applies to the columns in the matrix.
The rank of a matrix r[a] is the number of linearly independent rows or columns
in a. If r[a] = S, then there is a square submatrix size S x S whose determinant is
nonzero.
2.2.2

Matrices as Transformations

So far, matrices have been described as isolated rectangular arrays of real or

complex numbers. Consider now the matrix a as an operator that transforms an
"input" vector \ into an "output" vector yj

Computationally, the transformation y = a x is a linear combination of the

columns of a according to the entries in x.
If matrix a NxN is noninvertible, there will be vectors x that are normal to
the columns of a and map to y = a x = 0; those vectors are the subspace of x
called the null space (Figure 2.3). On the other hand, not all the space of y_ is
reachable from x.; the range of a is the subset of the space of y_ reachable by
the transformation (Equation 2.38). The fact that a is noninvertible indicates that
some of the columns in a are linearly dependent, and they will not contribute
to the dimensionality of the range. Hence, the dimension of the range is the
rank r[a]:

It follows from these definitions that the sum of the dimensions of the null
space and the range is N:

If a is invertible, y = a x = Q only if x = 0, and the dimension of the null

space is zero. For a simple visualization of these concepts, consider the transformation matrix _ = [(1,0,0), (0,1,0), (0,0,0)], with rank r[] = 2. All vectors
x = (0,0, x3) map to y = a x = (0,0,0); therefore, they are in the null space of
the transformation. On the other hand, only vectors y = (y t , y 2 ,0) are reachable
by the transformation; this is the range.

MATRIX ALGEBRA

25

Figure 2.3 Definition of null space and range of a transformation a [N x N]. The dimension of the range is equal to the rank of a. The dimension of the range plus the dimension
of the null space is equal to N

The transformation matrix a is positive definite if

for all nonzero vectors x. If XT a x > 0 then a is positive semidefinite. Typically,

the matrix a is positive definite when the elements along the main diagonal of a
are positive and when they are also the largest elements in the matrix.
2.2.3

Eigenvalues and Eigenvectors

If a is a square matrix, and y_ is obtained either as matrix multiplication a x or

as scalar multiplication Ax,

26

MATHEMATICAL CONCEPTS

then x is an eigenvector of a and A is its corresponding eigenvalue. The eigenvalues of a are obtained by solving the polynomial

where I is the identity matrix. For each eigenvalue A p , the corresponding eigenvector Xp is computed by replacing Ap in Equation 2.42,

where 0 is an array of zeros. The eigenvectors corresponding to distinct eigenvalues of a Hermitian or symmetric matrix are orthogonal vectors; that is, the dot
product is equal to zero. The eigenvalues of a Hermitian or symmetric matrix
are real. The eigenvalues of a symmetric, positive-definite matrix are real and
positive A; > 0, and the matrix is invertible. Last, for any given matrix a [M x N],
the eigenvalues of (aT a) and (a aT) are nonnegative and their nonzero values
are equal.

2.2.4

Matrix Decomposition

The solution of systems of equations, including matrix inversion, can be more

effectively implemented by decomposing the matrix into factors.

Eigen Decomposition
A invertible square matrix a with distinct eigenvalues can be expressed as the
multiplication of three matrices

where the columns of X are the eigenvectors of a, and A is a diagonal matrix

that contains the eigenvalues of a. The order of the eigenvectors in matrix X
must be the same as the order of the eigenvalues in matrix A. The inverse of a
is (Equation 2.31)

MATRIX ALGEBRA

The elements in the diagonal matrix A

If a is symmetric, then X"1 = XT.

27

are the inverse of the eigenvalues 1/Aj.

Singular Value Decomposition (SVD)

Any real matrix a [MxN] with M > N, and rank r < N, can be expressed as

where
U [M x M]

Orthogonal matrix
Its columns are eigenvectors of a aT (in order as in A)
Vectors u t . . . ur span the range of a

A [M x N]

Diagonal matrix A; k = 0 for i ^ k

Values A; j = A; are the singular values in descending order
A; are the nonnegative square root of eigenvalues of a aT
or a7 a
Singular values Aj > ... Ar > 0 and singular values
Ar+1 = ... =A N = 0

V [N x N]

Orthogonal matrix
Its columns are eigenvectors of aT a (in same order as A in A)
The null space of a is spanned by vectors v r+1 ... VN

For a real, the resulting three matrices are also real. The SVD is generalized to
complex matrices using the Hermitian instead of the transpose. The method is
equally applicable when the size of the matrix is M < N, with proper changes in
indexes.

Other Decompositions
Two efficient algorithms are used to solve systems of equations that involve
square matrices a [N x N]. The LU decomposition converts a into the multiplication of a lower triangular matrix L (L; j = 0 if i < j) and an upper triangular matrix
U (Uj j = 0 if i > j), such that a = L U. Furthermore, if the matrix a is symmetric
and positive definite, the Cholesky decomposition results in a = UT U, where U
is upper triangular.

28

2.3

MATHEMATICAL CONCEPTS

DERIVATIVES - CONSTRAINED OPTIMIZATION

A linear function of multiple variables f = al-xl+&2-x2 +... can be expressed

in matrix form as

(Note that for a family of functions f!...f M , the array a becomes matrix a.)
Likewise, the partial derivatives 6f/9x; are organized into an array

The following set of equations facilitates the derivation of explicit solutions in

optimization problems that are captured in matrix form (Chapter 9):
(as shown above)

for a symmetric
In each case, the function is written in explicit form, partial derivatives are
computed, and the result is once again expressed in matrix form.
Given M-measurements yj that depend on N-parameters xk, the partial derivative 3yj/9xk indicates the sensitivity of the i-th measurement to the k-th parameter.
The Jacobian matrix is the arrangement of the M x N partial derivatives in matrix
form. The Jacobian matrix is useful to identify extrema and to guide optimization
algorithms.
The extremum of a function is tested for minimum or maximum with the
Hessian matrix Hes formed with the second derivatives of f(x):