A P P E N D I X

Review of Vectors and Matrices

D.1 D.1.1

VECTORS Definition of a Vector Let p1, p2, , pn be any n real numbers and P an ordered set of these real numbers that is, P = 1p1, p2, , pn2

Then P is an n-vector (or simply a vector). The ith component of P is given by pi. For example, P = 11, 22 is a two-dimensional vector. D.1.2 Addition (Subtraction) of Vectors Consider the n-vectors P = 1p1, p2, , pn2 Q = 1q1, q2, , qn2 R = 1r1, r2, , rn2 For R = P ; Q, component i is computed as ri = pi ; qi. In general, given the vectors P, Q, and S, P + Q = Q + P 1P ; Q2 ; S = P ; 1Q ; S2 P + 1 - P2 = 0 1Commutative law2 1Associative law2 1zero or null vector2
CD-145

CD-146

Appendix D

Review of Vectors and Matrices

D.1.3

Multiplication of Vectors by Scalars Given a vector P and a scalar (constant) quantity u, the new vector Q = uP = 1up1, up2, , upn2 is the scalar product of P and u. In general, given the vectors P and S and the scalars u and g, u1P + S2 = uP + uS 1Distributive law2 u1gP2 = 1ug2P 1Associative law2

D.1.4

Linearly Independent Vectors The vectors P1, P2, , Pn are linearly independent if, and only if
n j=1

a ujPj = 0 Q uj = 0, j = 1, 2, , n
n j=1

If a ujPj = 0, for some uj Z 0 P1 = 11, 22, P2 = 12, 42 u1P1 + u2P2 = 0 D.2 D.2.1 MATRICES Definition of a Matrix A matrix is a rectangular array of elements. The element aij of the matrix A occupies the ith row and jth column of the array. A matrix with m rows and n columns is said to be of size (or order) m * n. For example, the following matrix is of size 14 * 32. a11 a A = 21 a31 a41 D.2.2 Types of Matrices 1. A square matrix has m = n. 2. An identity matrix is a square matrix in which all the main diagonal elements equal 1 and all the off-diagonal elements equal zero. For example, a 13 * 32 identity matrix is given by 1 0 0 I3 = 0 1 0 0 0 1 a12 a22 a32 a42 a13 a23 = 7 aij 7 4 * 3 a33 a43

then the vectors are linearly dependent. For example, the vectors

are linearly dependent because for u1 = 2 and u2 = - 1,

D.2

Matrices

CD-147

3. A row vector is a matrix with one row and n columns. 4. A column vector is a matrix with m rows and one column. 5. The matrix AT is the transpose of A if the element aij in A is equal to element aji in AT for all i and j. For example, 1 A = 2 3 4 1 5 Q AT = a 4 6 2 5 3 b 6

6. A matrix B = 0 is a zero matrix if every element of B is zero. 7. Two matrices A = 7 aij 7 and B = 7 bij 7 are equal if, and only if, they have the same size and aij = bij for all i and j. D.2.3 Matrix Arithmetic Operations In matrices only addition (subtraction) and multiplication are defined. Division, though not defined, is replaced by inversion (see Section D.2.6). Addition (Subtraction) of Matrices. Two matrices A = 7 aij 7 and B = 7 bij 7 can be added together if they are of the same size 1m * n2. The sum D = A + B is obtained by adding the corresponding elements. Thus, 7 dij 7 m * n = 7 aij + bij 7 m * n

If the matrices A, B, and C have the same size, then A + B = B + A A ; 1B ; C2 = 1A ; B2 ; C 1A ; B2T = AT ; BT 1Commutative law2 1Associative law2

Product of Matrices. The product D = AB of two matrices, A = 7 aij 7 and B = 7 bij 7 , is defined if, and only if, the number of columns of A equals the number of rows of B. If A is of size 1m * r2 and B is of size 1r * n2, then D must be of size m * n, where m and n are arbitrary positive integer values. In this case, the elements of D are computed as
r

dij = a aikbkj, for all i and j

k=1

For example, given A = a we have D = a 11 * 5 + 3 * 6211 * 7 + 3 * 8211 * 9 + 3 * 02 1 3 5 7 9 b b = a ba 12 * 5 + 4 * 6212 * 7 + 4 * 8212 * 9 + 4 * 02 2 4 6 8 0 23 31 9 b = a 34 46 18 1 2 5 3 b, B = a 6 4 7 8 9 b 0

In general, AB Z BA even if BA is defined.

CD-148

Appendix D

Review of Vectors and Matrices

Matrix multiplication follows these general properties: I mA = AI n = A, I m and I n are identity matrices 1AB2C = A1BC2 C1A ; B2 = CA ; CB 1A ; B2C = AC ; BC a1AB2 = 1aA2B = A1aB2, a is a scalar

Multiplication of Partitioned Matrices. Let A be an 1m * r2 matrix and B an 1r * n2matrix. Assume that A and B are partitioned as follows: A = a B11 B12 A 11 A 12 A 13 b , B = B21 B22 A 21 A 22 A 23 B32 B32

The partitioning assumes that the number of columns of A ij is equal to the number of rows of Bij for all i and j. Then A * B = a For example, 4 + 2 + 24 1 3 4 112142 + 12 32a b 30 8 = = 44 5 1 = 1 0 5 1 4 40 6 8 61 a b142 + a ba b a b + a b 2 5 0 8 8 53 A 11B11 + A 12B21 + A 13B31 A 11B12 + A 12B22 + A 13B32 b A 21B11 + A 22B21 + A 23B31 A 21B12 + A 22B22 + A 23B32

12 1 0 25

D.2.4

Determinant of a Square Matrix Consider the n-square matrix a11 a A = 21 o an1 Next, define the product Pj1j2 jn = a1j1a2j2 anjn such that each column and each row of A is represented exactly once among the subscripts of j1, j2, , and jn. Next, define H j1j2 jn = e 1, j1 j2 jn even permutation 0, j1 j2 jn odd permutation a12 a22 o an2 o a1n a2n o ann

D.2

Matrices

CD-149

Let r represents the summation over all n! permutations, then the determinant of A, det A or A , is computed as a H j1j2 jnPj1j2 jn
r

As an illustration, consider a11 A = a21 a31 Then A = a111a22 a33 - a23 a322 - a121a21 a33 - a31 a232 + a131a21 a32 - a22 a312 The properties of a determinant are: 1. The value of the determinant is zero if every element of a row or a column is zero. 2. A = AT . 3. If B is obtained from A by interchanging any two rows or any two columns, then B = - A. 4. If two rows (or two columns) of A are multiples of one another, then A = 0. 5. The value of A remains the same if scalar a times a column (row) vector is added to another column (row) vector. 6. If every element of a column or a row of a determinant is multiplied by a scalar a, the value of the determinant is multiplied by a. 7. If A and B are two n-square matrices, then AB = A B Definition of the Minor of a Determinant. The minor Mij of the element aij in the determinant A is obtained from the matrix A by striking out the ith row and jth column of B. For example, for a11 A = a21 a31 M11 = ` a22 a32 a12 a22 a32 a13 a23 a33 a13 `, a33 a12 a22 a32 a13 a23 a33

a23 a ` , M22 = ` 11 a33 a31

Definition of the Adjoint Matrix. Let A ij = 1 - 12i + jMij be defined as the cofactor of the element aij of the square matrix B. Then, the adjoint matrix of A is the transpose of 7 A ij 7 , and is defined as: adj A = 7 A ij 7 T A 11 A = 12 o A 1n A 21 A 22 o A 2n o A n1 A n2 o A nn

CD-150

Appendix D

Review of Vectors and Matrices

For example, if 1 A = 2 3 2 3 3 3 2 4

then, A 11 = 1 - 12213 * 4 - 2 * 32 = 6, A 12 = 1 - 12312 * 4 - 3 * 22 = - 2, , or 6 adj A = - 2 -3 D.2.5 Nonsingular Matrix A matrix is of a rank r if the largest square array in the matrix having a non-zero determinant is of size r. A square matrix with a non-zero determinant is called a full-rank or nonsingular matrix. For example, consider 1 A = 2 3 A is a singular matrix because A = 1 * 121 - 202 - 2 * 114 - 122 + 3 * 110 - 92 = 0 But A has a rank r = 2 because a D.2.6 Inverse of a Nonsingular Matrix If B and C are two n-square matrices such that BC = CB = I, then B is called the inverse of C and C the inverse of B. The common notation for the inverse is B-1 and C -1. Theorem. is unique. If BC = I and B is nonsingular, then C = B-1, which means that the inverse 1 2 2 b = -1 Z 0 3 2 3 5 3 4 7 1 -5 3 -5 4 -1

Proof. By assumption, BC = I then B-1BC = B-1I or IC = B-1

D.2

Matrices

CD-151

or C = B- 1 Two important results can be proved for nonsingular matrices. 1. If A and B are nonsingular n-squre matrices, then 1AB2-1 = B-1A-1 2. If A is nonsingular, then AB = AC implies that B = C. Matrix inversion is used to solve n linearly independent equations. Consider a11 a 21 o an1 a12 a22 o an2 o a1n x1 b1 a2n x2 b = 2 o o o ann xn bn

where xi represents the unknowns and aij and bi are constants. These n equations can be written in matrix form as AX = b Because the equations are independent, A must be nonsingular. Thus A-1AX = A-1b or X = A-1b D.2.7 Methods of Computing the Inverse of a Matrix1 Adjoint Matrix Method. Given A, a nonsingular matrix of size n, A 11 1 1 A = adj A = 12 o A A A 1n A 21 A 22 o A 2n o A n1 A n2 o A nn

A-1

For example, for 1 A = 2 3 6 adj A = - 2 -3

1

2 3 3

3 2 4 -5 4, A = -7 -1

1 -5 3

TORAs inverse module is based on LU decomposition method. See Press and Associates (1986)

CD-152

Appendix D

Review of Vectors and Matrices

Hence A-1 6 1 = -2 -7 -3 1 -5 3 -5 -6 7 2 4 = 7 3 -1 7 -1 7
5 7 -3 7 5 7 -4 7 1 7

Row Operations (Gauss-Jordan) Method. Consider the partitioned matrix 1A I2, where A is nonsingular. Premultiplying by A-1, we obtain 1A-1A A-1I2 = 1I A-12 Thus, applying a specific sequence of row transformations, A is changed to I and I is changed to A-1. To illustrate the procedure, consider the system of equations: 1 2 3 2 3 3 3 x1 3 2 x2 = 4 4 x3 5

The solution of X and the inverse of the basis matrix can be obtained directly by considering A-11A I b2 = 1I A-1 A-1b2

The following iterations detail the transformation operation: Iteration 0 1 2 3 Iteration 1 1 0 0 Iteration 2 1 0 0 Iteration 3 1 0 0 0 1 0 0 -7 2 03 7 1 3
7 6

2 3 3

3 1 230 4 0

0 1 0

0 3 0 3 4 1 5

2 -1 -3

3 1 -4 3 -2 -5 -3

0 1 0

0 3 0 3 -2 1 -4

0 1 0

-5 -3 43 2 7 3

2 -1 -3

0 -1 0 3 2 1 2

-1 7
5 7 -3 7

5 7 -4 7 1 7

3 7 36 7 2 7

D.2

Matrices

CD-153

6 2 This gives x1 = 3 7 , x2 = 7 , and x3 = 7 . The inverse of A is given by the right-hand-side matrix, which is the same as obtained by the method of adjoint matrix.

Product Form of the Inverse. Suppose that two nonsingular matrices, B and Bnext, dif-1 fer exactly in one column. Further, assume that B-1 is given. Then the inverse Bnext can be computed using the formula
-1 Bnext = EB-1

The matrix E is computed in the following manner. If the column vector Pj in B is replaced with the column vector Pr to produce Bnext, then E is constructed as an m-identity matrix with its rth column replaced by - 1B-1Pj21 - 1B-1Pj22 1 o j = , 1B-1Pj2r Z 0 +1 ; rth place 1B-1Pj2r o - 1B-1Pj2m

-1 If 1B-1Pj2r = 0, then Bnext does not exist. -1 The validity of the formula Bnext is proved as follows. Define F as an m-identity matrix whose rth column is replaced by B-1Pjthat is,

F = 1e1, er - 1, B-1Pj, er + 1, , em2 Because Bnext differs from B only in that its rth column is replaced with Pj, then Bnext = BF Thus,
-1 = 1BF2-1 = F -1B-1 Bnext

The formula follows by setting E = F -1. The product form can be used to invert any nonsingular matrix, B, in the follow-1 . Next, construct B1 as an identity matrix, except ing manner. Start with B0 = I = B0 that the first column is replaced with the first column in B. Then
-1 -1 B1 = E1B0 = E1I = E1

In general, if we construct Bi as an identity matrix with its first i columns replaced with the first i columns of B, then
1 -1 = EiE i - 1 E1 Bi-1 = EiBi-1 = E iE i - 1Bi - 2 =

This means that for the original matrix B, B-1 = EnEn - 1 E1

CD-154

Appendix D

Review of Vectors and Matrices

The following example illustrates the application of the product form of the inverse. Consider 2 B = 0 4 Iteration 0 B0 = Iteration 1 2 B1 = 0 4 0 1 0 0 0 1
-1 B0

1 2 0

0 0 1 0 1 0 0 0 1

1 = 0 0

2 ;r = 1 -1 B0 P1 = P1 = 0 4 +1 2 E1 =
-0 2 -4 2
1 2

0 1 0 0 1 0

0 0 1 0 0 1

-1 B1

= 0 -2

Iteration 2 2 B2 = 0 4
-1 B1 P2 = 0 -2 1 2

1 2 0 0 1 0

0 0 = B 1
1 0 1 2 0 2 = 2 ; r = 2 1 0 -2

1 E2 = 0 0 1 B
-1

1 2

2 +1 2

- -22 -1 4
1 2

1 0 = 0 0 1 0 1
1 2

-1 4
1 2

0 0 1
1 2

1 0 1 0 0 1

-1 B2

-1 E2B1

= 0 0

0 0 -2

-1 4
1 2

0 0 1

0 = 0 -2

D.2

Matrices

CD-155

Partitioned Matrix Method. Suppose that the two n-nonsingular matrices A and B be partitioned as shown as follows: A 11 1p * p2 A = A 21 1q * p2 B11 1p * p2 B = B21 1q * p2 A 12 1p * q2 , A 11 nonsingular A 22 1q * q2 B12 1p * q2 B22 1q * q2

If B is the inverse of A, then from AB = I n, we have A 11B11 + A 12B21 = I p A 11B12 + B12A 22 = 0 Also, from BA = I n, we get B21A 11 + B22A 21 = 0 B21A 12 + B22A 22 = I q
-1 Because A 11 is nonsingular, A 11 exists. Solving for B11, B12, B21, and B22, we get

B11 B12 B21 B22 where

= = = =

-1 -1 -1 A 11 + 1A 11 A 122D -11A 21 A 11 2 -1 -1 - 1A 11 A 122D -1 - D -11A 21 A 11 2 -1 D

-1 A 122 D = A 22 - A 211A 11

To illustrate the use of these formulas, partition the matrix 12 A = 2 3 33 such that 2 3 A 11 = 112, A 12 = 12, 32, A 21 = a b , A 22 = a 3 3 2 b 4 3 2 4

CD-156

Appendix D

Review of Vectors and Matrices

-1 In this case, A 11 = 1 and

D = a

3 3

2 2 -1 b - a b11212, 32 = a 4 3 -3 -5 3 4 b = a 7 -1 -3 7
5

-4 b -5
1b 7

D -1 = - 1 7a Thus, B11 =

-4 7

1 A -6 7 B , B12 = A - 7 2 5 7 -3 7

5 7

B
1b 7

B12 = a 7 3 b , B22 = a
7

-4 7

which directly give B = A-1 D.2.8 Matrix Manipulations Using Excel Excel provides facilities for automatically performing the following matrix manipulations: 1. 2. 3. 4. Transpose. Multiplication. Inverse of a nonsingular matrix. Determinant value of a nonsingular matrix.

Figure D.1 provides illustrative examples (file excelMatManip.xls). In Example 1 (Transpose), A is a 2 * 3 matrix whose elements are entered in the range A4:C5. Transpose(A), or AT, appears in the user-specified range E4:F6. The steps for obtaining the output in the selected range are: 1. 2. 3. 4. Enter the formula = TRANSPOSE1A4 : C52 in cell E4. Select (highlight) the output cells E4:F6. Press F2. Press CTRL + SHIFT + ENTER.

In Example 2, the elements of the input matrices A and B are entered in the respective ranges A10:C13 and A16:A18. The output matrix is in the (user-selected) range E10:E13. Next enter the formula = MMULT(A10:C13,A16:A18) in cell E10 and follow steps 2 through 4 exactly as in Example 1 (replacing E4:F6 with E10:E13). Notice that MMULT(A16:A18,A10:C13) is undefined. In Example 3, the inverse of the matrix in the range A22:C24 is assigned to the range E22:G24 by entering the formula = MINVERSE1A22 : C242 in cell E22, then following steps 2, 3, and 4 as in Example 1. Finally, in Example 4, the determinant of the matrix in the range A28:C30 is obtained by entering the formula = MDETERM1A28 : C302 in the user-selected cell E28.

D.3

Quadratic Forms

CD-157

FIGURE D.1 Matrix manipulations using Excel (file excelMatManip.xls)

D.3

QUADRATIC FORMS Given and a11 a A = 21 o an1 the function Q1X2 = X TAX = a a aijxixj
i=1 j=1 n n

X = 1x1, x2, , xn2T a12 a22 o an2 o a1n a2n o ann

CD-158

Appendix D

Review of Vectors and Matrices

1a ij + a ji2 2

is called a quadratic form. The matrix A can always be assumed symmetric because each element of every pair of coefficients aij and aij 1i Z j2 can be replaced by without changing the value of Q(X). As an illustration, the quadratic form 1 Q1X2 = 1x1, x2, x32 2 3 is the same as 1 Q1X2 = 1x1, x2, x32 1 2 1 7 3 2 x1 3 x2 2 x3 0 7 0 1 x1 6 x2 2 x3

Note that A is symmetric in the second case. We will assume henceforth that A is always symmetric. The quadratic form is said to be 1. Positive-definite if Q1X2 7 0 for all X Z 0. 2. Positive-semidefinite if Q1X2 0 for all X, and there exists X Z 0 such that Q1X2 = 0. 3. Negative-definite if - Q1X2 is positive-definite. 4. Negative-semidefinite if - Q1X2 is positive-semidefinite. 5. Indefinite in all other cases. It can be proved that the necessary and sufficient conditions for the realization of the preceding cases are 1. Q(X) is positive-definite (-semidefinite) if the values of the principal minor determinants of A are positive (nonnegative). In this case, A is said to be positive definite (semidefinite). 2. Q(X) is negative-definite if the value of the kth principal minor determinant of A has the sign of 1 - 12k, k = 1, 2, , n. In this case, A is called negative-definite. 3. Q(X) is negative-semidefinite if the kth principal minor determinant of A either is zero or has the sign of 1 - 12k, k = 1, 2, , n.

The kth principal minor determinant of A n * n is defined by

a11 4 a21 o a k1

a12 a22 o ak2

a1k a2k 4 , k = 1, 2, , n o akk

Problems

CD-159

D.4

CONVEX AND CONCAVE FUNCTIONS A function f(X) is said to be strictly convex if, for any two distinct points X 1 and X 2, where 0 6 l 6 1. Conversely, a function f(X) is strictly concave if - f1X2 is strictly convex. A special case of the convex (concave) function is the quadratic form (see Section D.3) f1X2 = CX + X T AX where C is a constant vector and A is a symmetric matrix. It can be proved that f(X) is strictly convex if A is positive-definite and f(X) is strictly concave if A is negative definite. f1lX 1 + 11 - l2X 22 6 lf1X 12 + 11 - l2f1X 22

PROBLEMS
1. Show that the following vectors are linearly dependent. 1 -2 1 (a) - 2 4 - 2 3 -2 -1 2 4 -3 -6 (b) 4 8 5 10 2. Given 1 A = 2 3 4 5 7 9 7 -8, B = 9 2 3 -1 4 6 2 8 10

find (a) A + 7B (b) 2A - 3B (c) 1A + 7B2T

3. In Problem 2, show that AB Z BA 4. Consider the partitioned matrices 1 5 2 -6 A = 3 7 4 9 7 2 9 , B = 1 2 3 1 3 -4 2 6 1 0 5 7 9

Find AB using partitioned matrix manipulation. 5. In Problem 2, find A-1 and B-1 using the following: (a) Adjoint matrix method (b) Row operations method (c) Product form of the inverse (d) Partitioned matrix method

CD-160

Appendix D 6. Consider

Review of Vectors and Matrices

2 B = 0 4

1 2 0

5 2 4 1 -1 1, B = 2 -1 5

-5 8

1 4 1 2

-3 8 -1 4
1 2

Suppose that the third vector P3 is replaced with the V3 = P1 + 2P2. This means that the resulting matrix is singular. Show how the product form of the inverse discovers the singularity of the matrix. 7. Use the product form of the inverse to verify whether each of the following equations has a unique solution, no solution, or an infinity of solutions. (a) x1 + 2x2 = 3 x1 + 4x2 = 2 (b) x1 + 2x2 = 5 - x1 - 2x2 = - 5 (c) x1 + 7x2 + x3 = 5 4x1 + x2 + 3x3 = 8 x1 + 3x2 - 2x3 = 3 8. Verify the formulas given in Section B.2.7 for obtaining the inverse of a partitioned matrix. 9. Find the inverse of A = a 1 H G b , B nonsingular B

10. Show that the following quadratic form is negative definite.

2 Q1x1, x22 = 6x1 + 3x2 - 4x1x2 - 2x2 1 - 3x 2 -

27 4

11. Show that the following quadratic form is positive definite.

2 2 Q1x1, x2, x32 = 2x2 1 + 2x 2 + 3x 3 + 2x1x2 + 2x2x3

12. Show that the function f1x2 = ex is strictly convex over all real values of x. 13. Show that the quadratic function f1x1, x2, x32 = 5x12 + 5x2 2 + 4x3 2 + 4x1x2 + 2x2x3 is strictly convex. 14. In Problem 13, show that - f1x1, x2, x32 is strictly concave.

SELECTED REFERENCES
Hadley, G., Matrix Algebra, Addison-Wesley, Reading, MA, 1961. Hohn, F., Elementary Matrix Algebra, 2nd ed., Macmillan, New York, 1964. Press, W., B. Flannery, B. Teukolsky, and W. Vetterling, Numerical Recipes: The Art of Scientific Computing, Cambridge University Press, Cambridge, England, 1986.