Lecture 2

Orthogonal Vectors and Matrices, Norms

MIT 18.335J / 6.337J

Introduction to Numerical Methods

Per-Olof Persson
September 12, 2006

Transpose and Adjoint

For real A, the transpose of A is obtained by interchanging rows/columns

a11 a12 a a a a21 a22 = AT = 11 21 31 A= a12 a22 a32 a31 a32 The adjoint or hermitian conjugate also takes complex conjugate a11 a12 a a a a21 a22 = A = 11 21 31 A=
a12 a22 a32 a31 a32

If real A = AT , then A is symmetric. If A = A , then A is hermitian.

2

Inner Product

Inner product of two column vectors x, y Cm

m

x y =
i=1

x i yi

Euclidean length of x x = Angle between x, y

m 1/2

x x =
i=1

|xi |2

x y cos = xy

In MATLAB

Quantity Transpose AT Adjoint A Inner product x y

MATLAB Syntax

Comment Transpose only Transpose + complex conjugate or

A. A x*y dot(x,y)

* assumes column vectors * assumes column vector

Length x

sqrt(x*x) or norm(x)

Orthogonal Vectors

The vectors x, y Rm are orthogonal if x y = 0 The sets of vectors X, Y are orthogonal if

every x

X is orthogonal to every y Y

A set of (nonzero) vectors S is orthogonal if

vectors pairwise orthogonal, i.e., for x, y and orthonormal if, in addition, every x

S, x = y x y = 0

S has x = 1
5

Orthogonal and Unitary Matrices

A square matrix Q Cmm is unitary (orthogonal in real case), if Q = Q1 For unitary Q

Q Q = I , or qi qj = ij

Interpretation of unitary-times-vector product: x = Q b = solution to Qx = b = the vector of coefcients of the expansion of b

in the basis of columns of Q

Preservation of Geometry Structure

Inner product is preserved under multiplication by unitary Q (Qx) (Qy) = x Q Qy = x y Therefore, lengths of vectors and angles between vectors are preserved A real orthogonal Q is either a rigid rotation or reection
Rotation Reection

Qu Qv v
Qv v

u
7

Qu

Vector Norms

A norm is a function : Cm R satisfying (2) x + y x + y (3) x = || x Example: The Euclidean length function x2 = x x 2 is a special case of the p-norms
m 1/p

(1) x 0, and x = 0 only if x = 0

xp =

i=1

|xi |p
8

(1 p < )

Examples of Vector Norms

x1 = x2 =

i=1

|xi |
1/2

i=1 1im

|xi |

x x

x = max |xi |
m 1/2

xW = W x2 =

i=1

|wi xi |2

Verication of Norm Conditions

Show that x1 =
(1)
m i=1

|xi | is a norm

x1 = |x1 | + |x2 | + + |xm | 0, equality only if x = 0

(2)
m m

x + y1 = =

i=1 m

|xi + yi |
m

i=1

|xi | + |yi |

i=1

|xi | +
m

i=1

|yi | = x1 + y1
m

(3)
m

x1 =

i=1

|xi | =

i=1

|| |xi | = ||

i=1

|xi | = || x1

10

Induced Matrix Norms

For a matrix A Cmn , the induced matrix norm is A(m,n) Ax(m) = = sup xCn x(n)
x=0 xCn x (n) =1

sup Ax(m)

where (n) and (m) are given vector norms

The maximum stretching by A

11

Examples of Matrix Norms

A1 = max

1jn

i=1
n

|aij | |aij |
1/2

maximum column sum

A = max
m

1im

maximum row sum

j=1
n

AF = A2 =

i=1 j=1

|aij |2

The Frobenius norm More later

max (A A)

12

Properties of Matrix Norms

Bound on Matrix Product - Induced norms and Frobenius norm satisfy AB AB

but some matrix norms do not!

Invariance under Unitary Multiplication - For A Cmn and unitary Q Cmm , we have QA2 = A2 ,
Proof. Since Qx2

QAF = AF

follows from the denition of induced norm. For the Frobenius norm,

= x2 (inner product is preserved), the rst result tr((QA) (QA)) = tr(A A) = AF

13

QAF = =

tr(A Q QA)

In MATLAB

Quantity MATLAB Syntax

x1 x2 xp x A1 A2 A AF

sum(abs(x)) or norm(x,1) sqrt(x*x) or norm(x) sum(abs(x). p). (1/p) or norm(x,p) max(abs(x)) or norm(x,inf) max(sum(abs(A),1)) or norm(A,1) norm(A) max(sum(abs(A),2)) or norm(A,inf) sqrt(A(:)*A(:)) or norm(A,fro)

14