EE547: Fall 2018

Lecture 5: Normed Spaces

Lecturer: L.J. Ratliff

Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They may be distributed outside this class only with the permission of the Instructor.

5.1 Normed Linear Spaces

Remark 1.1 (Why? ) We need to be able to measure distances between things and hence, we need to
formally introduce a metric onto the linear spaces we consider. This is going to be important for classical
stability notions.

Intuitively the norm of a vector is a measure of its length. Of course, since elements of a linear space can be
vectors in Rn , matrices, functions, etc. Hence, the concept of norm must be defined in general terms.
Let the field F be R or C.

Definition 1.1 (Normed Linear Space.) A linear space (V, F ) is said to be a normed linear space if
there exists a map k · k : V → R+ satisfying
1. the ‘triangle inequality’ holds:

kv1 + v2 k ≤ kv1 k + kv2 k, ∀ v1 , v2 ∈ V

2. length under scalar multiplication satisfies

kαvk = |α|kvk, ∀ v ∈ V, α ∈ F

3. the only element with zero length is the zero vector:

kvk = 0 ⇐⇒ v = 0

The expression kvk is read ’the norm of v’ and the function k · k is said to be the norm on V . It is often
convenient to label the space (V, F, k · k).

Example 1.1
1. `p –norms:
n
!1/p
X
p
kxkp = |xi |
i=1

• `1 –norm (sum norm)

n
X
kxk1 = |xi |
i=1

• `2 –norm (Euclidean norm)

n
!1/2
X
2
kxk2 = |xi |
i=1

5-1
5-2 Lecture 5: Normed Spaces

• `∞ –norm (sup norm)

kxk∞ = max |xi |
i
n
2. Function norms: C([t0 , t1 ], R )
• L1 –norm Z t1
kf k1 = kf (t)k dt
t0

where kf (t)k is any of the `p norms from above

• L2 –norm
Z t1 1/2
2
kf k2 = kf (t)k dt
t0
n
Note. since f maps into R , this is equivalent to each component function of f being in C([t0 , t1 ], F )
and since (as we will see) all norms on Rn are equivalent, it does not matter which norm we choose on
Rn .

Remark 1.2 (What do we do with norms? ) A typical engineering use of norms is in deciding
whether or not an iterative technique converges or not—this will be very important in our analysis of
stability.

Definition 1.2 (Convergence.) Given a normed linear space (V, F, k · k) and a sequence of vectors
{vi }∞ ∞
i=1 ⊂ V , we say the sequence {vi }i=1 converges to the vector v ∈ V iff the sequence of non-negative
real numbers kvi − vk tends to zero as i → ∞.

We denote convergence by vi → v or limi→∞ vi = v.

5.1.1 Equivalence of Norms

Two norms k · ka and k · kb on (V, F ) are said to be equivalent if ∃ m` , mu ∈ R+ such that for all v ∈ V

m` kvka ≤ kvkb ≤ mu kvka

It is crucial to note that the same m` and mu must work for all v ∈ V .

Fact 1.1 Consider (V, F ) and two equivalent norms k · ka and k · kb . Then, convergence is equivalent—i.e.
i→∞ i→∞
vi −→ v in k · ka ⇐⇒ vi −→ v in k · kb

Example 1.2
1. √
kxk2 ≤ kxk1 ≤ nkxk2
2. √
kxk∞ ≤ kxk2 ≤ nkxk∞
3.
kxk∞ ≤ kxk1 ≤ nkxk∞

Many other things are equivalent too:

• Cauchy nature of sequences is equivalent—i.e.

(vi )∞ ∞
i=1 Cauchy in k · ka ⇐⇒ (vi )i=1 Cauchy in k · kb
Lecture 5: Normed Spaces 5-3

• Completeness is equivalent—i.e.

(V, F ) complete in k · ka ⇐⇒ (V, F ) complete in k · kb

• density is equivalent—i.e.

X a dense subset of V in k · ka ⇐⇒ X a dense subset of V in k · kb

Remark 1.3 (What is the point of this? ) As far as convergence is concerned, k · ka and k · kb lead
to the same answer, yet in practice the analysis may be more easy in one than the other.

5.1.2 Continuity

Continuous maps have the desirable property that a small perturbation in x results in a small perturbation
in the operator at that point. They are paramount in the study of robustness and the effect of perturbations
of data.

Definition 1.3 (Continuity.) Let F = R (or C) and consider two normed linear spaces (U, G, k · kU )
and (V, F, k · kV ). Let f be a map (or operator) s.t. f : U → V .
a. (Local continuity). We say f is continuous at u ∈ U iff for every ε > 0, there exists δ > 0 (possibly
depending on ε at u) such that considering points u0 ∈ U

ku0 − ukU < δ =⇒ kf (u0 ) − f (u)kV < ε

b. (Global continuity). We say f is continuous on U iff it is continuous at every u ∈ U .

5.1.3 Induced Norms

Definition 1.4 (Induced Operator Norm.) The induced (operator) norm of f is defined to be
 
kf (u)kV
kf k = sup
u6=0 kukU

Note: sup =‘supremum’ is the least upper bound and inf =‘infimum’ is the greatest lower bound. The
concepts of infimum and supremum are similar to minimum and maximum, but are more useful in analysis
because they better characterize special sets which may have no minimum or maximum.
Let A : U → V be a (continuous) linear operator. Let U and V be endowed with the norms k · kU and k · kV ,
resp. Then, the induced norm of A is defined by

kAukV
kAki = sup
u6=0 kukU

The induced norm kAk is the maximum gain of the map A over all directions; moreover, kAk depends on
the choice of the norms k · kU and k · kV in the domain and co-domain, resp.

Definition 1.5 (Bounded Operator.) Let V, W be normed vector spaces over field F (=R or C). A
linear transformation or linear operator A : V → W is bounded if ∃ 0 < K < ∞ such that

kAxkW ≤ KkxkV , ∀x ∈ V
5-4 Lecture 5: Normed Spaces

Theorem 1.1 (Connections between bounded and continuous linear operators.) Let V, W be
normed vector spaces and let A : V → W be a linear transformation. The following statements are
equivalent:
a. A is a bounded linear transformation
b. A is continuous everywhere in V
c. A is continuous at 0 ∈ V .

Proof. [a. =⇒ b.] Suppose a. Let K as in the definition of bounded linear transformation. By linearity of
A we have that
kA(x) − A(x̃)kW = kA(x − x̃)kW ≤ Kkx − x̃kV
which implies b.

[b. =⇒ c.] trivial

[c. =⇒ a.] If A is continuous at 0 ∈ V , ∃ δ > 0 such that for v ∈ V with kvk < δ we have that kAvk < 1
since continuity at a point x implies that for every ε > 0, ∃ δ > 0 s.t.
kx − x0 kV < δ =⇒ kA(x) − A(x0 )kQ < ε
so, in particular, it is true for ε = 1.

Now, let x ∈ V with x 6= 0. Then

 
δ δ δ

2kxkV x = < δ =⇒ A

<1
x
V 2 2kxkV W
But by linearity of A,
   
≤ kxkV · 1 = 2 kxkV
2kxkV x 2kxkV x
kAxkW = A δ = A δ
δ 2kxkV W δ 2kxkV W δ δ

And, since kAxkW ≤ (2/δ)kxkV for x = 0 too, then K = 2/δ.

The following are facts about induced norms.

Fact 1.2 Let (U, F, k · kU ), (V, F, k · kV ), and (W, F, k · kW ) be normed linear spaces and let A : V → W ,
à : V → W and B : U → V be linear maps. Note that ∀α ∈ F , αA : V → W , A + à : V → W and
AB : U → W are also linear maps. Then we have that
1. kAvkW ≤ kAkkvkV
2. kαAk = |α|kAk, ∀ α ∈ F
3. kA + Ãk ≤ kAk + kÃk
4. kAk = 0 ⇐⇒ A = 0 (zero map)
5. kABk ≤ kAkkBk

Matrix norms are induced norms or operator norms. Consider A ∈ F m×n , A : F n → F m . The induced
matrix norm is defined by
 
kAxk
kAk = sup {kAxk : x ∈ F n , kxk = 1} = sup : x ∈ F n, x =
6 0
kxk
Lecture 5: Normed Spaces 5-5

With this definition, we can show the following

• 1–norm (column sums)
m
X
kAki,1 = max |aij |
1≤j≤n
i=1

• 2–norm (largest singular value—we will discuss singular values later)

p
kAki,2 = λmax (A∗ A) = σmax (A)

• Forbenius norm  1/2

m X
X n
kAki,F =  |aij |2 
i=1 j=1

Note: kAk2 ≤ kAkF

• ∞–norm (row sums)
n
X
kAki,∞ = max |aij |
1≤i≤m
j=1

Suppose A is an m × n matrix and we have the induced norm

kAxkp
kAkp = sup
x6=0 kxkp

(’sup’ stands for supremum—aka, least upper bound). Since kxkp is a scalar, we have

kAxkp Ax
kAkp = sup = sup
x6=0 kxkp x6=0 kxkp p

Since x/kxkp has unit length, we can express the induced norm as follows:

kAkp = sup kAxkp

kxkp =1

That is, kAkp is the supremum of kAxkp on the unit ball kxkp = 1. Note that kAxkp is a continuous function
of x and the unit ball B1 (0) = {x| kxkp = 1} is closed and bounded (compact).

Fact 1.3 On a compact set, a continuous function always achieves its maximum and minimum values.

The above fact enables us to replace the ’sup’ with a ’max’. Indeed,
kAxkp
kAkp = max = max kAxkp
x6=0 kxkp kxkp =1

When computing the norm of an operator A, the definition is the starting point and the remainder of the
process is the following:
step 1. First, find a candidate for the norm, call it K for now, that satisfies

kAxkp ≤ Kkxkp , ∀ x

step 2. Then, find at least one non-zero x0 for which

kAx0 kp = Kkx0 kp

step 3. Finally, set kAkp = K.

5-6 Lecture 5: Normed Spaces

We can do this because

kAxkp ≤ kAkp kxkp
which is clearly the case by inspecting the definition of kAkp .

Recall that
kxk∞ = max |xi |
i

Fact 1.4 The sup norm is the max of row sums:

 
Xn
kAk∞ = max  |aij | (max of row sum)
i
j=1

Proof. We prove this using the above steps. Let A ∈ Rm×n . Then, Ax ∈ Rm is a vector so that
Pn
k=1 a1k xk


kAxk∞ =
.
.
Pn .


k=1 amk xk

∞
 n 
X 
= max  aik xk 
 
i∈{1,...,m}  
k=1
n
X
≤ max |aik xk |
i∈{1,...,m}
k=1
n
!
X
≤ max |aik | max |xk |
i∈{1,...,m} k∈{1,...,n}
k=1
n
!
X
= max |aik | kxk∞
i∈{1,...,m}
k=1

so that we have completed step 1—that is,

n
X
K= max |aik |
i∈{1,...,m}
k=1

so that
kAxkp ≤ Kkxk∞ , ∀ x

Step 2 above requires that we find non-zero x0 for which equality holds in the above inequality. Examination
reveals that we have equality if x0 is defined to have the components
 ai∗ k
|ai∗ k | , aik 6= 0
xk = 1≤k≤n
1, a i∗ k = 0

where i∗ above is
n
X
i∗ = arg max |aij |
i∈{1,...,m}
j=1

You can check this by starting with the definition for kAxk∞ and plugging in this choice of x0 to see that

kAx0 k = Kkx0 k
Lecture 5: Normed Spaces 5-7

P 
 n
maxi∈{1,...,m}  j=1 aij xj 

kAxk∞
kAk∞ = sup = sup
kxk6=0 kxk∞ kxk6=0 maxi∈{1,...,m} |xi |
 
 n
X 

= sup max  aij xj 
kxk=1 i∈{1,...,m}  j=1 
n
X
≤ sup max |aij ||xj | (triangle inequality)
kxk=1 i∈{1,...,m} j=1
n
X
≤ sup max |aij | max |xj |
kxk=1 i∈{1,...,m} j=1 j

n
X
= max |aij |
i∈{1,...,m}
j=1

where equality holds when we let x be the vector with kxk∞ = 1 such that it extracts the max row sum,
i.e. for 1 ≤ j ≤ n, let  ai∗ j
|ai∗ j | , 6 0
ai∗ j =
xj = (5.1)
1 , ai∗ j = 0
where
n
X
i∗ = arg max |aij |
i∈{1,...,m}
j=1

Pn
Allowing for the vector x to have the above property, we have kAk∞ = maxi∈{1,...,m} j=1 |aij |

Definition 1.6 (Positive definite matrix.) A matrix P ∈ Rn×n such that for all x ∈ Rn

xT P x ≥ 0, whenever x 6= 0

is said to be positive definite.

A positive definite matrix P has only positive eigenvaules. Indeed, for all (non-trivial) eigenvectors x we
have that
xT P x = λxT x = λkxk2 > 0
and thus, λ > 0.

Fact 1.5 Every positive definite matrix induces a norm via

kxk2P = xT P x

5.2 Inner Product Space (Hilbert Spaces)

Let the field F be R or C and consider the linear space (H, F ). The function

h·, ·i : H × H → F [(x, y) 7→ hx, yi]

is called inner product iff

5-8 Lecture 5: Normed Spaces

1. hx, y + zi = hx, yi + hx, zi, ∀x, y, z ∈ H

2. hαx, yi = αhx, yi, ∀α ∈ F, ∀x, y ∈ H
3. kxk2 = hx, xi > 0 ⇐⇒ x ∈ H s.t. x 6= 0
4. hx, yi = hy, xi
where the overbar denotes the complex conjugate of hy, xi. Moreover, in (c) k · k is called the norm induced
by the inner product.

Definition 2.1 (Hilbert Space.) A space equipped with an inner product is called a Hilbert space and
denoted by (H, F, h·, ·i).

Example 2.1 The following are examples:

1. (F n , F, h·, ·i) is a Hilbert space under the inner product
n
X
hx, yi = x̄i yi = x∗ y
i=1

2. L2 ([t0 , t1 ], F n ) (space of square, integrable F n -valued functions on [t0 , t1 ]) defined by the inner product
by
t1
Z
hf, gi = f (t)∗ g(t) dt
t0

Theorem 2.1 (Schwawrz’s Inequality.) Let (H, F, h·, ·i) be an inner product space.

|hx, yi| ≤ kxkkyk ∀ x, y ∈ H

This inequality implies the triangle inequality for the inner product norm kxk2 = hx, xi.

5.2.1 Orthogonality

Definition 2.2 (Orthogonality.) Two vectors x, y ∈ H are said to be orthogonal iff hx, yi = 0.

If two vectors are orthogonal, they satisfy

kx + yk2 = kxk2 + kyk2 (Pythogoras’ Theorem)

Definition 2.3 (Orthogonal Complement.) If M is a subspace of a Hilbert space, the subset

M ⊥ = {y ∈ H| hx, yi = 0, ∀x ∈ M }

is called the orthogonal complement of M .

Fact 2.1 M ∩ M ⊥ = {0}.

Proof. Suppose not. Then there exists x 6= 0 with x ∈ M ∩ M ⊥ . Hence,

hx, yi = 0 ∀y ∈ M
but x ∈ M so
hx, xi = 0
This in turn implies that x = 0 →←.
Lecture 5: Normed Spaces 5-9

5.2.2 Adjoints

Let F be R or C and let (U, F, h·, ·iU ) and (V, F, h·, ·iV ) be Hilbert spaces. Let A : U → V be continuous
and linear.

Definition 2.4 (Adjoint.) The adjoint of A, denoted A∗ , is the map A∗ : V → U such that

hv, AuiV = hA∗ v, uiU

Consider A : V → W .

matrix type notation Adjoint property

transpose AT hAf, gi = hf, A∗ gi, ∀ f ∈ V, g ∈ W (adjoint)

symmetric A = AT hAf, gi = hA∗ f, gi, ∀ f ∈ V, g ∈ W (self-adjoint)

orthognal A−1 = AT hf, gi = hAf, Agi, ∀ f ∈ V, g ∈ W (isometry)

Example 2.2 Let f (·), g(·) ∈ C([t0 , t1 ], Rn ) and define A : C([t0 , t1 ], Rn ) → R by

A : f (·) 7→ hg(·), f (·)i

Find the adjoint map of A.

soln.
A∗ : R → C([t0 , t1 ], Rn )
such that
hv, Af (·)iR = hA∗ v, f (·)iC([t0 ,t1 ],Rn )
where v ∈ R and f (·) ∈ C([t0 , t1 ], Rn ).
c.f. with
hA∗ v, f (·)i =⇒ A∗ : v 7→ v ∗ g(·)
(multiplication by g(·)). Indeed,
hv, Af (·)iR = v ∗ hg(·), f (·)iC([t0 ,t1 ],Rn )
= hv ∗ g(·), f (·)iC([t0 ,t1 ],Rn )
We can also compute the matrix of an adjoint of a linear operator.

Definition 2.5 (T ) he adjoint of matrix A is

A∗ = AT

Regarding a matrix representation of a linear operator T , if the matrix A is its representation with respect
to bases α and β of V and W , respectively, then we would like for the adjoint of A to coincide with the
matrix of the adjoint of T with respect to β and α.
We need the following definition.

Definition 2.6 (Orthonormal Basis.) An orthonormal basis for an inner product space V with
finite dimension is a basis for V whose vectors are orthonormal —that is, they are all unit vectors and
orthogonal to each other.

For example, the standard basis for Rn is an orthonormal basis.

5-10 Lecture 5: Normed Spaces

Theorem 2.2 (L) et V, W be finite dimensional inner product spaces with α, β as orthonormal bases for
V and W , respectively. If T : V → W is a linear operator, then

[T ∗ ]βα = ([T ]α
β)
∗

where [T ]βα denotes the matrix representation of T wrt the bases α and β for V and W , respectively.

Proof. If (aij ) is the matrix representation of T wrt α, β, then

X
T αi = aij βj , 1 ≤ i ≤ m
j

Since α is orthonormal,
hT αi , βj i = aij , 1 ≤ i ≤ m, 1 ≤ j ≤ n
Indeed, plugging in T αi from above we have
DX E X
ai` β` , βj = ai` hβ` , βj i = aij
` `

∗
since hβ` , βj i = 1 only when ` = j. Similarly, if B = (bij ) is the representation of ([T ]α
β ) , then

hT ∗ βi , αj i = bij

Then
bij = hT ∗ βi , αj i = hβi , T αj i = hT αj , βi i = a∗ji
so that
B = AT

Remark 2.1 (“Necessity”) This is NOT true in general—i.e. it is true when both α and β are
orthonormal.

We can still write [T ∗ ]α β m

β in terms of [T ]α without assuming α and β are orthonormal. Indeed, let α = {αi }i=1
n
and β = {βj }j=1 . Then
[T ∗ ]α
β =C
−1
([T ]βα )∗ B
where  
hα1 , α1 i hα2 , α1 i · · · hαm , α1 i
 hα1 , α2 i hα2 , α2 i · · · hαm , α2 i 
C=
 
.. .. .. .. 
 . . . . 
hα1 , αm i hα2 , α1 i ··· hαm , αm i
and  
hβ1 , β1 i hβ2 , β1 i ··· hβn , β1 i
 hβ1 , β2 i hβ2 , β2 i ··· hβn , β2 i 
B=
 
.. .. .. .. 
 . . . . 
hβ1 , βn i hβ2 , β1 i ··· hβn , βn i

5.2.3 Self-Adjoint Maps

Let (H, F, h·, ·iH ) be a Hilbert space and let A : H → H be a continuous linear map with adjoint A∗ : H → H.
Lecture 5: Normed Spaces 5-11

Definition 2.7 (Self-Adjoint.) The map A is self-adjoint iff A = A∗ , or equivalently

hx, AyiH = hAx, yiH , ∀x, y ∈ H

Definition 2.8 (Hermitian matrices.) Let H = F n and let A be represented by a matrix A :

n
F →F ,A∈Fn n×n
. Then A is self-adjoint iff the matrix A is Hermitian (i.e. A = A∗ , viz. aij = āij ,
∀i, j ∈ {1, . . . , n}.

Fact 2.2 Let A : H → H be a self-adjoint, continuous linear map. Then

a. all eigenvalues of A are real
b. if λi and λk are distinct eigenvalues of A, then any eigenvector vi associated with λi is orthogonal to
any eigenvector vk associated with λk .

Proof of part (a). From Avi = λi vi and A∗ = A, we get that

hvi , Avi i = λi kvi k2 = hAvi , vi i = hvi , Avi i

It follows that λi is real.

Proof of part (b). From Avi = λi vi and Avk = λk vk we obtain

λi hvk , vi i = hvk , Avi i

and
λk hvk , vi i = hAvk , vi i = hvk , Avi i
Subtracting the last two equations, we get

(λi − λk )hvi , vk i = 0

Since λi 6= λk ,
hvi , vk i = 0

5.2.4 Properties of the Adjoint

Let A be a linear continuous map from U to V (both Hilbert spaces). Then, A∗ is linear and continuous
with induced norm kA∗ k = kAk. Moreover, A∗∗ = A.

Theorem 2.3 (Finite Rank Operator Lemma (FROL).) Let F be R or C. Let (H, F, h·, ·iH ) be a
Hilbert space and consider F m as the Hilbert space (F m , F, h·, ·iF m ). Let A : H → F m be a continuous
linear map with adjoint A∗ : F m → H. Then
A∗ : F m → H, AA∗ : F m → F m , A∗ A : H → H
are continuous linear maps with AA∗ and A∗ A self-adjoint. Furthermore,
⊥ ⊥
a. H = R(A∗ ) ⊕ N (A), F m = R(A) ⊕ N (A∗ )
b. The restriction A|R(A∗ ) is a bijection of R(A∗ ) onto R(A) and
N (AA∗ ) = N (A∗ ), R(AA∗ ) = R(A)

c. The restriction A∗ |R(A) is a bijection of R(A) onto R(A∗ ) and

N (A∗ A) = N (A), R(A∗ A) = R(A∗ )
5-12 Lecture 5: Normed Spaces

R(A∗ ) R(A)

A∗
0H 0m

N (A) N (A∗ )

H Fm

Figure 5.1: The orthogonal decomposition of the domain and the co-domain of a finite rank operator A :
H → F m and its associated bijections.

Proposition 2.0 Let V and W be finite-dimensional nonzero inner product spaces. Let T ∈ L(V, W ).
Then ker(T ∗ ) = (im(T ))⊥

Proof. We must show that ker(T ∗ ) ⊂ (im(T ))⊥ and ker(T ∗ ) ⊃ (im(T ))⊥ .
Let w ∈ ker(T ∗ ). Then T ∗ (w) = 0. Hence, for all v ∈ V , we have 0 = hv, 0i = hv, T ∗ (w)i which implies that
hT (v), wi = 0 for all v ∈ V . Therefore w is orthogonal to every vector in im(T ) so w ∈ (im(T ))⊥ .
Now, let w ∈ (im(T ))⊥ . Then every vector in im(T ) is orthogonal to w, that is hT (v), wi = 0 for all v ∈ V .
Thus hv, T ∗ (w)i = 0 for all v ∈ V . But if hv, T ∗ (w)i = 0 for all v ∈ V then this must imply that T ∗ (w) = 0
so w ∈ ker(T ∗ ).

Proposition 2.0 Let V and W be finite dimensional inner product spaces and let T ∈ L(V, W ). Then
im(T ∗ ) = (ker(T ))⊥ .