Eigenvalues and Singular values

Juliette Ryan
Paris 13
February 2014
Singular values
Numerical Computation
Unitary matrix factorization, normal matrices
Theorem : For any A(N,N), A = UTU

where U is an unitary matrix (UU

= U

U = Id)
and T is an upper triangular matrix
Denition : A is normal if AA

= A

A
Theorem : A is normal <=> there exists U unitary matrix
such that A = UDU

and D is diagonal
Corollary : A hermitian is diagonalizable
Corollary : A real symmetric is diagonalizable
Corollary : A hermitian matrix is positive denite <=> all
eigenvalues
i
> 0
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Eigenvalue localization
Two Gerschgorin-Hadamard theorems
Gerschgorin disk for A(N,N) :
D
k
= z (, [z a
kk
[
k
=

j =k
[a
kj
[
Theorem 1 : All eigenvalues of A

k=1,...,N
D
k
Theorem 2 : If A not reducible , if

k=1,...,N
D
k
, then
belongs to all Gerschgorin circles.
Note : A not reducible i.e ; A can not be expressed as an upper
triangular block matrix

Consequences on dominant matrices

Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Eigenvalue localization : Proof theorem 1
, eigenvalue, u eigenvector such that max[u
i
[ = 1,
with k such that u
k
= 1
( a
kk
)u
k
=

j =1,j =k
a
kj
u
j
thus ([ a
kk
[

j =1,j =k
[a
kj
[
and D
k
k not known but you are sure that

k=1,...,N
D
k
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Eigenvalue localization : Proof theorem 2
possible eigenvalue but not
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Eigenvalue localization : Proof theorem 2
not in

k=1,...,N
D
k
, thus for any k, [ a
kk
[
k
i such that [ a
ii
[
i
[ a
ii
[ =
i
I = i , [u
i
[ = 1 = max[u
k
[ , = , and

j =1,N,j =i
[a
ij
[[u
j
[ [( a
ii
)u
i
[ =
i
=

j =1,N,j =i
[a
ij
[
thus

j =1,N,j =i
[a
ij
[(1 [u
j
[) 0
By denition [u
j
[ 1 thus [a
ij
[(1 [u
j
[) = 0
let J such that J

I = 1, 2, ...N and J

I =
J = otherwise the matrix would be reducible and
[a
kk
[ = [(a
kk
)u
k
[ = [

k
,= ja
kj
u
j
[

k
,= j [a
kj
[ =
k
,
for any k

k=1,...,N
D
k
and

k=1,...,N
D
k
D
k
for all k
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Dominant matrices
A(N,N)
Denitions
A diagonally dominant <=> i = 1, ...N, [a
ii
[

j =i
[a
ij
[
A strictly diagonally dominant <=>
i = 1, ...N, [a
ii
[ >

j =i
[a
ij
[
A strongly diagonally dominant <=> A is diagonally
dominant and k such that [a
kk
[ >

j =i
[a
kj
[
Propositions
If A strictly diagonally dominant, A is invertible
If A strongly diagonally dominant and irreductible, A is
invertible
If A is either strictly diagonally dominant or irreductible and
strongly diagonally dominant and if all diagonal elements
a
ii
are real and a
ii
> 0 the i , Re(
i
) > 0
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Singular Values
(See http ://en.wikipedia.org/wiki/Singular value decomposition
and
Denition : A (M,N) rectangular matrix M > N. Singular
values of A (
i
) are the square roots 0 of A

A
eigenvalues (A

A is (N,N) square matrix)

Theorem : A (M,N) rectangular matrix M > N. There exits
U(M,M) and V(N,N) unitary square matrices such that
U

AV = where =
_
_
_
_
_
_
_
_

2
... _
_
N
_
_
_
_
_
_
_
_
_
and
i
are A singular values
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Singular Values
The right singular vectors of A are eigenvectors of A

A
The left singular vectors of A are eigenvectors of AA

Corollary : rank(A) = number of non zero values

i
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Pseudo inverse
Denition : Let =
_
_
_
_
_
_
_
_

2
... _
_
N
_
_
_
_
_
_
_
_
_
,
the pseudo inverse of is

+
=
_
_
_
_

1
[

2
[
... _ [ _
_
N
[
_
_
_
_
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Pseudo inverse
Denition : Let A = UV

, the pseudoinverse of a is
A
+
= V
+
U

Note : If A is a square regular matrix, AA

+
= A
+
A = Id
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Real number representation
References :
http://docs.sun.com/source/806-3568/ncg_
goldberg.html
http://www5.in.tum.de/

huckle/bugse.html
Decimal representation : innite computer memory ;
Approximated representation : oating point ;
x m.b
p
with b : base, m mantissa, p exponent ;
mantissa : number written as
m = 0, a
1
a
2
. . . a
N
=
N

k=1
a
k
b
k
, b
1
m < 1 ;
Consequence : relative accuracy
x
x
=
m
m

b
N
b
1
= b
1N
=
Consequence :
real simple accuracy 10
7
real double accuracy 10
14
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Finite accuracy Arithmetics
Let real numbers represented with 3 signicant digits
Compute x +y +z with x = 8, 22 y = 0, 00317 z = 0, 00432
x +y = 8, 22317 8, 22
(x +y) +z 8, 22432 8, 22
y +z = 0, 00749
x +(y +z) 8, 22749 8, 23
Sum is not associative !
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Round off errors on arithmetics operators
Round off error on a sum
(x +y) ([x[ +[y[)
If s
n
=
n

k=1
u
k
, then s
n
s
n1
+s
n
(s
n
) ([u
n
[ +2[u
n1
[ +3[u
n2
[ + +(n1)[u
2
[ +(n1)[u
1
[)
Better accuracy starting the sum with small absolute
value terms
Round off error on mutiply/Divide
(xy) [xy[
(x

1
1
x

2
2
x

n
n
) ([x

1
1
[+[x

2
2
[+ +[x

n
n
)[x

1
1
x

2
2
x

n
n
[
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Solve equation x
2
1634x +2 = 0
Computation is made with 10 signicative digits.
Classical formulas :

= 667487,

816, 9987760
x1 = 817 +

1633, 998776
x2 = 817

0, 0012240
5 signicative digits are lost on x
2
!
x
1
x
2
= 2, thus
x
2
=
2
x
1
1, 223991125.10
3
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Approximate computation of e
10
Use the series e
10

k=0
(1)
k
10
k
k!
general term : [u
k
[ =
10
k
k!
;
[u
9
[ = [u
10
[ =
10
10
10!
2, 755.10
3
e
10
4, 5.10
5
Comparison u
10
et e
10
:
u
10
: 2 7 5 5, . . . . . .
e
10
: 0, 0 0 0 0 4 5
Loss of signicative digits !
Cure : e
10
=
1
e
10
with e
10

k=0
10
k
k!
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Recurrence computation
Compute I
n
=
_
1
0
x
n
10 +x
, n N
Simple recurrence :
_
I
0
= ln
11
10
I
n
=
1
n
10I
n1
Error : I
n
10I
n1
10
n
I
0
Computation I
36
10
22
!
Bound :
1
11(n+1)
I
n

1
10(n+1)
Inverse iteration I
n1
=
1
10
_
1
n
I
n
_
Cure : I
n1

1
10
I
n
;
Approximate I
46

1
1147
Compute I
36
from approximation of I
46
: I
36
10
10
!
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Iterative computation
Compute the sequence u
0
= 2, u
n+1
= [ ln(u
n
)[
u
0
2,000000000 2,000000001 1,999999999 5.10
10
u
5
5,595485181 5,595484655 5,595485710 9.10
8
u
10
0,703934587 0,703934920 0,703934252 5.10
7
u
15
1,126698502 1,126689382 1,126707697 8.10
6
u
20
1,266106839 1,266256924 1,265955552 10
4
u
24
1,000976376 1,001923276 1,000022532 10
3
u
25
0,000975900 0,001921429 0,000022532 100%
u
26
6,932150628 6,254686211 10,700574400 50%
u
30
0,880833175 0,691841353 1,915129896 100%
Numerical error :f (x) = [f

(x)[x
f (x)
|f (x)|
=
|f

(x)||x|
|f (x)|
x
|x|
=
1
| ln x|
x
|x|
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Errors due to the condition number
Solve Ax = b et Ay = b +b with
A =
_
_
_
_
10 7 8 7
7 5 6 5
8 6 10 9
7 5 9 10
_
_
_
_
, b =
_
_
_
_
32
23
33
31
_
_
_
_
, b =
_
_
_
_
0, 01
0, 01
0, 01
0, 01
_
_
_
_
x =
_
_
_
_
1
1
1
1
_
_
_
_
, y =
_
_
_
_
1, 82
0, 36
1, 35
0, 79
_
_
_
_
,
1
x = y x =
_
_
_
_
0, 82
1, 36
0, 35
0, 21
_
_
_
_
Relative perturbation of 3.10
4
error of 0.8 (2500).
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Errors due to the condition number. . .
Solve Ax = b et (A +A)z = b with
A =

10 7 8 7
7 5 6 5
8 6 10 9
7 5 9 10

, A =

0 0 0, 1 0, 2
0, 08 0, 04 0 0
0 0, 02 0, 11 0
0, 01 0, 01 0 0, 02

, b =

32
23
33
31

Solving the 2 systems :

x =
_
_
_
_
1
1
1
1
_
_
_
_
, z =
_
_
_
_
81
137
34
22
_
_
_
_
,
2
x = z x =
_
_
_
_
82
136
35
21
_
_
_
_
Relative error of 10
2
relative perturbation of 80.
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Analysis
det(A) = 1 ;
A
1
=
_
_
_
_
25 41 10 6
41 68 17 10
10 17 5 3
6 10 3 2
_
_
_
_
Eigenvalues (
i
)
i =1, ,4
of A?
min
i =1, ,4
[
i
[ 0, 01
max
i =1, ,4
[
i
[ 30
max
i =1, ,4
[
i
[
min
i =1, ,4
[
i
[
3000 = C
Coincidence ?
|
1
x|
2
|x|
2
= 0, 82
C.
b
2
b
2
= 1
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Error Analysis by right hand side perturbation
Recall perhaps : matrix norm induced from a vector norm
|A| = sup
x=0
|Ax|
|x|
= sup
x=1
|Ax|
x solution of A.x = b et y= x +
1
x solution of A.y = b +b.
_

1
x = A
1
b
b = A.x

_
|
1
x| |A
1
|.|b|
|b| |A|.|x|
|
1
x|
|x|

_
|A||A
1
|
_
|b|
|b|
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Error Analysis by perturbation of the matrix
x solution of Ax = b et z solution of (A +A).z = b.

2
x = A
1
A(x +
2
x) |A
1
|.|A|.|x +
2
x|
|
2
x|
|x +
2
x|

_
|A|.|A
1
|
_
|A|
|A|
Ryan Eigenvalues and Singular values
Singular values
Numerical Computation
Finite precision computation
Numerical instabilities
Denition of the condition number and properties
Denition
Cond(A) = |A||A
1
|
Properties
1
cond(A) 1
2
cond(A) = cond(A
1
)
3
cond(A) = cond(A) ; ,= 0
4
cond
2
(A) = |A|
2
|A
1
|
2
=

max

min
where
min
and
max
are
respectively highest and lowest singular values of A
Ryan Eigenvalues and Singular values