6

Linear Algebra and Inverse Problem

6.1

Introduction

Direct problem ( Forward problem) is to nd eld quantities satisfying Governing equations, Boundary conditions, Initial conditions.
The direct problem can be formulated as a well-posed problem.
Inverse problem ( Backward problem) is to nd some parameters related to Governing equation,
Boundary conditions, Initial conditions, Geometry conditions from observation data.
In general, inverse problems become nonlinear and ill-posed. So inverse problems are solved after
linearization and regularization.
Examples of inverse problems
Example 1 inuence line
f j (j=1, ..., n)

u i (i=1, ..., m)

Let Aij , yi and xj be inuence functions, measured data and parameters to be determined, respectively. Then we have
ui =

Aij fj (i = 1, . . . , m) or

{u} = [A] {f } .

(313)

j=1

Example 2 Greens function in solid mechanics

T
t(z)

u(x)

Using the Greens function G(, ), the displacement u due to the surface traction t is expressed
by

u() =
G(, )t()dS.
(314)
S

Due to the Saint-Venants principle, the distributed forces t(), applied at far eld from the observation point , can be approximated by the resultant force T without any change of u. It means
that it is dicult to nd a unique solution t, i.e., the inverse problem is an ill-posed problem.
Example 3 geological prosecting

1
t

0 Dt

The problem is to determine the location, shape and constitution of subterranean bodies from
measurements at the earths surface.
39

As shown in the above gure, consider a one dimensional problem that the mass density, x(t),
distributed along the t-axis for 0 t 1 is determined from the vertical component of force, y(s),
on the surface line s. The vertical force y(s) due to a small mass element x(t)t is written as
y(s) = g

x(t)t
x(t)t
cos = g
.
2
(s t) + 1
((s t)2 + 1)3/2

where g is the gravity constant. It then follows that

1
x(t)dt
y(s) = g
((s

t)2 + 1)3/2
0

(315)

(316)

Example 4 simplied tomography

I x detector

(x,y)

x
I 0 emitter
Consider the simplied tomography in which the radiation absorption coecient f (x, y) of the
object is determined from the measured intensity Ix of the radiation, as shown in the above gure.
Then the radiation intensity Ix satises the following dierential equation.
dIx
= f Ix
dy

(317)

The solution Ix is obtained in the form of the function p(x)

p(x) ln(I0 /Ix ) =

y(x)

f (x, y)dy

(318)

y(x)

Suppose that the object has

a circular region and the absorption coecient f (x, y) has the form of
f (x, y) = f (r), where r = (x2 + y 2 ). Then we have

p(x) =

2r
f (r)dr
x2

r2

(319)

Example 5 structural dynamics

As shown in the above gure, we consider here structural dynamics with single degree of freedom
governed by the equation of motion as follows.
m
x + cx + kx = 0 or x
+ ax + bx = 0

(320)

subjected to the initial conditions

x(0) = x0 , x(0)

= x 0 .

(321)

The inverse problem is to nd the constant values a and b from the time history of the mass x(t).
Integrating eq.(320) twice using the initial conditions leads to
t
t
x(t) x0 x 0 t + a
(x(s) x0 )ds + b
x(s)(t s)ds = 0, t > 0.
0

40

(322)

Suppose that displacements x1 , x2 , . . . , xn at the time steps of t = h, 2h, . . . , nh are measured. If

the integrals in eq.(322) are evaluate by the trapezoidal rule, eq.(322) can be discretized as follows:
Ek (a, b) =

xk x0 x 0 kh + a(

xj h xk h/2 x0 kh)

j=1
k1

+b(

xj (k j)h2 + x0 kh2 /2) k = 1, 2, . . . , n

(323)

j=1

Since eq. (323) is an approximated equation to eq.(322), E

k (a, b) is generally nonzero. For n > 2,
n
two parameters, a and b, are determined so that E(a, b) = k=1 (Ek (a, b))2 becomes minimum. To
this end, least squares method is generally used to nd optimal values for a and b.
Problem 6.1

6.2

1.

Solve eq.(320) with the initial conditions x(0) = 1, x(0)

= 1 with m = 1, c = 2,
k = 5, and calculate the displacements x(tk ) at the time steps of t = tk (k =
1, . . . , n), where tn 1.

2.

Then generate noisy data by adding the noise k to the calculated displacement
x(tk ), namely, xk = x(tk ) + k , where k is a uniformly distributed random number
in [, ].

3.

Write a program to estimate a and b by means of the least square method to minimize
E(a, b).

4.

Compare results calculated from data with various noise levels and discuss the stability of the method with respect to .

Linearized inverse problems

As seen in the last section, many inverse problems are formulated in integral forms, which can be reduced
to the following system of equations after discretization.
n

{b} = m [A] {x}.

(324)

Note that in general, [A] is a non-square matrix with the size m n.

The system of equation (324) is classied into three categories depending on the numbers of observation
points and measurement points, namely, m and n, as follows.
(a) m > n
Consider the case of m = 3 and n = 2. Then

y1
y2
=

y3

we have three equations for two unknowns x1 and x2 ;

}
A11 A12 {
x1
A21 A22
.
(325)
x2
A31 A32

The above equation is an overdetermined set of linear equations. As illustrated in Fig. 11, there
exists no exact solution, but the least squares solutions may be found in this case.
X2

X2

X1

X1

Figure 11:

Figure 12:

(b) m = n

41

(b)1 If A is regular, a unique solution is obtained as

x = A1 b

(326)

For m = n = 2, the behavior of equations is shown in Fig. 12.

(b)2 If A is singular, there exists no solution as shown in Fig. 13.
X2

X1

Figure 13:
(b)3 When A is nearly singular, the solution is very sensitive to erros involved observed data.
For example, we here consider the system of equations given as follows.
{
} [
]{
}
y1
A11
A12
x1
=
(327)
y2
A11 A12 +
x2
Then the solution x is obtained by
[
{
}
1
A12 +
x1
=
A11
x2
A11

A12
A11

]{

y1
y2

}
(328)

As seen in Fig. 14, the solution is very sensitive to the error involved in the component of
X2

X2

X1

X1

Figure 14:

Figure 15:

the matrix. The stability of solution will be discussed later.

(c) m < n
In this case, the number of equations is less than the number of unknowns. As seen in Fig. 15 for
m = 1 and n = 2, we cannot obtain unique solution.
Problem 6.2.1 Discuss the existence, uniqueness and stability of solutions for the following system
of equations.
{
}
[
] x1
{
}
1 1
2
=
(329)
x2
[
]{
}
{
}
1 1
x1
2
=
(330)
1 1
x2
3

[
] x1
{
}
2 1 1
3
x2
=
(331)
1 1 0
2

x3

}
2 1 {
4
1 1 x1
3
=
(332)
x2

1 0
1

}
2 1 {
4
1 1 x1
3
=
(333)
x2

1 0
2
42

6.3

Singular Value Decomposition (SVD)

6.3.1

SVD of a square regular matrix

If A is an m m square matrix, then A is decomposed on the basis of its eigenvalues and eigenvectors.
The statement that ui is an eigenvector associated with eigenvalue i can be written
Aui = ui i .

(334)

If we now write the column vectors u1 , . . . , um in order to form the square matrix

..
..
..
.
.
.

u
u
.
.
.
u
U =
1
2
m ,

..
..
..
.
.
.

(335)

the formulae (334) for i = 1, . . . , m may be written as

AU = U .

(336)

where is the diagonal matrix of eigenvalues. Multiplying eq.(336) by the inverse matrix U 1 , the
matrix A can be decomposed as follows.
A = AU U 1 = U U 1 .

(337)

Eq.(337) expresses the singular value decomposition of the matrix A. The set of eigenvalues corresponds
to the spectrum of A. If two or more eigenvalues of A are identical, the spectrum of the matrix is called
degenerate. The spectrum nomenclature is the exact analogy with the idea of the spectrum of light
as depicted in a rainbow. The brightness of each color of the spectrum tell us how much light of that
wavelength exists in the undispersed white light. For this reason, the procedure of SVD is often referred
to as a spectral decomposition.
When A is a real symmetric square matrix, the eigenvectors show the orthogonality of U T U =
U U T = I, i.e., U 1 = U T , which is a useful mathematical property. Thus we have
A = AU U T = U U T =

k uk uTk .

(338)

k=1

Using the property of eq.(338), it is easy to compute any power of a real symmetric square matrix A
An = (U U T )n = U n U T ,

(339)

Since is diagonal, n can simply be computed by raising each of its diagonal elements to the nth
power.
The decomposition (338) means that the action of the real symmetric matrix A on an input vector
x Rm may be understood in the following three steps:
1. uTk x gives the component of the input vectgor x in the direction of the eigenvector uk . by which
the input vector x is resolved along the k-th eigenvector uk .
2. The component in the k-th eigenvectors direction is multiplied by the eigenvalue k ,
3. The product of k uTk x shows how much the k-th component along the eigenvector uk contributes
in Ax.
The most useful result of eq.(339) is the inverse of the matix A as follows
A1 = U 1 U T =

1
uk uTk .
k

(340)

k=1

Problem 6.3.1 Derive the singular value decomposition and

the following square matrices.

6 3 7
5
1 2
1 , 2
5 3 6
5
43

determine the inverse matricies of

2
2
2

5
2
5

(341)

6.3.2

SVD of a non-square matrix

The SVD for a square matrix can be extended to a decomposition for a non-square matrix A of size
m n. We may consider two square symmetric matrices AT A and AAT , being of size n n and m m,
respectively.
Since AT A and AAT are square and symmetric, the eigenvectors and eigenvalues for those matrices
can be computed. The eigenvectors of AT A and AAT can be chosen to form orthonormal bases in nand m-dimensional spaces, respectively. Also all eigenvalues are non-negative for the positive semidenite
matrices. Let v be the eigenvector of AT A corresponding to the eigenvalue , i.e.,
AT Av = v.

(342)

Multiplying on the left by v T and grouping the terms,

(v T AT )(Av) = (v T v)

(343)

The square norm of Av on the left hand side is a non-negative quantity, and v T v on the right hand side
is positive. Therefore, the eigenvalue becomes necessarily non-negative.
Let v i be the eigenvector associated with the i-th eigenvalue i of the matrix AT A, and assume that
we sort i so that
AT Av i = i v i , and 1 2 . . . n 0.
(344)
Similarly, let ui and i be the i-th eigenvector and eigenvalue of AAT and assume that
AAT ui = i ui , and 1 2 . . . m 0.

(345)

Multiplying the both sides of eq.(344) with i = 1 by A yields

A(AT A)v 1 = (AAT )(Av 1 ) = 1 (Av 1 ).

(346)

If 1 is not equal to zero, the vector Av 1 is also non-zero. Thus, we have

(AAT )

Av 1
Av 1
= 1
.
||Av 1 ||
||Av 1 ||

(347)

From the comparison of the above equation with eq.(345), Av 1 /||Av 1 || and 1 are considered as the
eigenvector and the eigenvalue of AAT , respectively. Namely,
u1 =

Av 1
, 1 = 1 .
||Av 1 ||

(348)

We can deduce similar results for k = 2, 3, . . . and then obtain

uk =

Av k
, k = k (k = 1, 2, . . . , r)
||Av k ||

(349)

where r is the rank of the matrix A or AT , r m and r n, and r+1 = . . . = n = 0 and that
r+1 = . . . = m = 0.
By multiplying both sides of eq.(345) by AT , we obtain the following relation between v k and uk ,
vk =

AT u k
, k = k (k = 1, 2, . . . , r)
||AT uk ||

(350)

The norms ||Av k || and ||AT uk || may be evaluated as follows.

||Av k ||2 = (Av k )T (Av k ) = v Tk (AT A)v k = v Tk k v k = k

(351)

where v k is normalized so that v Tk v k = 1. Since ||AT uk ||2 = k and k = k > 0, we may dene k as
the square root of the eigenvalue to write

(352)
||Av k || = ||AT uk || = k = k = k , (k = 1, 2, . . . , r)
Hence, eqs.(349) and (350) are simplied as
Av k
AT u k

= k uk ,
= k v k .
44

(353)
(354)

The eect of the linear transformation A on the unit vector v k Rn is to transfer v k to the vector
of the length k in the direction of the unit vector uk Rm . The eect of the linear transformation AT
on the unit vector uk Rm is to transfer uk to the vector of the unit vector v k Rn multiplied by the
amplitude k .
On the other hand for k > r, the eigenvalue k of AT A associated with v k is zero and so AT Av k = 0.
Multiplying this equation by v Tk shows that ||Av k ||2 = 0 and hence that Av k = 0. We thus have the
following equation.
Av k = 0 for k = r + 1, . . . , n
(355)
In the same way, it then follows that
AT uk = 0 for k = r + 1, . . . , m.

(356)

Both equations (353) and (355) describe the action of A on the vectors {v k } for k = 1, . . . , n. The matrix
A with such action can be written as
r

A=
k uk v Tk .
(357)
k=1

It is easy to check (using the orthonormality of the basis {v k } ) that the right hand side of eq.(357) does
have the same action as shown in eqs.(353) and (355). Taking the transpose of (357) gives
AT =

k v k uTk ,

(358)

k=1

and it is also easy to check that this is consistent with (354) and (356).
The orthonormal vectors {v k } are known as the right singular vectors, while the vectors {uk } are
known as the left singular vectors, and the scalars {k } are called the singular values of the matrix A.
We may write the column vectors uk and v k in order of k = 1, 2, . . . to form the m m matrix U and
the n n matrix V . The equation (357) may then be written in matrix form as
A = U V T

(359)

where is an mn matrix whose only non-zero elements are the rst r entries in the diagonal components.
Problem 6.3.2 Express the following matrix in the form of the singular value decomposition as
shown in eq.(359).
[
]
3 1 1
(360)
1 3 1

45