Householder Matrices

DEFINITION. A linear transformation P of

n
is said to be a projection if
P
2
= P* = P.
PROPOSITION. If P is a projection, then 1 - P is a projection.
PROOF. We have that
(1 - P)* = 1 - P
and
(1 - P)(1 - P) = 1 - (1)P - P(1) + PP = 1 - P - P + P = 1 - P.
Thus, the linear transformation 1 - P is a projection. Q.E.D.
EXAMPLE. Let a be a unit vector in
n
. Let a!a denote the linear transformation
a!a(x) = <x, a>a.
Then a!a is a projection. This was demonstrated in Homework 8. The matrix of this linear transformation is A =
aa* since
Ax = aa*x = (a*x) a = <x. a> a.
EXAMPLE. More generally, let a and b be vectors in
n
. Let a!b denote the linear transformation
a!b(x) = <x, b>a.
The matrix of a!b is given by ab*.
DEFINITION. Let a be a unit vector in
n
. The linear transformations of the type U
a
given by
U
a
(x) = x - 2<x, a>a
is called a Householder transformations.
PROPOSITION. Let a be a unit vector in
n
. Then the Householder transformation U
a
is a selfadjoint unitary
transformation.
PROOF. The Householder transformation U
a
can be expressed as
U
a
= (1 - P
a
) - P
a
.
where P
a
= P is the projection
P(x) = <x, a>a.
So we have that
U
a
* = ((1 - P) - P)*
= (1 - P)* - P*
= (1 - P) - P = U
a
and
U
a
*U
a
= U
a
U
a
= ((1 - P) - P)((1 - P) - P)
= (1 - P)
2
-P(1 - P) - (1 - P)P + P
2
= (1 - P) - (P - P
2
) - (P - P
2
) + P
= (1 - P) - (P - P) - (P - P) + P
= 1 - P + P = 1.
So we have that U
a
is a selfadjoint unitary. Q.E.D.
Householder Matrices page 1
PROPOSITION. Let w = (0, , 0, cos ", 0, , 0, sin ", 0, , 0). Then
A Householder transformation has a simple geometric description.
PROPOSITION. Let a be a unit vector. Then U
a
(a) = - a and U
a
(x) = x for every x perpendicular to a. Here x is
perpendicular to a if <x, a> = 0.
PROOF. We have that
U
a
(a) = a - 2<a, a>a = a - 2a = -a
and
U
a
(x) = x - <2x, a>a = x
if x is perpendicular to a. Q.E.D.
The diagram for the Householder transformation is the following:
The geometry of the Householder transformation is useful in deriving formulae for moving vectors aroud via a
Householder transformation.
PROPOSITION. Let x and y be vectors of the same length in
n
with x # y. Then Householder transformation U
w
with
w =
x - y
||x - y||
takes x onto y.
REMARK. If x = y in the preceding formula, then the Householder transformation U
0
= 1 maps x onto y = x.
PROOF. We have that v = x + y is perpendicular to w since
<w, v> =
x - y
||x - y||
, x + y =
1
||x - y||
<x, x> + <x, y> - <y, x> -<y, y> = 0.
Here we used the fact that
<x, x> = <y, y>
from the hypothesis on the length and the fact that
<x, y> = <y, x>
since x and y are in $
n
. So we have that
U
w
(x + y) = x + y.
Thus, we get that
Householder Matrices page 2
U
w
(x) = U
w
x - y
2
+
x + y
2
=
1
2
U
w
(x - y) +
1
2
U
w
(x + y) = -
1
2
(x - y) +
1
2
(x + y) = y.
Q.E.D.
A Householder matrix can be used to drive a matrix to so-called Hessenberg form.
DEFINITION. An n " n matrix A = (a
i j
) is said to be in (upper) Hessenberg form if a
i j
= 0 if i > j + 1.
The definition means that A is in Hessenberg from if everything below the subdiagonal of a matrix is 0. The
subdiagonal is the diagonal from the upper left hand corner to the lower right hand corner that lies immediately
below the main diagonal.
We now show that there is a Householder transformation that will "zero-out" the lower entries of a vector.
LEMMA (Zeroing Out Lemma). Let a = (a
1
, , a
n
)
T
be a vector in !
n
. Then there is a Householder matrix
U
w
such that
U
w
a = ||a|| e
1
(respectively, U
w
a = - ||a||e
1
).
PROOF. First we see that the vector w% = a - ||a|| e
1
is perpendicular to a + ||a||e
1
since
<a - ||a||e
1
, a + ||a||e
1
> = ||a||
2
- ||a|| <a, e
1
> + ||a|| < e
1
, a> - ||a||
2
<e
1
, e
1
> = 0
Setting
w = w%/||w%||,
if w% # 0, we get that
U
w
(a) =
1
2
(U
w
(a - ||a||e
1
) +
1
2
U
w
(a + ||a||e
1
)) = -
1
2
(a - ||a||e
1
) +
1
2
(a + ||a||e
1
) = ||a||e
1
.
If w% = 0, then we take U equal to the identity.
A similar proof holds for the second case. Q.E.D.
REMARK. A same Lemma holds for complex vectors. We show that there is a complex number & of modulus 1
and a unit vector w with the propery
U
w
a = &||a||e
1
.
Here we assume that there is some need for the zeroing out, i.e., that
|a
2
| + + |a
n
| > 0.
For every complex number & of modulus 1, the vector
a - &||a||e
1
= (a
1
- &||a||, a
2
, , a
n
)
is nonzero. We shall give the proper choice of & presently. We form the Householder transform U
w
where w is the
vector
w =
a - &||a||e
1
|| a - &||a||e
1
||
.
By the geometry of the Householder matrix, we get
U
w
(a - & ||a||e
1
) = - (a - & ||a||e
1
)
and
U
w
(a + &||a||e
1
) = a + &||a||e
1
since
Householder Matrices page 3
<a -&||a||e
1
, a + &||a||e
1
> = ||a||
2
- &<||a||e
1
, a > + &<a, ||a||e
1
> - |&|
2
||a||
2
.
Now we see that the proper choice of & is the complex number of modulus 1 with the property
&<||a||e
1
, a > ' $.
This is the number
& =
<||a||e
1
, a >
<||a||e
1
, a >
if &<||a||e
1
, a >is nonzero or 1 if &<||a||e
1
, a > = 0. Then we have that
&<||a||e
1
, a > = &<||a||e
1
, a > # &<a, ||a||e
1
>
so that
- &<||a||e
1
, a > + &<a, ||a||e
1
> = 0.
Now we can compute U
w
a as
U
w
(a) =
1
2
(U
w
(a - &||a||e
1
) +
1
2
U
w
(a + &||a||e
1
)) = -
1
2
(a - &||a||e
1
) +
1
2
(a + &||a||e
1
) = &||a||e
1
.
To prove the second part we use the vector a + &||a||e
1
THEOREM. Let A be an n " n complex matrix. There are Householder matrices U
1
, , U
n -1
such that
U
n -1
U
2
U
1
A
is upper triangular.
PROOF. We do this in the ususal way by a deflation process. We examine the first column a
1
of A and find a
Householder matrix U
1
such that
U
1
a
1
=
(
0

0
.
Here U
1
is found as in the previous part so that
U
1
a
1
= ||a||e
1
.
Then we have that
U
1
A = [U
1
a
1
, U
1
a
2
, , U
1
a
n
]
where a
i
is the ith column of i. So we have that
U
1
a =
b
11
b
12
b
1n
0 b
22
b
2n

0 b
nw
b
2n
nn
= B.
We find an (n - 1) " (n - 1) dimensional Householder matrix U%
2
with
U%
2
B =
c
22
c
23
c
2n
0 c
33
c
3n

0 c
n3
c
nn
.
We then have that
U%
2
= 1
n -1
- 2a!a
for some a in subspace spanned by e
2
, , e
n
. So we have that
U
2
=
1 0
0 U%
2
= 1 - 2&a!a
is a Householder matrix. But now we have that
Householder Matrices page 4
U
2
B =
1 0
0 U%
2

b
11
b
12
b
1n
0 b
22
b
2n

0 b
n2
b
2n
=
b
11
b
12
b
1n
0 c
22
c
23
c
2n
0 c
33
c
3n
0
0 0 c
n3
c
nn
.
So we have set up a recurrence and we can get to upper triangular form in n - 1 steps. Q.E.D.
Now we show that we can reach Hessenberg form with n - 1 successive transforms
ad U
i
where U
i
are Householder matrices. We need a little modification of the preceding zeroing-out Lemma to preserve
the Hessenberg form under the right multiplication of U
i
.
LEMMA. Let a = (a
1
, , a
n
)
T
be a vector in
n
. Then there is a unit vector w (respectively, w%) of the form such
(0, w
2
, , w
n
) such that the Householder matrix U
w
satisfies
U
w
a = (a
1
, ||a%||, 0, , 0)
T
(respectively, U
w%
a = (a
1
, - ||a%||, 0, , 0)
T
) where a% = (a
2
, , a
n
)).
PROOF. This a modification of the previous zeroing-out lemma. By the previous zeroing out lemma, there is a
unit vector
w% = ( w
1
, , w
n - 1
)
in
n - 1
such that
U
w%
a% = (||a%||, 0, , 0)
T
.
However, the matrix
U =
1 0
0 U
w%
is a Householder matrix for the unit vector
w = (0, w
1
, , w
n - 1
)
in
n
. To verify this we only need to compute as follows:
1 - 2(w!w) =
1 0 0
0 1 0

0 0 1
- 2
0
w
1

w
n - 1
0, w
1
, , w
n - 1
Householder Matrices page 5
=
1 0 0
0 1- 2w
1
w
1
- 2w
1
w
n - 1

0 - 2w
n -1
w
1
1- 2w
n -1
1w
n - 1
=
1 0
0 U
w%
.
Finally we have that
Ua =
a
1
U
w%a%
= (a
1
, ||a%||, 0, , 0)
T
.
Q.E.D.
Now we have the reduction to upper Hessenberg form.
THEOREM. Let A be an n " n matrix. Then there are at most n - 1 Householder matrices U
1
, , U
n - 1
such that U
n - 1
U
n - 2
U
1
AU
1
U
n - 2
U
n - 1
is a Hessenberg matrix. If A is selfadjoint, then
U
n - 1
U
n - 2
U
1
AU
1
U
n - 2
U
n - 1

is a tridiagonal matrix.
PROOF. We start a recurrence using the previous zeroing-out lemma. We can find a Householder matrix
U
1
= U
w
1
with w
1
a unit vector of the form
w
1
= (0, w
12
, , w
1n
)
such that U
1
a
1
has the form
U
1
a
1
= (($ ($ 0, , 0)
T
.
Here a
1
is the first column of A and in general a
i
is column i of A. Since U
1
has the form
U
1
=
1 0
0 U
w%
we get that
U
1
A = [U
1
a
1
, , U
1
a
n
] =
* * * *
* * * *
0 * * *

0 * * *
.
Consequently, we have that
Householder Matrices page 6
U
1
AU
1
= (U
1
A)U
1
=
* * * * *
* * * * *
0 * * * *
0 * * * *

0 * * * *

1 0
0 U
w%
=
b
11
b
12
b
13
b
14
b
1n
b
11
b
22
b
33
b
34
b
2n
0 b
32
b
33
b
34
b
3n
0 b
42
b
43
b
44
b
4n

0 b
n2
b
n3
b
n4
b
nn
.
Here ( indicates that the place is filled with an entry which may not be 0.
Now we do the recurrence step l Using the matrix
B =

b
22
b
23
b
24
% b
2n
b
32
b
33
b
34
% b
3n
b
42
b
43
b
44
% b
4n
% % % % %
b
n2
b
n3
b
n4
% b
nn
,
there is a unit vector w%% '
n -1
with 0 in the first coordinate such that
U
w%%

b
11
b
12
b
13
b
14
% b
1n
b
21
b
22
b
23
b
24
% b
2n
0 b
32
b
33
b
34
% b
3n
0 b
42
b
43
b
44
% b
4n
% % % % % %
0 b
n2
b
n3
b
n4
% b
nn
U
w%%
=
c
11
c
12
c
13
% c
1n
c
21
c
22
c
23
% c
2n
0 c
32
c
33
% c
3n
% % % % %
0 c
n2
c
n3
% c
nn
Since the first coordinate of w%% is 0, we have as before that
U
w%%
=
1 0
0 U
w%%%
and that the n " n matrix
U
2
=
1 0 0
0 1 0
0 0 U
w%%
is also a Householder matrix. Here w%%% are the last n - 2 coordinates of w%%. Pre and post multiplying U
1
AU
1
by U
2
does not disturb the first column. So we have that
U
2
U
1
AU
1
U
2
= U
2


b
11
b
12
b
13
b
14
% b
1n
b
21
b
22
b
23
b
24
% b
2n
0 b
32
b
33
b
34
% b
3n
0 b
42
b
43
b
44
% b
4n
% % % % % %
0 b
n2
b
n3
b
n4
% b
nn
U
2
=
b
11
c
12
c
13
c
14
% c
1n
c
21
c
22
c
23
c
24
% c
2n
0 c
32
c
33
c
34
% c
3n
0 0 c
43
c
44
% c
4n
% % % % % %
0 0 c
n3
c
n4
% c
nn
.
So now we get the proof of the theorem from recurrence.
If A s selfadjoint, then U
1
A has first column equal to (b
11
$ b
21
, 0 , 0)
T
and has first row equal to the original
first row of A. So we have ththe first row of
Householder Matrices page 7
U
1
AU
1
= (U
1
A)U
1
is equal to
(b
11
$ b
21
, 0 , 0).
So we are heading to a tridiagonal matrix and the further iterations confirm this Q.E.D.
Homework
Find Householder matrices U
1
and U
2
such that U
2
U
1
AU
1
U
2
is tridiagonal where
A =
120 80 40 - 16
80 120 16 -40
40 16 120 -80
-16 -40 -80 120
.
Householder Matrices page 8
Homework 9
Find Householder matrices U
1
and U
2
such that U
2
U
1
AU
1
U
2
is tridiagonal where
A =
120 80 40 - 16
80 120 16 -40
40 16 120 -80
-16 -40 -80 120
.
SOLUTION. We let
w%
1
=
1
|| a%
1
- ||a%
1
||e
1
||
a%
1
- ||a%
1
||e
1
where a
1
is the first column of A and a%
1
= (80, 40, -16)
T
. We calculate
||a%
1
|| =
80
40
16
&
80
40
16
= 90.863
w%
1
=
80
40
16
-
90.863
0
0
=
10.863
40
16
and
w
1
=
1
0
2
+ (10.863)
2
+ 40
2
+ (- 16)
2
0
10.863
40
16
=
0
0.24449
0.9003
0.36012
.
This gives the first Householder matrix as
U
1
=
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
2w
1
w
1
T
=
1 0 0 0
0 0.88045 0.44023 0.17609
0 0.44023 0.62108 0.64843
0 0.17609 0.64843 0.74063
.
We then have
U
1
AU
1
=
120 90.863 0 0
90.863 157.21 56.97 36.378
0 56.97 152.85 1.0047
0 36.378 1.0047 49.94
.
Now we find U
2
. We have that
56.97
36.378
&
56.97
36.378
= 67.594
and
Householder Matrices page 9
0
0
56.97
36.378

0
0
67.594
0
=
0
0
124.56
36.378
so that
1
0
0
124.56
36.378
&
0
0
124.56
36.378
0
0
124.56
36.378
0
0
124.56
36.378

=
0 0 0 0
0 0 0 0
0 0 0.92141 0.26909
0 0 0.26909 0.078588
with
U
2
=
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
2
0 0 0 0
0 0 0 0
0 0 0.92141 0.26909
0 0 0.26909 0.078588
=
1 0 0 0
0 1 0 0
0 0 0.84282 0.53819
0 0 0.53819 0.84282
.
This produces
U
2
U
1
AU
1
U
2
=
120 90.863 0 0
90.863 157.21 67.594 0
0 67.594 123.95 46.257
0 0 46.257 78.836
which is tridiagonal.
Householder Matrices page 10