Linear Algebra

Chapter 1.
Vectors in
n
R
and
n
C
, spatial vectors
A vector
v
is a member of a Vector Space
n n
C , R
.
If
V v , u
where V is a vector space, and
,
are
scalars (numbers) then
v u ) v u ( + +
,
u u u ) ( + +
,
u ) ( ) u (
are all in V .
A vector is an ordered collection of numbers
(quantities), an array.
e.g.
0.1] 3.5 - 3 . 2 [ v

off) on on off on on ( v
yellow) white blue green blue ( v
In physics, vectors are things with a magnitude
and a direction. Or, it is a point in a space.

) 3 , 4 (
vector addition, vector amplifications.
We may depict a vector either as a row of
numbers or a column of numbers. It is up to us,
but weve to be consistent.
If
) u ,..., u , u ( u
n 2 1

then
1
1
1
1
]
1

n
2
1
t
u
..
u
u
u v
is the
transpose of the vector
u
. Conversely,
u v
t

.
Here,
vector row u
, vector column v
Vector addition, scalar multiplication. Negation of
a vector ( if v, then what is v?).
A vector
) u ,..., u , u ( u
n 2 1

is a tuple in an n-space.
The components
i
u
are also called coordinates,
entries, elements, etc.
One may consider a set of unit vectors comprising
this ) C or ( R
n n
space. The corresponding vector-
space that it collectively spans allows us to express
each vector as follows:
k
n
1 k
k n n k k 2 2 1 1
u i u i ... u i ... u i u i u

+ + + + +

If the vector space is orthogonal, each unit vector

spanning the space is orthogonal to the other. The
collection of the unit vectors that span completely
any vector in the vector space is called a basis.
In the above, the set of unit vectors
} i {
j
form an
orthonormal basis.
e.g. A three-dimensional orthogonal space
Norm (or Length) of a vector.
The norm or length of a vector
n
R u
is
|| u ||
.
If
) u ,..., u , u ( u
n 2 1

then

n
1 i
2
i
u || u ||
u
is a unit vector if
1 || u ||
.
Therefore,
|| v ||
v
v
is a unit vector.
Accordingly, if
u
and
v
are two points in the n-
space
n
R
, the distance between them is
PQ
where

,
_

n
1 i
2
i i
) v u ( PQ

if the notion of a distance is meaningful in that
space.
In such a space, the dot product (or the inner
product) between two vectors is defined in this
way:
v
P
u
Q
For
) u ,..., u , u , u ( u
n 3 2 1

and
) v ,..., v , v ( v
n 2 1

, the
dot product is
v . u
or
> < v | u
(this Dirac notation
will be explained shortly)

+ + + > <
i
i i n n 2 2 1 1
v u ) v u ... v u v u ( v | u v . u
In terms of the physics-metaphor
cos | v || u | v | u > <
, where is the angle
between the two vectors.
The norm of a vector
u
is then
> < u | u
. We
should be careful. This is only one variety of norm
that we can think of with a vector. In fact, we can
have a number of them:

n
1 i
i 1 1
| u | || u || : norm l
|| u || || u || : norm l
2 2

as we have defined it.
p / 1
n
1 i
p
i p p
| u | || u || : norm l

,
_

| u | max || u || : norm l
i
i


For our sake, wed mostly consider
norm l
2

.
An inner product space V is a vector space where
norm of a vector is defined and one could form a
dot product
> < v | u
between any pair of vector
u
and
v
in it with the following conditions:
1.
0 u | u > <
the length of a vector is never
negative.
2.
> > < < u | v v | u
symmetry
3.
> < + > < > + < v | w v | u v | w u
4.
> < > < v | u v | u
with

a constant
5.
v | v u | u v u | v u > < + > < > + + <
triangle
inequality
6.
> >< < > < v | v u | u v | u
Cauchy-Schwarz
inequality
ex.
2) - 3 (1 v 1) - 0 2 ( u
5 1 0 4 u | u + + > <
14 4 9 1 v | v + + > < and
14 5 0 (-2)) (-1) 3 0 1 (-2 v | u + + > <
ex. Let
2
R V
be an inner product space where
the dot products are defined in the following term:
bd ac )) d , c ( ), b , a (( +

Then 0 b a )) b , a ( ), b , a ((
2 2
+ (condition 1 is
fulfilled and
2
R
is an inner product space.
ex. A polynomial space
n
P
is an inner product
space where every vector is an n-degree
polynomial (polynomial-format?) like
n
n
2
2 1 0
x a ... x a x a a ) x ( p + + + +
n
n
2
2 1 0
x b ... x b x b b ) x ( q + + + +
Then
n n 1 1 0 0
b a ... b a b a q | p + + + > <
is an inner
product between two such vectors in
n
P
.
Another inner product in
n
P
may be defined as

> <
1
0
dx ) x ( q ) x ( p q | p
ex. Another inner product space is the space of
continuous functions where the dot product
between
) x ( f
and
) x ( g
is defined as

> <

-
f(t)g(t)dt g | f

In this set up, the basis set
} ,..., v , v { S
2 1

is an
infinite set of vectors like
2
1
v
1

,
t cos
1
v
2

,
t sin
1
v
3

,
nt cos
1
v
n 2

,
nt sin
1
v
1 n 2

+
That these form an orthonormal basis is evident
from the fact that
n m for 0 dt nt sin mt cos

,
n m for 0 dt nt cos mt cos

n m for 0 dt nt sin mt sin

ex. A weighted Eucledian inner product space

n
R

may be designed to yield dot products like
n n n 2 2 2 1 1 1
v u ... v u v u v | u + + + > <
with
1 ...
n 2 1
+ + +
The neural networks are developed on such
spaces.
More observation.
A set S in an inner product space V is called
orthogonal if any two distinct vectors in S are
orthogonal. If also each vector is a unit vector, the
set S is called orthonormal.
The set of unit vectors in
n
R
comprise an
orthonormal space if
j k jk
i | i < >
(Kronecker delta)

otherwise 0 k, j if 1
jk

Notice that a finite orthonormal set S is spanned
by linearly independent vectors.
Accordingly, if a set of vectors
V S
can be
identified such that any vector V v can be
expressed uniquely as a linear combination of
vectors in S , the set S is a basis for V . Thus,
n n 3 3 2 2 1 1
v i

... v i

v i

v i

v + + + +
if each
S i

k

.
A vector space may have several distinct bases but
each will have same number of basis vectors in
them. The number of basis vectors spanning a
vector space is called the dimension of a vector
space.
Ex. The standard basis spanning the
3
R
is the set
of three vectors
} e , e , e { S
3 2 1

where
0) 1 0 ( e , 0) 0 1 ( e
2 1

and
1) 0 0 ( e
3

Any three-dimensional vector can be expressed in
this basis. e.g.
3 2 1
e 3 e 5 4e 3) 5 - 4 ( v +
Obviously other bases in this vector space are
possible. Suppose, we choose these as our basis set
} u , u , u { ' S
3 2 1

with
2) 3 (0 u 1), 2 - 1 ( u
2 1

and
1) - 1 2 ( u
3

Then,
3 3 2 2 1 1
u u u 3) 5 - 4 ( v + +
and the
coefficients are determined by the equations
3 2
5 3 2
4 2
3 2 1
3 2 1
3 1
+
+
+



Distances, angle and projections
The distance between the two vectors (the distance
between the terminals of the two vectors)
> < v | u ) v , u ( d
If
u
and
v
are vectors in an inner product space
V, the angle between these two vectors is given
by
|| v |||| u ||
v | u
cos
> <

The two vectors are orthogonal (perpendicular to
each other) if
0 v | u > <
Projection of a vector on another vector (see the
diagram)
Projection of
u
on vector
v
is
v
u proj( u.v )
is
v
<u.v> u.v
u proj( u.v ) u cos = ||u||
||u||||v|| || v ||

< >


v
u

u cos