Machine Learning

National University of Colombia

Dimensionality Reduction

Yesica Alejandra Salas Cárdenas

December 7th, 2018

 Content
1

MOTIVATION
PRINCIPAL METHODS

PCA
What and How?

The Matematical formulation

Steps for performing PCA
Chossing K (Number of principal components)

Executing on python

| The Feather Beamer Theme

 MOTIVATION
2

| The Feather Beamer Theme

 What is dimensionality reduction?
3

Dimensionality reduction is the procedure of reducing the number of

random variables under some considerations, with the goal of
obtaining a set of principal variables, which capture the highest
possible variance and information about the original data set.

If we have large data set with 50 variables or more, and we need to

train a learning algorithm, then we probably have to compress the
data.

| The Feather Beamer Theme

 4

AIMS
I FEATURE SELECTION:
Choosing the p < k most important features
I FEATURE EXTRACTION:
It allow us reduce dimensional space to a lower dimension
space.

X (i) ∈ R k , =⇒ z (j) ∈ R p , p < k

| The Feather Beamer Theme

 5

¿Why compress the data?

I Less memory or less space on the disk.
I Speed up learning algorithm
I Quit the features which aren’t relevant in our problem.
I Visualize data in 2D or 3D.

| The Feather Beamer Theme

 6

Disadvantages of Dimesionality Reduction

I Loss data.
I Find undesirables linear correlations between variables.
I PCA can fail sometimes where mean and covariance are not
enough.
I The quantity of principal components is subjective.

| The Feather Beamer Theme

 MOTIVATION
7

PRINCIPAL METHODS
I Principal Component Analysis(PCA)
I Linear Discriminant Analysis(LDA)
I Quadratic Discriminant Analysis(QDA)
I Regularized Discriminant Analysis (RDA)
I Multidimensional scalling (MDS)

| The Feather Beamer Theme

 PCA
8

¿What is PCA?
I PCA reducing the dimension of a data into specified smaller
dimension.
I Unsupervised learning algorithm.
I Quantify the variability of the data.
I PCA helps to visualize better the high dimensional data.

| The Feather Beamer Theme

 PCA
9

For PCA it conveniently turns out when determining the principal

components:
I The maximal variance of the projected data.
I Minimizes the square of euclidean distance between data and
their projections.

¿What plot is more convenient for the principal components?

| The Feather Beamer Theme

 ¿How does it? 10

I PCA transforms variables into new set of variables.

I PCA is an orthogonal and linear transformation. The new
variables are known as PRINCIPAL COMPONENTS.
I The first component has the highest variance followed by
second, third and so on.

| The Feather Beamer Theme

 11

The Matematical formulation

Let X = [x1 , x2 , x3 , . . . , xk ]
Σ = E[(x − E(x))T (x − E(x))], E(x) = 0

1
COV (X ) = (X T X )
n−1
Let v1 the first pc, v1 = θ11 x1 + θ12 x2 + θ13 x3 . . . θ1k xk

| The Feather Beamer Theme

 12

I The projected points of X onto V es called z = x T v

I Find v such that Var(z) is maximized

Var (v T x) = E[(v T x − v T µ)2 ]

= E[(v T x − v T µ)(v T x − v T µ)]
= E[v T (x − µ)(x − µ)T v ]
= v T E[(x − µ)(x − µ)T ]v
= v T Σv

where Var (x) = E[(x − µ)(x − µ)T ] = Σ

| The Feather Beamer Theme

 The first principal component 13

Maximize Var(z) subject to ||v||=1

max{v1T Σv1 − λ1 (v1T v1 − 1)}

v1

L(v1 ) = v1T Σv1 − λ1 (v1T v1 − 1)

 
∂L(v1 )
= 2Σv1 − 2λ1 v1 = 0
∂v1

Σv1 = λ1 v1
v1 is an eigenvector of Σ
Choose the one with the largest eigenvalue for Var(z) to be max
var (x T v1 ) ≈ v1T Σv1 = v1T λ1 v1 = λ1 v1T v1 = λ1

| The Feather Beamer Theme

 The second principal component
Maximize Var (z2 ) subject to ||v2 || = 1 and orthogonal to v1 v2T v2 = 1 14

and v2T v1 = 0

max{v2T Σv2 − λ2 (v2T v2 − 1) − δ(v2T v1 − 0}

v2

L(v2 ) = v2T Σv2 − λ2 (v2T v2 − 1) − δ(v2T v1 )

 
∂L(v2 )
= 2Σv2 − 2λ2 v2 − δ(v1 ) = 0
∂v2
2v1T Σv2 − 2λ2 (
v1T T
2 ) − δ(v1 v1 ) = 0
v (1)
In the other way:

2v1T Σv2 = 2v2T Σv1 = 2v2T (Σv1 ) = 2v2T (λ1 v1 ) = λ1 (

v2T
v
1) = 0

Replace in the equation (1) this, we obtain δ = 0 and hence:

Σv2 − λ2 (v2 ) = 0

Σv2 = λ2 v2
where v is another eigenvector of Σ
| The Feather Beamer Theme
 15
T
var(x v2 ) ≈ v2T Σv2 = v2T λ2 v2 = λ2 (v2T v2 ) = λ2

PROCEEDING ONWARDS
I The eigenvalue, λ1 and their corresponding eigenvector v1 of Σ
such that var (x T V1 ) ≈ v1T Σv1 = λ1
I The eigenvalue, λ2 ≤ λ1 and their corresponding eigenvector v2
of Σ such that var (x T V2 ) ≈ v2T Σv2 = λ2
I The eigenvalue, λ3 ≤ λ2 and their corresponding eigenvector v3
of Σ such that var (x T V3 ) ≈ v3T Σv3 = λ3
..
.
I The eigenvalue, λp ≤ λp−1 and their corresponding eigenvector
vp of Σ such that var (x T Vp ) ≈ vp0 Σvp = λp

| The Feather Beamer Theme

 Steps for performing PCA
16

STEPS FOR PERFOMING PCA:

x (i) −mean(x (i) )
I STEP 0: Noramlized the data with stdev (x (i) )
I STEP 1: Find the matrix of covariance Σ
I STEP 2: For Σ find all eigenvalues and their correspoding
eigenvectors and sort this.
I STEP 3: we construct a matrix with the first p principal
components, Preduce = [v1 v2 . . . vp ].
I STEP 4: Finally to get our resulting data in p-dimensions,
compute Z = XPreduce .

| The Feather Beamer Theme

 Chossing K (Number of principal components)
17

I If our objective is visualize the data, then picking the first two or
three principal components that makes sense.
I Typically, choose K to be smallest value so that 99% of variance
is retained, as below:

1
Pm
m i=1 kx
(i)
− xaprox (i) k2
1
P m (i) 2
≤ 0.01
m i=1 kx k

For K given,
Pk
Sii
1 − Pi=1
n ≤ 0.01
i=1 Sii

Pick smallest value of k for which

Pk
Sii
=⇒ Pni=1 ≥ 0.99
i=1 Sii
| The Feather Beamer Theme
 Chossing K (Number of principal components)
18

I The total proportion of of Variance (PoV) explained is above 85%

| The Feather Beamer Theme

 Chossing K (Number of principal components)
19

I Plot of PoV vs K

| The Feather Beamer Theme

 Executing on python
20

Executing on python
I To execute PCA in python, simply import PCA from sklearn
library.
I Let’s to implement this tool learned FOR RECOGNITION OF
HANDWRITTEN DIGITS DATA SET.

| The Feather Beamer Theme

Thank you!