Sie sind auf Seite 1von 25

Support Vector Machine

SVM
MATLAB code examples

Poli Amlo

Agosto 2014

Poli Amlo

Support Vector Machine

Outline

Poli Amlo

Support Vector Machine

Outline

Poli Amlo

Support Vector Machine

Introduction
Support Vector Machine (SVM) is a tool that we can use to
classification between a sets. We have a set xi of training data
and labels yi that describe at which class belong.
Then we have data training xi and labels yi , now we want
to predict at which class belongs a new element x with the
similar characteristics of training data.
When you are working with classification is because you
have your elements as an arrange of vectors, if not we suggest that you transform your data in vectors, you can do it
using kernels. But if you are new using SVM we suggest to
do the simple case using points of two elements to plot them
in the Cartesian plane and it could help you to see clearest all
the steps for SVM.

Poli Amlo

Support Vector Machine

The simple case


The simplest case is the classification of two sets, in this
example we choose 6 points. . .
Training data
Class one x1 = (2, 2), x2 = (3, 1), x3 = (3, 4)
Class two x4 = (7, 2), x5 = (6, 4), x6 = (8, 4)

Labels
Class one y1 = 1, y2 = 1, y3 = 1
Class two y4 = 1, y5 = 1, y6 = 1

You can plot them to see them in the Cartesian plane.

F IGURE : Plot of training points


Poli Amlo

Support Vector Machine

The simple case


The simplest case is the classification of two sets, in this
example we choose 6 points. . .
Training data
Class one x1 = (2, 2), x2 = (3, 1), x3 = (3, 4)
Class two x4 = (7, 2), x5 = (6, 4), x6 = (8, 4)

Labels
Class one y1 = 1, y2 = 1, y3 = 1
Class two y4 = 1, y5 = 1, y6 = 1

You can plot them to see them in the Cartesian plane.

F IGURE : Plot of training points


Poli Amlo

Support Vector Machine

The simple case


The simplest case is the classification of two sets, in this
example we choose 6 points. . .
Training data
Class one x1 = (2, 2), x2 = (3, 1), x3 = (3, 4)
Class two x4 = (7, 2), x5 = (6, 4), x6 = (8, 4)

Labels
Class one y1 = 1, y2 = 1, y3 = 1
Class two y4 = 1, y5 = 1, y6 = 1

You can plot them to see them in the Cartesian plane.

F IGURE : Plot of training points


Poli Amlo

Support Vector Machine

The simple case


The simplest case is the classification of two sets, in this
example we choose 6 points. . .
Training data
Class one x1 = (2, 2), x2 = (3, 1), x3 = (3, 4)
Class two x4 = (7, 2), x5 = (6, 4), x6 = (8, 4)

Labels
Class one y1 = 1, y2 = 1, y3 = 1
Class two y4 = 1, y5 = 1, y6 = 1

You can plot them to see them in the Cartesian plane.

F IGURE : Plot of training points


Poli Amlo

Support Vector Machine

MATLAB Code

clc; clear all;close all;


X=zeros(6,2);
X(1,:)=[2,2];
X(2,:)=[3,1];
X(3,:)=[3,4];
X(4,:)=[7,2];
X(5,:)=[6,4];
X(6,:)=[8,4];
figure
for i=1:6
scatter(X(i,1), X(i,2)); hold on
end

Poli Amlo

Support Vector Machine

Means of the two classes

Now we calculate the means of the two classes


C+ =

1
xi
m+

(1)

C =

1
xi
m

(2)

Where m+ and m are the number of elements for the two


classes. In our example m+ = 3 and m = 3.
We calculated them and we obtained : C+ = (2.666, 2.333)
and C = (6.0, 3.5)

Poli Amlo

Support Vector Machine

Plotting means

F IGURE : Plot of mean classes

Poli Amlo

Support Vector Machine

MATLAB Code

Cplus = mean(X(1:3,:));
Cminus = mean(X(3:6,:));
figure
scatter(Cplus(1,1), Cplus(1,2)); hold on
scatter(Cminus(1,1), Cminus(1,2));
axis([0 10 0 6]); grid on;

Poli Amlo

Support Vector Machine

Calculating the middle point

Now we need to obtain the middle point and it will be useful to predict new points.
(C+ + C )
2
In our case this point is : C = (4.333, 2.916)
C=

Poli Amlo

Support Vector Machine

(3)

Predicting new points


For new points x that we will want to classify we calculate
B = x C and W = C+ C . We verify the angle between
these vectors B and W and if the angle is less than 90 degrees then it belongs to the first class if not then it belong to
the second class.
For example, for the point x = (5, 2) we can see the plot
and guess where it belongs, with the algorithm we obtain that
it belongs to second class due to the angle is 106.73 degrees

F IGURE : Find the class for a the point x = (5, 2)

Poli Amlo

Support Vector Machine

Predicting new points


Another example, x = (2, 2). For this new point we obtained
and angle of 2.15 degrees so this point belongs to the first
class.

F IGURE : Find the class for a the point x = (2, 2)

Poli Amlo

Support Vector Machine

MATLAB Code

x=[5,2];
scatter(x(1),x(2),green);hold on;
B=x-C;
W=Cplus-Cminus;
angle=dot(B,W)/(norm(B)*norm(W));
h=acos(angle);
a=h*180/pi

Poli Amlo

Support Vector Machine

Labels

Now we use the function sign to classify with the labels 1 if


it belongs to first class and 1 to second class.
y = sgn(h(x C), W i)

(4)

For the point x = (5, 2) we obtain y = 1 and for the point


x = (2, 2) we obtain y = 1.
MATLAB code :
y=sgn(dot(B,W));

Poli Amlo

Support Vector Machine

Outline

Poli Amlo

Support Vector Machine

Kernels

As we said in the first part, most of the time our data needs
to be in the featuring space, this means that we need to transform our data to obtain vectors that we can manage like in the
example. Schlkopf defines a kernel as :
Kernel
Function that, given two patterns x and x 0 returns a real
number characterizing their similarity.
A kernel k is symmetric (i.e. k (x, x 0 ) = k (x 0 , x))

Poli Amlo

Support Vector Machine

Outline

Poli Amlo

Support Vector Machine

Some concepts

Once that we understood the idea of a basic classification,


we continue with some concepts that are necessary before
the next example.
We will use hyperplanes to classify our classes and we can
find the optimal hyperplane using :
max min kx xi k|x H , hw, xi + b = 0, i = 1, ..., m

xH ,b

Where w is orthogonal to the hyperplane.

Poli Amlo

Support Vector Machine

(5)

Margins

For a hyperplane x H |hw, xilb = 0 a margin is :


hw,xi (x, y ) :=

y (hw, xi + b)
||w||

(6)

Note that when we multiply by y we are forcing that in the correct classification we will obtain a positive and almost if it is
correctly classify a point x the margin is the distance from x to
the hyperplane.
The margin will be 0 in the hyperplane. You can see the
geometrical representation in the next figure.

Poli Amlo

Support Vector Machine

Margins

F IGURE : Geometrical representation of margins 1 .


1. This image was taken from the book "Learning with Kernels", Schlkopf and Smola, 2002
Poli Amlo

Support Vector Machine

Support Vectors

The name of support vectors come from the optimization


problem to find the margins.

Poli Amlo

Support Vector Machine

For Further Reading I

B. Schlkopf and A. Smola.


Learning with Kernels.
The MIT Press, 2002.
M. Hearst.
Support vector machines.
IEEE Intelligent Systems,pag :18-28, 1998.

Poli Amlo

Support Vector Machine