Support Vector Machine

IOTGYAN SUMMER
INTERNSHIP
Support Vector Machines
Support vector machines (SVMs) are a particularly powerful and flexible class of supervised algorithms
for both classification and regression.
kNN provides a generative classification. However rather than modeling each class it would be simple to find a line or
curve(two dimensions) or manifold(multiple dimensions) that divides the classes from each other.
Consider the simple case of a classification task, in which the two classes of points are well separated:

A linear discriminative classifier would attempt to draw a straight line separating the two sets of data, and thereby
create a model for classification. There is more than one possible dividing line that can perfectly discriminate between
the two classes.
Depending on the line chosen, a new data point (e.g., the one marked by the "X in this plot) will be assigned a
different6/25/2018
label SMARTBRIDGE 2
 Challenge in the scenario
Simple intuition of "drawing a line between classes" is not enough, and we need to think a bit deeper.
Solution
Support vector machines offer one way to improve on this. The intuition is this: rather than simply drawing a zero-
width line between the classes, we can draw around each line a margin of some width, up to the nearest point.

In support vector machines, the line that maximizes this margin is the one we will choose as the optimal model
Support vector machines are an example of such a maximum margin estimator.

6/25/2018 SMARTBRIDGE 3
Example
The dividing line maximizes the margin between the two sets of points.
A few of the training points just touch the margin: they are indicated by the black circles in this figure. These
points are the pivotal elements of this fit, and are known as the support vectors, and give the algorithm its
name.

6/25/2018 SMARTBRIDGE 4
A Support Vector Machine (SVM) can be imagined as a surface that defines a
boundary between various points of data which represent examples plotted in
multidimensional space according to their feature values.
The goal of an SVM is to create a flat boundary, called a hyperplane, which leads to
fairly homogeneous partitions of data on either side.
SVMs are used to model highly complex relationships.
SVMs can be adapted for use with nearly any type of learning task, including both
classification and numeric prediction.
Application areas include:
✓Classification of microarray gene expression data in the field of
bioinformatics to identify cancer or other genetic diseases
✓Text categorization, such as identification of the language used in a
document or organizing documents by subject matter
✓The detection of rare yet important events like combustion engine
failure, security breaches, or earthquakes
6/25/2018 SMARTBRIDGE 5
 Classification with hyperplanes
Typically by class values, SVMs use a linear boundary called hyperplane to partition data into groups.
Ex:

The task of SVM algorithm is to identify a line that separates the two classes.

6/25/2018 SMARTBRIDGE 6
nding Maximum Margin
The Maximum Margin Hyperplane creates the greatest separation between the two classes.
i.e. the line that leads to the greatest separation will generalize the best to future data.
The support vectors are the points that are closest to the MMH.
Each class must have at least one support vector.
SVMs provide a compact way to store a classification model even if the number of features is extremely large.

6/25/2018 SMARTBRIDGE 7
Non-linearly separable data
Data might be not linearly separable most of the times.
In such case slack variable van be used which creates a soft margin that allows points to fall on the incorrect side
of the margin.

A cost value is applied to all points that violate the constraints and rather than finding the maximum margin, the
algorithm attempts to minimize the total cost.
Using kernels for non-linear spaces
Relationships between variables are non-linear in most of the problems.
SVM can still be trained on such data through addition of a slack variable.
SVMs map the problem into a higher dimension space using a process known as the kernel trick.
6/25/2018 SMARTBRIDGE 8
 Example
In the example below scatterplot depicts a non-linear relationship between a weather class(sunny or snow) and two features: Latitude and
Longitude.
Points at the center of the plot are members of snowy class and points at the margins are all sunny.

After the kernel trick has been applied, a dimension called altitude gives a new perspective to the data where the features are perfectly
linearly separable.
SVMs with non-linear kernels add additional dimensions to the data in order to create separation.
The kernel trick adds new features that express mathematical relationship between measured characteristics.
Ex:
Altitude feature can be expressed mathematically relationship between latitude and longitude.
i.e. the closer
6/25/2018 SMARTBRIDGE 9