Chapter 7:

Regularization for
Deep Learning

Deep Learning Textbook Study Group, SF

Safak Ozkan
April 15, 2017

1 Safak Ozkan
 Chapter 7: Regularization for Deep Learning

L2 Parameter Regularization
L1 Parameter Regularization
Norm Penalties and Constrained Optimization
Regularization and Under-Constrained Problems
Dataset Augmentation
Noise Robustness
Injecting Noise at Output Targets
Early Stopping
Semi Supervised Learning
Multi-Task Learning
Parameter Tying and Parameter Sharing
Bagging and Other Ensemble Methods
Dropout
Adversarial Training
Tangent Distance, Manifold Tangent Classifier
2 of 13 Safak Ozkan
 Definition

Regularization is any modification we make to a

learning algorithm that is intended to reduce its
test error but NOT its training error.

Etrain : Training Error Etest : Test Error

(or Generalization Error)

3 of 13 Safak Ozkan
 L2 Regularization
(a.k.a. Weight decay, Tikhonov regularization, Ridge regression)

Regularization increases bias and reduces variance.

Regularization
parameter

Regularized Regularization term

cost function Unregularized
Cost function

Gradient Descent update rule:

Additional term

4 of 13 Safak Ozkan
 L2 Regularization

Lagrangian Constrained Optimization

Lagrangian
multiplier

is equivalent to optimizing
such that .

5 of 13 Safak Ozkan
 L2 Regularization

Lagrangian Constrained Optimization

We typically dont set explicitly,
We set .

Unregularized
solution

Regularized Large small

solution constraint region

Large

6 of 13 Safak Ozkan
 L2 Regularization

2nd degree Taylor Approximation of around :

unregularized
problem

At ,

Analysis through e-vector decomposition

Stretching in i th small eigen-directions will be affected

eigen direction: more than larger eigen-directions.

7 of 13 Safak Ozkan
 L2 Regularization
Normal Equations for Linear Regression

Assume:

Then, would shrink

more than components.

covariance of input features

covariance of
with the target values.
input features

regularization causes the learning algorithm to

perceive the input with increased variance.

8 of 13 Safak Ozkan
 L1 Regularization
(a.k.a. LASSO)

Regularization
Term

2nd degree Taylor Approximation

of around :

(Induces
Sparsity)

9 of 13 Safak Ozkan
 Under-Constrained Problems
E.g. Logistic Regression

Linearly non seperable Linearly separable

Well behaved problem. Under-determined problem.

( will continue to increase
in a GD Algorithm)

10 of 13 Safak Ozkan
 Data Augmentation
Best way to improve generalization of a model is
to train it on more data.
Data Augmentation works particularly well for
Object Recognition tasks.
Injecting noise to input works well for
Speech Recognition.
Affine Elastic
Distortion Noise Deformation
Original
Input Image

Horizontal Random Hue

Flip Translation Shift

11 of 13 Safak Ozkan
 Noise Robustness
Addition of noise with a small variance is
equivalent to imposing norm penalty on weights.
Noise on weights: A stochastic implementation of
Bayesian Inference (uncertainty on weights are
represented by a probability distribution)

For each input data,

apply noise on weights

modified cost
function

regularization term

12 of 13 Safak Ozkan
 Early Stopping

regularization
number of parameter
learning rate
steps
13 of 13 Safak Ozkan
 Early Stopping
HAPTER 7. REGULARIZATION FOR DEEP LEARNING
Early stopping: Terminate while validation set
performance is better

0.20
Loss (negative log-likelihood)

Training set loss

0.15 Validation set loss

0.10

0.05

0.00
0 50 100 150 200 250
Time (epochs)

gure 7.3: Learning curves showing how the negative log-likelihood loss changes o
14 of 13 Safak Ozkan