CMSC498L

Introduction to Deep Learning

Abhinav Shrivastava
Labels and Losses
Credit: blog
 What about these?

source
 What about images?
Input: 100x100x3 pixels
Output: Cat

Input: 100x100x3 pixels

Output: Dog

Credit: Krähenbühl, source: cat, dog

 What about images?
Input: 100x100x3 pixels
Output: Cat

Input: 100x100x3 pixels

Output: Dog

Credit: Krähenbühl, source: cat, dog

 How can we make machines learn this?
Input: 100x100x3 pixels
Output: Cat

Input: 100x100x3 pixels

Output: Dog

Credit: Krähenbühl, source: cat, dog

 How can we make machines learn this?
Input Data ! (()

! (#)

! (%)

! (&)

! (')

Credit: Krähenbühl, source: google image search

 How can we make machines learn this?
Input Data ! ()) Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 How can we make machines learn this?
Input Data ! ()) Model f Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

*(!, ,)
! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 How can we make machines learn this?
Input Data ! ()) Model f Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

*(!, ,)
! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 How can we make machines learn this?
Input Data ! ()) Model f Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

*(!, ,)
! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 How can we make machines learn this?
Input Data ! ()) Model f Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

*(!, ,)
! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 Model: Deep Neural Network

!(#, %)

ReLU

ReLU

ReLU

ReLU

ReLU
Input Layers of computation Output

Credit: Krähenbühl
 Model: Deep Neural Network

!(#, %)

ReLU

ReLU

ReLU

ReLU

ReLU
Input Layers of computation Output

Credit: Krähenbühl
 Model: Deep Neural Network

!(#, %) Model Parameters or Weights

ReLU

ReLU

ReLU

ReLU

ReLU
Input Layers of computation Output

Credit: Krähenbühl
 Training Model Weights
Input Data ! ()) Model f Output Label ( ())

! (#) Cat ( (#)

! (%) Dog ( (%)

*(!, ,)
! (&) Sheep ( (&)

! (') Pony ( (')

Credit: Krähenbühl, source: google image search

 Training Model Weights

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Credit: Krähenbühl
 Training Model Weights

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

Credit: Krähenbühl
 Training Model Weights

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

Provide signal to
improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label

Provide signal to
improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

Provide signal to
improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

• >0 for incorrect label Provide signal to

improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

• >0 for incorrect label )( (') ! = ) (')

Provide signal to
improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

• >0 for incorrect label )( (') ! = ) (')

Provide signal to
• Monotonically increasing improve

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

• >0 for incorrect label )( (') ! = ) (')

Provide signal to
• Monotonically increasing improve
• Important for gradient-based optimization

Credit: Krähenbühl
 Loss Functions

Data # (') Output )( (') Label ) (')

!(#, %) Dog Cat

Loss

• Zero for correct label )( (') == ) (')

• >0 for incorrect label )( (') ! = ) (')

Provide signal to
• Monotonically increasing improve
• Important for gradient-based optimization

Credit: Krähenbühl
Labels and Losses
 Labels

Credit: source
 Types of Labels
data

Credit: Krähenbühl, Source: Pac-Man fans

 Types of Labels
data dangerous class finer-class points actions
good pac-man pac-man 0.0 -
good bonus dot bonus 1.0 + points
bad bad guy clyde -1.5 die
bad bad guy blinky -1.9 die
good bonus cherry 100.0 +points, all disabled
good bonus strawberry 105.0 +points, all disabled
good bonus apple 300.2 +points, inky disabled
bad bad guy inky -10.5 die
Credit: Krähenbühl, Source: Pac-Man fans
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
Types of Labels
continuous/discrete *structured, rewards, etc.
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
Types of Labels
*structured, rewards, etc.

Image Word
2D/3D Pose Estimation

Sentence Parse tree

Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
 Regression
data points
0.0
1.0
-1.5
!(#, %) -1.9
100.0
105.0
300.2
-10.5
Credit: Krähenbühl, Source: Pac-Man fans
 Regression

Data # (') !(#, %) Output )( (') Label ) (')

• Continuous label ) (')

• Continuous network output )( (')

Credit: Krähenbühl
 Regression Loss

Data # (') !(#, %) Output )( (') Label ) (')

Loss
• Continuous label ) (')
• Continuous network output )( (')
• Loss

Credit: Krähenbühl
 Regression Loss

Data # (') !(#, %) Output )( (') Label ) (')

Loss
• Continuous label ) (')
• Continuous network output )( (')
• Loss
• L1 / MAE
• L2 / MSE / Least Squares
Credit: Krähenbühl
 Regression Loss

Data " (#) !(", %) Output '* (#) Label ' (#)

Loss
• Continuous label ' (#)
• Continuous network output '* (#)
• Loss
• L1 / MAE ! " # ,% − ' #

# # +
• L2 / MSE / Least Squares ! " , % − ' +

Credit: Krähenbühl
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
 Multi-class Classification
data multi-class
pac-man
dot bonus
clyde
!(#, %) blinky
cherry
strawberry
apple
inky
Credit: Krähenbühl, Source: Pac-Man fans
 Multi-class Classification
(#)
• Discrete label !
(#)
• Continuous network output !
%

pac-man
blinky
cherry
strawberry
… …

Credit: Krähenbühl
 Multi-class Classification via Regression
(#)
• Discrete label !
(#)
• Continuous network output !
%

0. pac-man
1. blinky
2. cherry
3. strawberry

Credit: Krähenbühl
 Multi-class Classification via Regression
(#)
• Discrete label !
(#)
• Continuous network output !
%

0. pac-man
1. blinky
2. cherry
3. strawberry

Credit: Krähenbühl
 Multi-class Classification via Regression
(#)
• Discrete label !
(#)
• Continuous network output !
%

0. pac-man
1. blinky
&(', )) 1. blinky
2. cherry
3. strawberry

Credit: Krähenbühl
 Multi-class Classification via Regression
(#)
✗
• Discrete label !

✗
(#)

✗
• Continuous network output !
%

0. pac-man
1. blinky
&(', )) 1. blinky
2. cherry
3. strawberry

Credit: Krähenbühl
 Multi-class Classification via Regression + 1-hot
(#)
• Discrete label !
(#)
• Continuous network output !
%

[1 0 0 0] pac-man
[0 1 0 0] blinky
&(', ))
[0 0 1 0] cherry
[0 0 0 1] strawberry

Credit: Krähenbühl
 Multi-class Classification via Regression + 1-hot
(()
• Discrete label '
(()
• Continuous network output '
)

[1 0 0 0] pac-man
[0 1 0 0] blinky
!(#, %)
[0 0 1 0] cherry
[0 0 0 1] strawberry

Credit: Krähenbühl
 ✗
Multi-class Classification via Regression + 1-hot
(()
• Discrete label '

✗
(()
• Continuous network output '
)

✗
[1 0 0 0] pac-man
[0 1 0 0] blinky
!(#, %)
[0 0 1 0] cherry
[0 0 0 1] strawberry

Credit: Krähenbühl
 Multi-class Classification
($)
• Discrete label "
• Regress to class probability & ' = " $

• Continuous network output "! ($)

• One continuous output per class &(' = blinky)
P(pac-man)
P(blinky)
P(cherry)
P(strawberry)
Credit: Krähenbühl
 Multi-class Classification
($)
• Discrete label "
• Regress to class probability & ' = " $

• Continuous network output "! ($)

• One continuous output per class &(' = blinky)
P(pac-man) = 0.5
P(blinky) = 0
/(0, 2)
P(cherry) = 0.5

Credit: Krähenbühl
P(strawberry) = 0
 Multi-class Classification – Softmax
• Probability
• Positive
• Sums to 1 P(pac-man)
P(blinky)
!(#, %)
P(cherry)
P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax
• Probability
• Positive z
• Sums to 1 10.18 P(pac-man)
12.91 P(blinky)
!(#, %)
-12.38 P(cherry)
18.19 P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax
• Probability
• Positive z
• Sums to 1 10.18 P(pac-man)
12.91 P(blinky)
!(#, %) exp norm
-12.38 P(cherry)
18.19 P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax
• Probability exp '(
! " =
• Positive ∑* exp '*
• Sums to 1

Credit: Krähenbühl
 Multi-class Classification – Softmax
• Probability exp '(
! " =
• Positive ∑* exp '*
• Sums to 1

Credit: Krähenbühl; refer to blog

 Multi-class Classification – Softmax Loss
• Maximum likelihood
• Minimize negative log-probability (NLL)
• Cross entropy
• Softmax-loss − log %(')
• log is numerically stable

Credit: Krähenbühl; refer to blog

 Multi-class Classification – Softmax Loss

− log %(')

Credit: Krähenbühl; refer to blog

 Multi-class Classification – Softmax Loss
P(pac-man)
P(blinky)
!(#, %)
P(cherry)
P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax Loss

P(pac-man)
P(blinky)
!(#, %)
P(cherry)
P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax Loss

P(pac-man)
P(blinky)
!(#, %)
P(cherry)
P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax Loss

P(pac-man)
P(blinky)
!(#, %)
P(cherry)
P(strawberry)

Credit: Krähenbühl
 Multi-class Classification – Softmax Loss

− log %(')

Credit: Krähenbühl; refer to blog

Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
 Binary Classification
data multi-class
good
good
bad
!(#, %) bad
good
good
good
bad
Credit: Krähenbühl, Source: Pac-Man fans
 Binary Classification
($)
• Discrete label "
• Regress to class probability & ' = " $

• Continuous network output "! ($)

P(good)
• Single output &(' = class)
P(bad) = 1 - P(good)
• Positive: class 1
• Negative: class 0

Credit: Krähenbühl
 Binary Classification – Sigmoid
• Probability 1
! "=0 =
• Positive 1 + exp(+)
• Sums to 1
1
! "=1 =
1 + exp(−+)

Credit: Krähenbühl
 Binary Classification – Loss
• Maximum likelihood
• Minimize negative log-probability (NLL)
• Cross entropy
• Softmax-loss − log %(')
• log is numerically stable

Credit: Krähenbühl
Sigmoid is a special case of Softmax
• How?
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class
Other types of losses
 Classification (max-margin)
SVM Loss

Credit: Krähenbühl, image credit

 Embeddings

Credit: Krähenbühl, image credit1, credit2

Embeddings
Embedding Learning

After training
Embedding Learning: Triplet Loss

! − <$ ! − <$

! − >$ ! − >$

! − <$

! − >$
Types of Labels
continuous/discrete
continuous discrete

regression how many classes?

examples:
• weight
2 >2
• age
• location binary classification multi-class classification
• velocity
examples: examples:
• distance
• dog vs. cat • Semantic classes
• score
• danger • actions
• price
• alive • etc.
• bood vs. bad
• class vs. not-class