Deep Learning

Andrew Ng

Thanks to Adam Coates, Kai Yu, Tong Zhang, Sameep Tandon,

Swati Dube, Brody Huval, Tao Wang, .

Andrew Ng
Virtuous circle of AI

Great product

AI!

Data Users

Andrew Ng
Data and machine learning

New AI methods
Performance!

(deep learning)

Most learning
algorithms

Amount of data!
Andrew Ng
Virtuous circle of AI

Great product

AI!

Data Users

Andrew Ng
Deep Learning

Adam Coates, Yoshua Bengio, Tom Dean, Jeff Dean, Nando de

Freitas, Jeff Hawkins, Geoff Hinton, Quoc Le, Yann LeCun,
Honglak Lee, Tommy Poggio, Ruslan Salakhutdinov, Yoram
Singer, Josh Tenenbaum, Kai Yu, Tong Zhang, . !
Andrew Ng
Things we want to do with data

Images! Label image!

Audio! Speech recognition!

Text! Web search!

Andrew Ng
STanford AI Robot (STAIR)

Andrew Ng
Computer vision: Identify coffee mug

Andrew Ng
Computer vision: Identify coffee mug

Mug Mug
Mug
Mug
Mug

Mug

Mug

Andrew Ng
Why is computer vision hard?

The camera sees :!

Andrew Ng
Computer vision

Learning
algorithm

Andrew Ng
Computer vision

Feature Learning
representation algorithm

Andrew Ng
Features for vision

SIFT! GIST! Shape context!

Andrew Ng
Features for machine learning

Images!

Image! Vision features! Detection!

Audio!

Audio! Audio features! Speaker ID!

Text! Web search!

!

Text! Text features!

Andrew Ng
Why is speech recognition hard?
Microphone recording:

Please find the coffee mug

Andrew Ng
Features for audio

Spectrogram! MFCC! Flux!

Andrew Ng
Features for text

Parser! Named entity! Stemming!

Andrew Ng
 The idea:

Most perception (input processing) in the

brain may be due to one learning algorithm.

Andrew Ng
The idea:

Build learning algorithms

that mimic the brain.

Most of human intelligence may

be due to one learning algorithm.

Andrew Ng
The one learning algorithm hypothesis

Auditory Cortex

Auditory cortex learns to see

[Roe et al., 1992]

Andrew Ng
The one learning algorithm hypothesis

Somatosensory Cortex

Somatosensory cortex learns to see

[Metin & Frost, 1989]

Andrew Ng
Neurons in the brain

Andrew Ng
Neural Network (Deep Learning)

x1

x2
Output
x3

Layer L4
x4
Layer L3
Layer L1 Layer L2

Andrew Ng
Deep Learning trends
Now

0-2 years 3-5 years

Tagged data Tagged & untagged data

Andrew Ng
Learning from tagged data (supervised)

Coffee mug Coffee mug Coffee mug

Coffee mug
Coffee mug Coffee mug

Testing: What is this?

Andrew Ng
Bigger is better
Performance

Size of neural network

[ Adam Coates]
Andrew Ng
Google Brain

Andrew Ng
AI as a computer systems problem

10 million connec+ons

1 billion connec+ons

Andrew Ng
Speech recognition, and more.

[with Vincent Vanhoucke]

Andrew Ng
Deep Learning applications

Speech recognition Image Search Ads; Web search

Andrew Ng
Tagged vs. untagged data

Coffee mug Coffee mug Coffee mug

Coffee mug
Coffee mug Coffee mug

Andrew Ng
Untagged data (unsupervised learning)

Unknown Unknown Unknown

Unknown Unknown Unknown

Andrew Ng
How does the brain process images?

Visual cortex looks for lines/edges.

Neuron #1 of visual cortex! Neuron #2 of visual cortex!

(model)! (model)!

Andrew Ng
 Start with Image patches

50

100

150

200

250

300

350 R14 x 14
400

450

500
50 100 150 200 250 300 350 400 450 500

Andrew Ng
! Sparse Coding

Input: Images patches x(1), x(2), (each in R14 x14)!

Learn: Set of matrices 1, 2, , 64 (also R14 x 14), so

that each input x can be approximately written as
a weighted sum of the js: !
64!
! ! !x aj j!
j=1!

! ! !s.t. ajs are mostly zero (sparse) !

!
Andrew Ng
 Sparse Coding
1, 2, , 64!

Test example

0.8 * + 0.3 * + 0.5 *

x! 0.8 * 36 + 0.3 * 42 + 0.5 * 63

Andrew Ng
Comparing to Biology

Brain (visual cortex) Learning algorithm

Andrew Ng
Comparing to Biology

[Evan Smith]
Andrew Ng
Comparing to Biology

[Evan Smith]
Andrew Ng
Learning from YouTube videos

Unknown Unknown Unknown

Unknown Unknown Unknown

Andrew Ng
Face neuron

[Le et al., 2012]

Andrew Ng
Cat neuron

[Le et al., 2012]

Andrew Ng
Deep Learning
object models!

object parts!

edges!

pixels!

Andrew Ng
16,000 CPUs is expensive

Andrew Ng
GPUs (Graphics Processor Unit)

[Adam Coates, Bryan Catanzaro, et al.]

Andrew Ng
Building huge neural networks

10 million connec+ons

1 billion connec+ons

10 billion connec+ons

[Adam Coates, Bryan Catanzaro]

Andrew Ng
Learning from tagged data

Deep learning
Performance!

Older
algorithms

Amount of data!
Andrew Ng
Highway perception

Andrew Ng
Deep Learning trends

0-2 years 3-5 years

Tagged data Tagged & untagged data

Andrew Ng
Untagged data and AI (unsupervised learning)

Andrew Ng
Computer vision (~6 years)

Feature
extraction! Prediction! Combine!
(SIFT)! (SVM)! (Maxpool)!
Deep Learning Deep Learning

Deep Learning
Andrew Ng
Speech recognition (next 2-3 years?)

Acoustic Language
model! Inference!
model!
(GMM)! (HMM)!
(ngram)!
Deep Learning Deep Learning

Deep Learning

Andrew Ng
AI will transform the internet

Speech Images Text

Andrew Ng
Speech Images

Andrew Ng
Speech recogni+on

Andrew Ng
Baidu Cool Box

Andrew Ng
Andrew Ng
35.5% 15.8%

Andrew Ng
 Baidu
Encyclopedia
Encyclopedia

Hyacinth

Flower shop

Andrew Ng
Image queries

Clothing Bags Fruits & Vegetables

Andrew Ng
Baidu Eye

Andrew Ng
Andrew Ng
Smart glasses designs

Andrew Ng
Extending human percep+on

Andrew Ng
Extending human percep+on

Andrew Ng
Comparison to smart glasses designs

Andrew Ng
Speech Images Text

Andrew Ng
From Control to Perception

Andrew Ng
Stanfords PR-1 robot

[Ken Andrew
Salisbury]
Ng
AI will transform the internet
Technology areas with potential for paradigm shift:
Computer vision
Speech recognition & speech synthesis
Language understanding: Machine translation;
Web search; Dialog systems; .
Advertising
Personalization/recommendation systems
Robotics
All this is hard: scalability, algorithms.

Andrew Ng
Virtuous circle of AI

Great product

AI!

Data Users

Andrew Ng
The AI mission

Andrew Ng
 Thank you.

Thanks to Adam Coates, Yu Kai, Zhang Tong, Sameep Tandon,

Swati Dube, Brody Huval, Tao Wang, .

Tutorial: deeplearning.stanford.edu
Andrew Ng
END END
END END
Andrew Ng
 Discussion:
Engineering vs. Data
Contribution to performance!

Human! Data/!
ingenuity! learning!

Andrew Ng
 Discussion:
Engineering vs. Data
Contribution to performance!

Time!
Now!
Andrew Ng
Correctly found mug Mistake
Andrew Ng
Correctly found mug Mistake
Andrew Ng
Correctly found mug Mistake
Andrew Ng
Correctly found mug Mistake
Andrew Ng