Beruflich Dokumente
Kultur Dokumente
Lecture 3a
Learning the weights of a linear neuron
Geoffrey Hinton
with
Nitish Srivastava
Kevin Swersky
weight
vector
y = wi xi = wT x
i
neurons
estimate of the
desired output
input
vector
portions
of fish
50
100
portions
of chips
portions of
ketchup
wi = xi (t y)
50
2
portions
of fish
50
5
portions
of chips
50
3
portions of
ketchup
E=
1
2
(t n y n )2
ntraining
E 1 y n dE n
= 2
n
wi
w
dy
i
n
= xin (t n y n )
n
wi =
E
= xin (t n y n )
wi n
Nitish Srivastava
Kevin Swersky
w1
w2
w1
w1
w2
constraint from
training case 2
w2
w1
w2
Nitish Srivastava
Kevin Swersky
Logistic neurons
These give a real-valued
output that is a smooth
and bounded function of
their total input.
They have nice
derivatives which
make learning easy.
y=
z = b + xi wi
1+ e
y 0.5
0
z = b + xi wi
i
z
= xi
wi
y=
1+ e
z
= wi
xi
dy
= y (1 y)
dz
1
1+ e
= (1 + ez )1
" 1
% " ez %
dy
1(ez )
' = y(1 y)
=
= $
' $$
'
z
2
z
z
dz
#
&
(1 + e )
1+ e
#1 + e &
ez
(1+ ez ) 1
(1+ ez ) 1
=
=
= 1 y
because
z
z
z 1 + ez
1+ e
1+ e
1+ e
y
z dy
=
= xi y (1 y)
wi
wi dz
delta-rule
E
=
wi
y n E
n n
n
n
n
=
x
y
(1
y
)
(t
y
)
i
n
wi y
n
extra term = slope of logistic
Nitish Srivastava
Kevin Swersky
output units
hidden units
input units
E =
1
2
(t j y j )2
joutput
E
= (t j y j )
y j
E
y j
E
yi
Backpropagating dE/dy
yj
j
zj
yi
i
dy j E
E
E
=
= y j (1 y j )
z j
dz j y j
y j
E
=
yi
dz j E
dy z =
i
j
j
wij
j
z j E
E
E
=
= yi
wij
wij z j
z j
E
z j
Nitish Srivastava
Kevin Swersky
w1
w2
output = y
input = x
Which output value should
you predict for this test input?