Artificial Neural Networks - 13

Dr. Aditya Abhyankar

Last Time!
BP Example
Dr. Dandawates lecture on VQ and SOFM

Today
BP -- continued !!!

Back-Propagation (BP)
Aims at balancing memorization and
generalization
Stage 1: Feedforward I/p training pattern
Stage 2: calculation and backpropagation of
associated error
Stage 3: Adjustments of the weights

Architecture

Hidden
Layer

Nomenclature

Nomenclature

Activation Function
Characteristics:

Continuous
Differentiable
Monotonically non-decreasing
Easily differentiable

Binary Sigmoid Function

range (0,1)

Bipolar Sigmoid Function

range (-1,1)

Algorithm: Training

Algorithm

Algorithm

Algorithm

Algorithm

Application

Example
X-or Problem (linearly not separable) using
2-4-1 backprop Net
Initial weights to
Initial Weights to
o/p layer
hidden layer

Solution
y1
z1

z2

x1

z3

x2

z4

Solution
Output
Layer

y1
Hidden
Layer

z1

z2

x1

z3

x2

z4

Input
Layer

Solution
Output
Layer

y1
Hidden
Layer

z1
v11

v12

z2

x1

v13

z3

x2

v14

z4

Input
Layer

Solution
Output
Layer

y1
Hidden
Layer

z1 v
21
v11

v12

z2 v v z3
v14 z4
22 13
v23
v24

x1

x2

Input
Layer

Solution
Output
Layer

y1
w11
Hidden
Layer

z1 v
21
v11

w21

v12

w31

w41

z2 v v z3
v14 z4
22 13
v23
v24

x1

x2

Input
Layer

Solution
w01

y1

w11

Hidden
Layer
v01
1

z1 v
21
v02

Output
Layer

v11
v03

v04

w21

v12

w31

w41

z2 v v z3
v14 z4
22 13
v23
v24

x1

x2

Input
Layer

Example
X-or Problem (linearly not separable) using
2-4-1 backprop Net
Initial weights to
Initial Weights to
o/p layer
hidden layer

Solution Training: step1

w01 0.1401
1

y1
w11

z1 v
21
v01
1

v02

v11
v03

v04

w21

v12

w31

w41

0.4919
0.2913
Output w

0.3979
Layer

0.3581

z2 v v z3
v14 z4
22 13
v23
v24

x1

x2

Solution Training: step1

w01 0.1401
1

y1
w11

z1 v
21
v01
1

v02

v11
v03

v04

w21

v12

w31

w41

0.4919
0.2913
Output w

0.3979
Layer

0.3581

z2 v v z3
v14 z4
22 13
v23
v24

x1

x2

v0 0.3378 0.2771 0.2859 0.3329

Solution Training: step1

w01 0.1401
1

y1
w11

z1 v
21
v01
1

v02

v11
v03
v04

w21

v12

w31

w41

0.4919
0.2913
Output w

0.3979
Layer

0.3581

z2 v v z3
v14 z4
22 13
v23
v24

x1

v0 0.3378 0.2771 0.2859 0.3329

x2
0.197 0.3191 0.1448 0.3394
v

0.3099
0.1904

0.0347

0.4861

Solution Training: step1

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

0.3581

y1

zin3

zin1 0.1691

z1

zin2
0.7866

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.1064

zin4

z3

0.4796

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step1

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

0.3581

y1

zin3

zin1 0.1691

z1

zin2
0.7866

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.1064

zin4

z3

0.4796

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step1

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

0.3581

y1

z3

z1 0.5422

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step1

yin1 0.1462

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

0.3581

y1

z3

z1 0.5422

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step1

yin1 0.1462

w01 0.1401

y1 0.4635

0.4919
0.2913
Output w

0.3979
Layer

0.3581

y1

z3

z1 0.5422

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.0023

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.0023

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.0023

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

-0.0567

1 in 0.0336
0.0459

-0.0413

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

x1

x2

0.197 0.3191 0.1448 0.3394

v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

yin1 0.1462

w01 0.0023

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer

k 0.1153 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012

-0.0009

z1 0.5422

y1 0.4635

z3

z1

-0.0567

1 in 0.0336
0.0459

-0.0413

z2
0.6871

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

z4

x1

x2

-0.0141
0.0072 0 0 0

j
2
0.0114

-0.0097

0.3394
0.197 0.3191 0.1448
v

0.3099
0.1904

0.0347

0.4861

Solution Training: step2

w01 0.0023

yin1 0.1462

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer
x2 t
k 0.1153 x 1 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012
z2

-0.0009
z1 0.6871
z1 0.5422
-0.0567
0.0336

in
0.0459
1

-0.0413

y1 0.4635

z3

z2

x1

v0 0.3378 0.2771 0.2859 0.3329

0.5266

z4

z3

0.3823

x2

z4

0.197 0.3191 0.1448 0.3394

0.3099 0.1904 0.0347 0.4861

-0.2815 0.1444 0.2287 -0.1950

v

-0.2815
0.1444
0.2287
-0.1950

* 1.0e-003

Solution Training: step2

yin1 0.1462

w01 0.0023

w01 0.1401

0.4919
0.2913
Output w

0.3979
Layer
x2 t
k 0.1153 x 1 0.3581

y1

1 -0.0012

-0.0016

w
-0.0012
z2

-0.0009
z1 0.6871
z1 0.5422
-0.0567
0.0336

in
0.0459
1

-0.0413

y1 0.4635

z3

z2

x1

0.5266

z4

z3

0.3823

x2

z4

0.197 0.3191 0.1448 0.3394

0.3099 0.1904 0.0347 0.4861

v0 0.3378 0.2771 0.2859 0.3329

-0.2815 0.1444 0.2287 -0.1950

v

v0 -0.2815 0.1444 0.2287 -0.1950

-0.2815
0.1444
0.2287
-0.1950

3
3

*10

*10

Solution Training: step3

yin1 0.1462

w01 0.0023

w01 0.1424

y1 0.4635

0.4907
-0.2929
Output w

-0.3991
Layer

k 0.1153 0.3572

y1

1 -0.0012

-0.0016
z3

w
z4
-0.0012
z2
0.5266

0.3823
-0.0009

z1 0.6871 z2
3
z1 0.5422
-0.0567
0.0336

in
-0.2815
0.0459
v
1
-0.2815

1
2
-0.0413
v0 -0.2815 0.1444 0.2287 -0.1950

v0 -0.3381

0.2772 0.2861 -0.3331

z4

x1

x2

0.1444 0.2287 -0.1950

0.1444 0.2287 -0.1950

* 1.0e-003

0.1967 0.3192 -0.1446 0.3392

0.3096 0.1905 -0.0345 -0.4863

Solution Training: step3

w01 0.1393

y1

z1

1
v0 -0.3377

0.4923
-0.2908
Output w

-0.3975
Layer

0.3584

z2

z3

x1
0.2770 0.2858 -0.3328

z4

x1

x2

x2
0.1970 0.3191 -0.1448 0.3394

0.3100 0.1904 -0.0347 -0.4861

Application