Chapter 6: Neural Network I (Adaline)

Chapter 6 Neural Network I

Adaline
Notation
scalars - small italic letters: a, b, c
vectors - small bold non-italic letters: a, b, c
matrices - capital BOLD non-italic letters: A, B, C

Single-Input Neuron
scalar input p * scalar weight w
bias b
its like a weight except that it has a constant input of 1
a neuron may have or may not have a bias
n - summer output (net input)
f - transfer function (activation function)
a - neuron output= f(wp+b)

Example
Consider the parameters: w = 3, p = 2, b = -1.5. What is the neuron output?
a = f(3*2 -1.5) = f(4.5), where f is chosen by the designer

Transfer Functions
The transfer function may be a linear or a nonlinear function of n. A particular transfer
function is chosen to satisfy some specification of the problem that the neuron is
attempting to solve. The most commonly used functions are discussed below
1- Hard limit (step) transfer function - used in perceptrons

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

1
0

a=f(n)=

n0
n0

Symmetrical Hard Limit

1
1

a=f(n)=

n0
n0

2- Linear Transfer Functions, used in ADALINE networks

a=n
3- Log-sigmoid and tan-sigmoid transfer functions, used in multilayer networks
trained with backpropagation

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Multiple-Input Neuron
The following is a neuron with R inputs

In Matrix form
n = Wp + b
a= f(Wp + b)

A single-layer network of S neurons

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Abbreviated Notation

How to Pick an Architecture

Problem specifications help define the network in the following ways:
Number of network inputs = number of problem inputs
Number of neurons in output layer = number of problem outputs
Output layer transfer function choice at least partly determined by problem
specification of the outputs

Example
Given a two-input neuron with the following parameters: b = 1.2, W = [3 2] and
p=[ 5 6]T, calculate the neuron output for the following transfer functions:
A Symmetrical hard limit transfer function
A tangent sigmoid (tansig) transfer function
Solution

a = hardlims(1.8)= 0
a=tansig(1.8)=0.9468

Example
A single-layer neural network is to have six inputs and two outputs. The outputs are to
be limited to and continuous over the range 0 to 1. What can you tell about the
network architecture?
Specifically:
How many neurons are required?
What are the dimensions of the weight matrix?
What kind of transfer functions could be used?
Is a bias required?
Solution
The problem specifications allow you to say the following about the network.
Two neurons, one for each output, are required.

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

The weight matrix has two rows corresponding to the two neurons and six
columns corresponding to the six inputs. (The product Wp is a two-element
vector.)
The logsig transfer function would be most appropriate.
Not enough information is given to determine if a bias is required.

Example
A single-layer neural network that have two inputs and two outputs with the following
parameters:
3

b =[ 1.2 1]T, W = 1

and p=[ 5 6]T,

calculate the neuron output for the transfer functions: Symmetrical Hard limit (step)
transfer function
Solution
1 .2
1.8
=

1
14
1.8
1
a = hardlims( 14 )= 1

3
1

n=Wp+b=

2
3

5
6

Perceptron
1958 - Frank Rosenblatt developed the perceptrons
Perceptron= Neuron +Learning algorithm

Perceptron Learning algorithm

Input: a set of training examples: {p1,t1}, {p2,t2},,{pn,tn}
Goal: classify all examples correctly
Steps
1- Randomly initialize the weights W and the biases b
2- Choose a random input-output pair {p,t} from the training set
3- Let the network to operate on the input to generate output a
4- Compute the output error e=t-a
5- Update weights:

Add a matrix W to the weight matrix W, which is proportional to the

product epT between the error vector and the input:

Add a vector b to the bias vector b, which is proportional to the error

vector:

6- Choose another random pair and do the correction again

7- Continue until the stopping criteria is satisfied: all examples are correctly
classified or a maximum number of epochs is reached

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Example - Apple/Banana Sorter

A produce dealer has a warehouse that stores a variety of fruits. He wants a machine
that will sort the fruit according to the type...There is a conveyer belt on which the
fruit is loaded it is then passed through a set of sensors, which measure 3 properties
of the fruit: shape, texture and weight.
Shape sensor: -1 if the fruit is round, 1 - if it is more elliptical
texture sensor: -1 if the surface is smooth, 1 - if it is rough
weight sensor: -1 if the fruit is > 500g , 1 - if < 500g
The sensor outputs will then be input to a NN...The purpose: is to recognize and
correctly sort the fruit. For simplicity - only 2 kinds of fruit (Bananas and Apples)

Solution
First construct the training set
Banana

Apple

Initial weights (random):

W = [0.5 1 0.5], b = 0.5
Applying p1:

a= hardlim( Wp1+ b)= hardlim 0.5 1

0.5 1 0.5 =hardlim(-0.5)=0

Find e=t1-a=1-0=1

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Updating the weights:

Applying p2:

a= hardlim( Wp1+ b)= hardlim 0.5 0

1.5 1 1.5 =hardlim(2.5)=1

find e=t2-a=0-1=-1
Updating the weights:

End of epoch; check if the stopping criteria is satisfied

The stopping criteria is satisfied. Stop

Example

a) Train by hand a perceptron with bias on this training set. Assume that all initial
weights (including the bias of the neuron) are 0. Show the set of weights
(including the bias) at the end of the each iteration. Apply the examples in the

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

given order. Use the hardlim step function. Stopping criteria: Patterns are correctly
classified.
b) How many epochs were needed? Is the training set linearly separable?
Solution

Check if the stopping criteria is satisfied each training example is applied to check if
it is correctly classified

1 epoch was needed to train the perceptron. The training set is linearly separable as
the perceptron was able to learn to separate it.

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Adaline
ADALINEs (ADAptive LInear Neuron) use the Widrow-Hoff algorithm or Least
Mean Square (LMS) algorithm to adjusts the weights of the linear network in order to
minimize the mean square error
Error - difference between the target and actual network output
mean square error

Adaline=Neuron +Learning algorithm

Adaline Learning algorithm

Input: a set of training examples: {p1,t1}, {p2,t2},,{pn,tn}
Goal: Error is small enough
Steps
8- Randomly initialize the weights W and the biases b
9- Choose a random input-output pair {p,t} from the training set
10- Let the network to operate on the input to generate output a
11- Compute the output error e=t-a
12- Update weights:

Add a matrix W to the weight matrix W, which is proportional to the

product epT between the error vector and the input:

Add a vector b to the bias vector b, which is proportional to the error

vector e:

13- Choose another random pair and do the correction again

14- Continue until the stopping criteria is satisfied: the performance measure
(error or accuracy) is small enough (below a threshold) or a maximum number of
epochs is reached.

Example - Apple/Banana Sorter

Learning rate: = 0.4
Stopping criteria: mse < 0.1
Solution
First construct the training set

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Initial weights (random):

W = [0 0 0], no bias

p1, e1= t1- a1 = -1 - [0.96 0.16

-0.16] 1
1

= -1 + 0.64 = -0.36

p1, e2= t2 - a2 = 1 - [0.96 0.16

-0.16] 1
1

= 1 1.28 = -0.28

mse=

(0.36) 2 (0.28) 2
=0.104>0.1
2

Stopping criteria is not satisfied => continue with apoch 2

10

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

Learning Rate

Too big: The system will oscillate as the correction will be too large and will
overshoot the target
Too small: The system will take a long time to converge
A constant: may never converge to an unchanging value but will oscillate
around
The best solution: gradually drops toward zero. Typically used:
= Constant/n
n- number learning trials

Capability and Limitations

Both ADALINE and perceptron suffer from the same inherent limitation - can
only solve linearly separable problems, for example the both can not be
trained to represent XOR function. LMS, however, is more powerful than the
perceptrons learning rule.
Perceptrons rule is guaranteed to converge to a solution that correctly
categorizes the training patterns but the resulting networkcan be sensitive to
noise as patterns often lie close to the decision boundary

Matlab
Example: Write Matlab code to implement the Adaline, and use it to train the
following data.
Input

Output(Target)

1,1,1
8,7,5
4,5,5
1,1,1

1,1
-1, -1
-1,-1
1,1

What is the output if the input is [1 1 2]'

----------------------------------------------------------------------------------------clear
% Training Rate
r=0.01
% Input Data (input*samples)
p =[1 8 4 1; 1 7 5 1; 1 5 5 1];
%Target Data (output *samples)
t = [1 -1 -1 1; 1 -1 -1 1];

11

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

epoch=2000
[r_in,c_in] = size(p)
[r_tar,c_tar]=size(t)
w{1} = rand(r_tar, r_in);
b{1}=rand(r_tar,1);
k = 1;
for j=1 :epoch
for i=1:c_in
a= w{k}*p(:,i)+b{k} ;
e=t(:,i)-a;
if e==0
continue;
end
k = k + 1;
w{k} = w{k-1} + r*e*p(:,i)';
b{k}=b{k-1}+r*e ;
end
error=zeros(2,4);
for i=1 : c_in
error(:,i)=t(:,i)- (w{k}*p(:,i)+b{k});
end
x(j)=mse(error);
figure(1);
plot(x);
end
title('AdaLine','Fontsize',14,'color',[0 0 1] );
----------------------------------------------------------------------------------------

The output for [1 1 2]' is

(w{k}*[1 1 2]'+b{k})

12

Dr. Essam Al Daoud

Chapter 6: Neural Network I (Adaline)

ans =
0.5668
0.6274

13

Dr. Essam Al Daoud