Beruflich Dokumente
Kultur Dokumente
Single-Input Neuron
scalar input p * scalar weight w
bias b
its like a weight except that it has a constant input of 1
a neuron may have or may not have a bias
n - summer output (net input)
f - transfer function (activation function)
a - neuron output= f(wp+b)
Example
Consider the parameters: w = 3, p = 2, b = -1.5. What is the neuron output?
a = f(3*2 -1.5) = f(4.5), where f is chosen by the designer
Transfer Functions
The transfer function may be a linear or a nonlinear function of n. A particular transfer
function is chosen to satisfy some specification of the problem that the neuron is
attempting to solve. The most commonly used functions are discussed below
1- Hard limit (step) transfer function - used in perceptrons
1
0
a=f(n)=
n0
n0
1
1
a=f(n)=
n0
n0
a=n
3- Log-sigmoid and tan-sigmoid transfer functions, used in multilayer networks
trained with backpropagation
Multiple-Input Neuron
The following is a neuron with R inputs
In Matrix form
n = Wp + b
a= f(Wp + b)
Abbreviated Notation
Example
Given a two-input neuron with the following parameters: b = 1.2, W = [3 2] and
p=[ 5 6]T, calculate the neuron output for the following transfer functions:
A Symmetrical hard limit transfer function
A tangent sigmoid (tansig) transfer function
Solution
a = hardlims(1.8)= 0
a=tansig(1.8)=0.9468
Example
A single-layer neural network is to have six inputs and two outputs. The outputs are to
be limited to and continuous over the range 0 to 1. What can you tell about the
network architecture?
Specifically:
How many neurons are required?
What are the dimensions of the weight matrix?
What kind of transfer functions could be used?
Is a bias required?
Solution
The problem specifications allow you to say the following about the network.
Two neurons, one for each output, are required.
The weight matrix has two rows corresponding to the two neurons and six
columns corresponding to the six inputs. (The product Wp is a two-element
vector.)
The logsig transfer function would be most appropriate.
Not enough information is given to determine if a bias is required.
Example
A single-layer neural network that have two inputs and two outputs with the following
parameters:
3
b =[ 1.2 1]T, W = 1
calculate the neuron output for the transfer functions: Symmetrical Hard limit (step)
transfer function
Solution
1 .2
1.8
=
1
14
1.8
1
a = hardlims( 14 )= 1
3
1
n=Wp+b=
2
3
5
6
Perceptron
1958 - Frank Rosenblatt developed the perceptrons
Perceptron= Neuron +Learning algorithm
Solution
First construct the training set
Banana
Apple
Find e=t1-a=1-0=1
Applying p2:
find e=t2-a=0-1=-1
Updating the weights:
Example
a) Train by hand a perceptron with bias on this training set. Assume that all initial
weights (including the bias of the neuron) are 0. Show the set of weights
(including the bias) at the end of the each iteration. Apply the examples in the
given order. Use the hardlim step function. Stopping criteria: Patterns are correctly
classified.
b) How many epochs were needed? Is the training set linearly separable?
Solution
Check if the stopping criteria is satisfied each training example is applied to check if
it is correctly classified
1 epoch was needed to train the perceptron. The training set is linearly separable as
the perceptron was able to learn to separate it.
Adaline
ADALINEs (ADAptive LInear Neuron) use the Widrow-Hoff algorithm or Least
Mean Square (LMS) algorithm to adjusts the weights of the linear network in order to
minimize the mean square error
Error - difference between the target and actual network output
mean square error
-0.16] 1
1
= -1 + 0.64 = -0.36
-0.16] 1
1
= 1 1.28 = -0.28
mse=
(0.36) 2 (0.28) 2
=0.104>0.1
2
10
Learning Rate
Too big: The system will oscillate as the correction will be too large and will
overshoot the target
Too small: The system will take a long time to converge
A constant: may never converge to an unchanging value but will oscillate
around
The best solution: gradually drops toward zero. Typically used:
= Constant/n
n- number learning trials
Both ADALINE and perceptron suffer from the same inherent limitation - can
only solve linearly separable problems, for example the both can not be
trained to represent XOR function. LMS, however, is more powerful than the
perceptrons learning rule.
Perceptrons rule is guaranteed to converge to a solution that correctly
categorizes the training patterns but the resulting networkcan be sensitive to
noise as patterns often lie close to the decision boundary
Matlab
Example: Write Matlab code to implement the Adaline, and use it to train the
following data.
Input
Output(Target)
1,1,1
8,7,5
4,5,5
1,1,1
1,1
-1, -1
-1,-1
1,1
11
epoch=2000
[r_in,c_in] = size(p)
[r_tar,c_tar]=size(t)
w{1} = rand(r_tar, r_in);
b{1}=rand(r_tar,1);
k = 1;
for j=1 :epoch
for i=1:c_in
a= w{k}*p(:,i)+b{k} ;
e=t(:,i)-a;
if e==0
continue;
end
k = k + 1;
w{k} = w{k-1} + r*e*p(:,i)';
b{k}=b{k-1}+r*e ;
end
error=zeros(2,4);
for i=1 : c_in
error(:,i)=t(:,i)- (w{k}*p(:,i)+b{k});
end
x(j)=mse(error);
figure(1);
plot(x);
end
title('AdaLine','Fontsize',14,'color',[0 0 1] );
----------------------------------------------------------------------------------------
12
ans =
0.5668
0.6274
13