Pattern Association

Klinkhachorn:CpE320

Associative Memory Neural Nets

An associate network is a neural network with essentially a single functional layer that associates one set of vectors with another set of vectors The weights are determined in such a way that the net can store a set of pattern associations Each association is an input-output vector pair, s:t

Klinkhachorn:CpE320

Associative Memory (Cont.)

If each vector t is the same as the vector s which it is associated, then the net is called an

autoassociative memory
If the ts are different from the ss, the net is called a heteroassociative memory The net not only learns the specific pattern pairs that were used for training, but also is able to recall the desired response pattern when given an input stimulus that is similar but not identical, to the training input.
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall Klinkhachorn:CpE320

Associated Pattern Pairs

Klinkhachorn:CpE320

Partial Data Presented

Who? Astro

Rosie

Look like?

Klinkhachorn:CpE320

Noisy Data Presented

Klinkhachorn:CpE320

Noisy Data Presented

Klinkhachorn:CpE320

Classification of ANN Paradigms

Klinkhachorn:CpE320

Architecture of an Associative memory Neural Net

Feedforward net: information flows from the input units to the output units Example: Linear associator Network Recurrent (iterative): there are connections among the units that form closed loop Example: Hopfield, BAM

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb Rule for Pattern Association

Simplest and most common method of determining the weights for an associative memory neural net Can be used with patterns that are represented as either binary or bipolar vectors x1 w1 xn wn
Klinkhachorn:CpE320

e = wixi

y = 1 if e 0 = 0 if e < 0

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Algorithm
Step 1: Initialize all weights (i=1,.,n; j = 1..,m) wij = 0 Step 2: For each input training-target output vector pair s:t, do Steps 3-5. Step 3: Set activations for input units to current training input (i = 1,.,n): xi=si Step 4: Set activations for output units to current target output (j = 1,.,m): yj = tj Step 5: Adjust the weights( I=1,n; j=1,.,m): wij(new) = wij(old) +xiyi
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall Klinkhachorn:CpE320

Weights Calculation: Outer Product

Let s-t be the input vector-output vector pairs The outer product of two vectors s = (s1,..,si,.,sn) and t = (t1,...,tj,.,tm) Is the matrix product of the n x 1 matrix S = sT and the 1 x m matrix T = t:

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Weights Calculation: Outer Product (Cont)

To store a set of associations s(p) : t(p), p=1,.,P, where s(p) = (s1(p),.,si(p),.,sn(p)) and t(p) = (t1(p),.,tj(p),.,tm(p)) The weight matrix W = {wij} is given by wij = si(p)tj(p)
p =1 P

This is the sum of the outer product matrices required to store each association separately. In general, W = sT(p)t(p) = sTt
p =1 P

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Perfect recall VS cross talk Hebb rule:

If the input vectors are uncorrelated (orthogonal), the Hebb rule will produce the correct weights, and the response of the net when tested with one of the training vectors will be perfect recall If the input vectors are not orthogonal, the response will include a portion of each of their target values. This is commonly called cross talk

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Suppose a Heteroassociative net is to be trained to store the following mapping from input row vectors s = (s1,s2,s3,s4) to output row vectors t = (t1,t2):

Pattern 1st s 2nd s 3rd s 4th s

s1,s2,s3,s4 (1, 0, 0, 0) (1, 1, 0, 0) (0, 0, 0, 1) (0, 0, 1, 1)

t1,t2 (1, 0) (1, 0) (0, 1) (0, 1)

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

x1 x2 x3 x4 w21 w22 w31 w32 w41 w42 w11 w12 x1 x1

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Weights calculation: Outer Product W = sT(p)t(p) = sTt
p =1 4

1 0 W= 0 0

1 1 0 0

0 0 0 1

0 1 0 1 . 1 0 1 0

0 2 0 1 = 1 0 1 0

0 w11 0 w21 = 1 w31 2 w41

w12 w22 w32 w42

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net for x = (1,0,0,0)

2 1 xW = (1,0,0,0) 0 0

0 0 = (2,0) > (1,0) 1 2

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net - input similar to the training input for x = (0,1,0,0) differs from the training vector (1,1,0,0) only in the first component.

2 1 xW = (0,1,0,0) 0 0

0 0 = (1,0) > (1,0) 1 2

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net - input not similar to the training input for x = (0,1,1,0) differs from the training input patterns in at least two components.

2 1 xW = (0,1,1,0) 0 0

0 0 = (1,1) > (1,1) 1 2

The two mistakes in the input pattern make it impossible for the net to recognize it!
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall Klinkhachorn:CpE320

Iterative Autoassociative Net

Klinkhachorn:CpE320

Hopfield Nets (Recurrent)

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

John Hopfield Computational neurobiology / biophysics

While the brain is totally unlike current digital computers, much of what it does can be described as computation. Associative memory, logic and inference, recognizing an odor or a chess position, parsing the world into objects, and generating appropriate sequences of locomotor muscle commands are all describable as computation. My current research focuses on the theory of how the neural circuits of the brain produce such powerful and complex computations. Olfaction is one of the oldest and simplest senses, and much of my recent work derives from considerations of the olfactory system. One tends to think of olfaction as "identifying a known odor," but in highly olfactory animals, the problems solved are much more complicated. Many animals use olfaction as a remote sense, to understand the environment around them in terms of identifying both where and what objects are remotely located. This involves at least coordinating when something is smelled with the wind direction, and untangling a weak

signal from an odor object from a background of other odors simultaneously present. The homing of pigeons or the ability of a slug to find favorite foods are examples of such remote sensing.

http://www.molbio.princeton.edu/faculty/hopfield.html

Klinkhachorn:CpE320

Binary Hopfield Processing Unit

x1 wj1 x2 x3 wjn wj2 wj3 ej=xiwji
i ij

yj = +1 if ej0 = 0 if ej<0 j

xn
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold Klinkhachorn:CpE320

Hopfield Architecture

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net Algorithm

Step1: Compute the connection weights, Let m be the total # of associated patterns, i.e. X = (x1,x2,xp,..,xm) and for each pattern p with n inputs xp = (xp1,xp2,,xpi,..xpn) Then, the connection weight from node i to node j, wij = (2xpi-1)(2xpj-1) . If xp =(0,1), and wii = 0
p=1
m m

or wij = xpixpj
p=1

. If xp =(-1,1), and wii = 0 . If xp =(-1,1), and wii = 0

Klinkhachorn:CpE320

or

W = XTX

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Hopfield Net Algorithm (Cont)

Step2: Initialize the network with unknown input pattern, x = (x1, x2,.. , xn), by assigning output yi(0) = xi, for i= 1 to n where yi(0) is the output of node i at time 0 Step 3: Iterate (update outputs) until convergence yj(t+1) = F[ wijyij(t)], j = 1 to n
i=1

where F[e] = +1. if e0, or = -1(0). if e<0

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Updating Procedure

Random Update Allows all units to have the same average update Sequence Update Repeat the sequence until a stable state is attained Most other neural network paradiagms have a layer of processing units updated at the same time (or nearly the same time). Hopfield, however, suggested that random update of the neuron has advantages both in implementation (each unit can generate its next update time) and in function (sequential updating can restrict the output states of the network in cases in which different stable states are equiprobable)
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold Klinkhachorn:CpE320

Hopfield Net: Updating Procedure

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Analysis

Convergence** Each state of the Hopfield network has an associated "Energy" value,
E= 1 i,jw ji x j xi 2 j j

Successive updating the network provides a convergence procedure thus the energy of the overall network gets smaller. Consider that unit j is the next processing unit to be updated,
1 1 E j = w ji x j xi = x j wji x i 2 i, i j 2 i ,i j

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Analysis

When unit j is updated,
1 E j = Einew E jold = x j w ji x i 2 where x j = x j new x jold

If xj changes from 0 to 1, then

x j = 1 and

w
i

ji

xi 0

Thus, E j 0
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold Klinkhachorn:CpE320

Hopfield Net: Analysis

If xj changes from 1 to 0, then
x j = 1 and

w
i

ji

xi < 0

Again, E j < 0

The network guaranteed to converge, when E taking on lower and lower values until it reaches a steady state.
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold Klinkhachorn:CpE320

Hopfield Net: Recalling

T+10% Noise -->T

T +20% Noise --> O

Klinkhachorn:CpE320

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Hopfield Net: Application

A binary Hopfield net can be used to determine whether an input vector is a known vector (i.e., one that was stored in the net) or an unknown vector. If the input vector is an unknown vector, the activation vectors produced as the net iterates will converge to an activation vector that is not one of the stored patterns; such a pattern is called a spurious stable state.

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Storage Capacity

Hopfield found experimentally that the number of binary patterns that can be stored and recalled in a net with reasonable accuracy, is approximately P 0.15 n, where n is the number of neurons in the net A detailed theoretical analysis showed P n/2 log2n

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320