NN 5

HOP
Associative Memories

• We consider now NN models for unsupervised learning

problems, called auto-association problems.
• Association is the task of mapping patterns to patterns.
• In an associative memory the stimulus of an incomplete or
corrupted pattern leads to the response of a stored pattern
that corresponds in some manner to the input pattern.
• A neural network model most commonly used for (auto-)
association problems is the Hopfield network.

NN 5 1

Example HOP

Corrupted image Intermediate state of Output

Hopfield net
• States Bit maps.
• Attractors Prototype patterns.
• Input An arbitrary pattern
(e.g. picture with noise).
• Output The best prototype for that pattern.
NN 5 2

Neural Networks 1
NN 5

HOP
HOPFIELD NETWORKS
• The Hopfield network implements a so-called content
addressable memory.
• A collection of patterns called fundamental memories is
stored in the NN by means of weights.
• Each neuron represents a component of the input.
• The weight of the link between two neurons measures
the correlation between the two corresponding
components over the fundamental memories. If the
weight is high then the corresponding components are
often equal in the fundamental memories.

NN 5 3

HOP

ARCHITECTURE: recurrent

Multiple-loop feedback system

with no self-feedback
z-1

z-1

z-1

unit-delay operator

NN 5 4

Neural Networks 2
NN 5

HOP
Hopfield discrete NN
• Input vectors values are in {-1,1} (or {0,1}).
• The number of neurons is equal to the input dimension.
• Every neuron has a link from every other neuron (recurrent
architecture) except itself (no self-feedback).
• The neuron state at time n is its output value.
• The network state at time n is the vector of neurons states.
• The activation function used to update a neuron state is the
sign function but if the input of the activation function is 0
then the new output (state) of the neuron is equal to the old
one.
• Weights are symmetric: w = w
ij ji

NN 5 5

HOP
Notation

• N: input dimension.
• M: number of fundamental memories.
• f µ i i-th component of the µ fundamental
memory.
• xi (n ) State of neuron i at time n.

NN 5 6

Neural Networks 3
NN 5

HOP
Weights computation

1. Storage. Let f1, f2, … , fM denote a known set of N-

dimensional fundamental memories. The synaptic weights of
the network are:

1 M


w ji =  M
∑
µ
fµ
=1
,i fµ, j j ≠ i, j = 1.......N
0 j =i

where wji is the weight from neuron i to neuron j. The elements
of the vectors fµ are in {-1,+1}. Once computed, the synaptic
weights are fixed.

NN 5 7

HOP
NN Execution
2. Initialisation. Let x probe denote an input vector (probe) presented to the
network. The algorithm is initialised by setting:
x j (0) = x probe , j j = 1, ... , N
where xj(0) is the state of neuron j at time n = 0, and x probe,j is the j-th
element of the probe vector x probe.

3. Iteration Until Convergence. Update the elements of network state vector

x(n) asynchronously (i.e. randomly and one at the time) according to the rule

N 
x j (n + 1) = sign ∑ w ji xi (n ) j = 1, 2, ... , N
 i =1 
Repeat the iteration until the state vector x remains unchanged.
4. Outputting. Let x fixed denote the fixed point (or stable state, that is such

that x(n+1)=x(n)) computed at the end of step 3. The resulting output y of the
network is:
y = x fixed
NN 5 8

Neural Networks 4
NN 5

HOP
Example 1
(-1, 1, 1) (1, 1, 1)
weight
1
(1, 1, -1)
+
- (-1, 1, -1)
+
-

-
2 3
(-1, -1, 1)
-
(1, -1, 1)
neuron

(-1, -1, -1) (1, -1, -1)

stable states
-1 1 -1

-1 -1 -1 1 1 -1 -1 1 1
attraction basin 1 attraction basin 2
1 -1 -1 -1 -1 1 1 1 1

1 -1 1

NN 5 9

HOP
Example 2
• Separation of patterns using the two fundamental memories
(-1 -1 -1) ( 1 1 1):
• Find weights to obtain the following behavior:

-1 -1 -1

-1 -1 1 -1 1 -1 1 -1 -1
1

-1 1 1 1 -1 1 1 1 -1

1 1 1 2 3

wij = ?

NN 5 10

Neural Networks 5
NN 5

HOP
CONVERGENCE
• Every stable state x is at an “energy minimum”. (A
state x is stable if x(n+1)=x(n))

What is “energy”?
Energy is a function (Lyapunov function):
E: States → 
such that every firing (change of its output value) of a
neuron decreases the value of E:

x → x’ implies E(x’) < E(x) E ( x ) = − 12 x T W x

= − 12 ∑ w ji x i x j
i, j
NN 5 11

HOP
CONVERGENCE

• A stored pattern establishes a correlation between pairs of

neurons: neurons tend to have the same state or opposite
state according to their values in the pattern.
• If wji is large, this expresses an expectation that neurons i
and j are positively correlated similarly. If it is small (very
negative) this indicates a negative correlation.

• ∑w x x
i, j
will thus be large for a state x, which is a stored
ij i j

pattern (since wji will be positive if xi xj > 0 and negative if xi

xj < 0).

• The negative of the sum will thus be small.

NN 5 12

Neural Networks 6
NN 5

HOP
Energy Decreases
Claim:
Firing a neuron decreases the energy E.

Proof:
Let l be the neuron that fires:
Either:

1. xl goes from -1 to 1 which implies

N

∑w
i =1
x >0
li i
(above threshold)

2. xl goes from 1 to -1 which implies

N

∑w
i =1
x <0
li i
(below threshold)
N
Thus in both cases: ( xl '− xl ) ∑ wli xi > 0
i =1
value after
NN 5 13

HOP
Proof

Now we try to “factor” xl from the energy function E:

N N
E = − 12 ∑ ∑ w ji x i x j
i =1 j =1
N N pulled out the i = l case
= − 12 ∑ x i ∑ w ji x j
i =1 j =1
N N N
= − 12 xl ∑ w jl x j − 12 ∑ x i ∑ w ji x j
can be added j =1 i =1 j =1
since wll = 0 j ≠l i ≠l
N N N N
= − 12 xl ∑ w jl x j − 12 ∑ x i wli xl − 12 ∑ x i ∑ w ji x j
j =1 i =1 i =1 j =1
j ≠l i ≠l i ≠l j ≠l
N
= − xl ∑ w jl x j + Term independent of xl
j =1
j ≠l
NN 5 14

Neural Networks 7
NN 5

HOP
Proof

How does E change with xl changing?

E = value of E before change
E ' = value of E after change
N N
E ' − E = − x l ' ∑ w j l x j + x l ∑ w jl x j
j =1 j =1
j ≠l j ≠l
N
= − ( xl ' − x l ) ∑ w j l x j
j =1
j ≠l

we showed in previous slide that this quantity is > 0

(without ‘-’ sign)
So E’-E < 0 always.

NN 5 15

HOP
Convergence result

• We have shown that the energy E decreases with each

neuron firing.
• The overall number of states is finite ⊆ {+1, -1}N.
Then
• The energy cannot decrease forever.
• Firing cannot continue forever.

NN 5 16

Neural Networks 8
NN 5

HOP
Computer experiment

• We illustrate the behavior of the discrete Hopfield network as a content

addressable memory.
• n = 120 neurons (⇒n2 - n = 14280 weights).
• The network is trained to retrieve 8 black and white patterns. Each
pattern contains 120 pixels. The inputs of the net assume value +1 for
black pixels and -1 for white pixels.
• Retrieval 1: the fundamental memories are presented to the network to
test its ability to recover them correctly from the information stored in
the weights. In each case, the desired pattern was produced by the
network after 1 iteration.

NN 5 17

Computer experiment

These patterns are used as fundamental

memories to create the weight matrix.

NN 5 18

Neural Networks 9
NN 5

Computer experiment

• Retrieval 2: to test error-correcting capability of the

network. A pattern of interest is distorted by
randomly and independently reversing each pixel
of the pattern with a probability of 0.25. The
distorted pattern is used as a probe for the
network.
• The average number of iterations needed to recall,
averaged over the eight patterns, is about 31. The
net behaves as expected.

NN 5 19

Computer experiment

NN 5 20

Neural Networks 10
NN 5

Computer experiment

NN 5 21

Spurious attractor states

• Problem of incorrect retrieval. For example the network with

input corrupted pattern “2” converges to “6”.

• Spurious attractor states: the next slide shows 108 spurious

attractors found in 43097 tests of randomly selected digits
corrupted with the probability of flipping a bit at 0.25.

NN 5 22

Neural Networks 11
NN 5

Spurious attractor states

Combination of digit Local minima of E not

1, digit 4 and digit 9 correlated with any of
the fundamental
memories

NN 5 23

Storage Capacity

• Storage capacity: the quantity of information that can be

stored in a network in such a way that it can be retrieved
correctly.
• Definition of storage capacity:

number of fundamenta l patterns

C=
number of neurons in the network

number of fundamental patterns

C=
number of weights in the network

NN 5 24

Neural Networks 12
NN 5

Storage Capacity: bounds

• Theorem: The maximum storage capacity of a discrete

Hopfield NN is bounded above by

M 1
C ≡ =
N 4 ln N

• That is, if β is the probability that the j-th bit of the m-th
fundamental memory is correctly retrieved for all j =1,..,N
and m = 1,…,M, then

lim N → ∞ β = 1 whenever M < N/ ( 4 ln N )

NN 5 25

Hopfield NN for Optimization

• Optimization problems (like Traveling Salesman)

can be encoded into Hopfield Networks

• Objective function corresponds to the energy of the

network

• Good solutions are stable states of the network

NN 5 26

Neural Networks 13
NN 5

TSP
• Travelling Salesman Problem (TSP):
Given N cities with distances dij .
What is the shortest tour?

NN 5 27

Encoding
• Construct a Hopfield network with N2 nodes.
• Semantics: nia = 1 iff town i is visited at step a
• Constraints:

∑n
i
ia = 1, ∀a ∑n
a
ia = 1, ∀i

The town distancies are encoded by weights, i.e.

w ijab = d ij
NN 5 28

Neural Networks 14
NN 5

Hopfield Network for TSP

place
city 1 2 3 4
A 0 1 0 0
B 1 0 0 0
C 0 0 0 1
D 0 0 1 0

Configuration not Configuration allowed

allowed

NN 5 29

Energy and Weights

1
E=− ∑ wijab nia n jb
2 i , j , a ,b
• Nodes within each row ( i=j ) connected with
weights wijab = −γ

• Nodes within each column ( a=b ) connected

with weights wijab = −γ

• Each node is connected to nodes in columns

left and right with weight wijab = − d ij
NN 5 30

Neural Networks 15