Department of Mechanical Engineering

MAK 660E INTELLIGENT SYSTEMS & SOFT COMPUTING

HOPFIELD NEURAL NETWORK

INTRODUCTION John J. Hopfield published two papers about an associative model that is one of the

milestones in the field of neural networks. This neural network needs no synchronization and it is very successful at optimization and pattern recognition. Many of ideas in these papers were based on the previous work of other researchers. These researchers had considered more general associative memory models in the 1970s, but by restricting the architecture of the network to a symmetric connection matrix with a zero diagonal, it was possible to design recurrent networks with stable states. In addition to this there are several other reasons why Hopfields papers have had such an impact. First, he identified a close analogy between his neural network and the Ising model of magnetic material. Hopfield also had close contacts with VLSI chip designers, because of the bell laboratories. The fact that Hopfield addressed the implementation issues of his network distinguished him from most previous neural network researchers. 1.1 y Terminology: Symmetric connections: if the weight between neurons doesnt change it is a

symmetric connection. y Asymmetric connections: if the weight between neurons doesnt change it is a

symmetric connection.

Symmetric and Asymmetric Connections y Asynchronous: when firing a neuron, first calculates the weighted input sum and

updates immediately. This can be done in a fixed order, or neurons can be picked at random, which is called asynchronous random updating. y Synchronous: the weighted input sums of all neurons are calculated without updating

the neurons. Then all neurons are set to their new value, according to the value of their weighted input sum. y Associative memory: Recalling a previously experienced item by thinking of

something that is linked with it, thus invoking the association. For example when we see red

light we understand it as stop and green as go. So colors are reminds us the commands. This is a associative memory. y Ising model: this physical system consists of particles that capable of adopting one of

two states. In the case of ferromagnetic materials, their atoms can be modeled as particles of spin 1/2 (up) or spin 1/2 (down). The spin points in the direction of the magnetic field. All tiny magnets interact with each other. This causes some of the atoms to flip their spin until equilibrium is reached and the total magnetization of the material reaches a constant level, which is the sum of the individual spins.

Ising model 2 2.1 HOPF ELD NEURAL NETWROK A physical analogy with memory To show the similarity between the energy and the memory, we can think a bowl in which a ball bearing is allowed to roll freely as shown in figure below. Suppose we let the ball go from a point somewhere up the side of the bowl. The ball will oscillates around bottom of the bowl and after some oscillations ball will be rest at the bottom. We can describe what happened, in terms of energy. At first, system have a potential energy according to its height, higher the ball the greater its energy.

Eventually the ball comes to rest at the bottom, energy is now at a minimum since any other location of the ball is associated with some potential energy, which may be lost on allowing the bowl to reach equilibrium. This equilibrium state doesnt change with the starting point. There is another way of thinking about this process that ties in with our ideas about memory. This example illustrated to understand the relation. Thus, we suppose that the ball comes to rest in the same place each time because it remembers where the bottom of the bowl is.

If we now use a corrugated surface instead of a single depression we may store many memories. If the ball is now started somewhere on this surface, it will eventually come to rest at the local depression that is closest to its initial starting point. That is, it evokes the stored pattern which is closest to its initial partial pattern or cue. This corresponds to an energy minimum of the system. The memories shown correspond to states x1, x2, x3 where each of these is a vector. There are therefore two complementary ways of looking at what is happening. One is to say that the system falls into an energy minimum; the other is that it stores a set of patterns and recalls that which is closest to its initial state. 2.2 Hopfield network Now we are going to apply principle we talk above, to the construction of a neural network capable of performing associative recall. Consider the network consisting of three nodes shown below. Each node is connected to every other node and the connection weights are symmetric. Thresholds are all assumed to be zero. Notice that the flow of information in this type of net is not in a single direction, since it is possible for signals to flow from a node back to itself via other nodes. We say there is feedback in the network or that it is recurrent because nodes may be used repeatedly to process information.

The state of the network at any time is given as (x1, x2, x3). Suppose we now start this net in some initial state, choose a node at random and let it update its output or fire. That is, the chosen node evaluates its activation in the normal way and outputs a 1 if this is greater than or equal to zero and a 0 otherwise. The net now finds itself either in the same state as it started in, or it changes one of the states. We now choose another node at random, let it update or fire, and repeat this many times. This process defines the dynamics of the net. In describing these state transitions it is convenient to attach a numeric label to each state and the most natural way of doing this is to interpret the Boolean state vector as a binary number, so that state (x1, x2, x3) is labeled with 4x1+2x2+x3. For example, (1, 1, 0) is state 6. For each network state, there are three possible outcomes for the next state depending on which of the three nodes is chosen to fire. Suppose, for example, the net starts in state (1, 0, 1) (label 5) and node 1 fires. The activation of this node is given by a=w13x3+w12 x2=21+10=2. Then, since this is less than 0, the new output is also 0 and the new state is (0, 0, 1) (label 1); in summary, state 5 goes to state 1 when node 1 fires. Repeating this working for nodes 2 and 3 firing, the new states are 7 and 4 respectively. By working through all initial states and node selections it is possible to evaluate every state transition of the net as shown as; x1 0 0 0 0 1 1 1 1 Initial State x2 0 0 1 1 0 0 1 1 x3 0 1 0 1 0 1 0 1 Label # 0 1 2 3 4 5 6 7 1 4 1 6 3 4 1 6 3 After A Node Firing 2 2 3 2 3 6 7 6 7 3 1 1 3 3 4 4 6 6

Notice that a state may or may not change when a node fires. This information may also be represented in graphical form as a state transition diagram, shown in figure below;

For example, starting in state 5 we see from the diagram that there is an equal probability of 1/3 of going to states 1, 7, or 4, which is reflected in the three arcs emanating from state 5 in the diagram. Again, starting in state 1 and updating nodes 1 or 3 results in no state change, so there is a probability of 2/3 that state 1 stays as it is; however, choosing node 2 to update results in a transition to state 3. The no-change condition gives an arc that starts and ends at the same state. Another firing method is that; classify the units of a network according to their state: the first set contains the units with state 1, the second set the units with state 1. There are edges linking every unit with all the others, so that some edges go from one set to the other. We now randomly select one of the units and compute its attraction by the units in its own set and the attraction by the units in the other set. The attraction is the sum of the weights of all edges between a unit and the units in its set or in the other one. If the attraction from the outside is greater than the attraction from its own set, the unit changes sides by altering its state. If the external attraction is lower than the internal, the unit keeps its current state.[74]

2.3

Defining an energy for the network The dynamics of the net are described completely by the state transition table or

diagram. It has demonstrated the existence of stable states. On the other hand energy based form is more accurate to find stable states. We can think every node effect other with the associated weight. The use of the energy concept in this context may be introduced using an analogy with a simple physical system. Thus, consider a pair of objects joined by a spring, use of a tension spring, which tends to pull the objects together. Since work has to be done to draw the objects apart, the energy minimum of this system occurs when the objects are close to each other or, in other words, tend towards the same position in space. Compression spring energy minimum occurs when the objects are far apart.

Now consider two nodes i, j in a Hopfield net connected by a positive weight +w, as shown above. We claim that positive and negative weights are analogous to the tension and compression springs respectively, since positive weights tend to make the nodes take on the same output values while negative weights tend to force different values. The constraint description may now be set within the energy framework by assigning high energies to states that tend to get destabilized and low energies to those that are reinforced. One way of doing this is to define the internode energy eij by

The energy of the whole network E is found by summing over all pairs of nodes

Since the connections are symmetric, the sum includes each pair twice so we put the coefficient. When we adding the threshold value for every node we obtain the energy formula for Hopfield neural network:

2.4

Finding the weights p to now nothing has been said about how the network can be made to store patterns

from a training set. In his original paper, Hopfield (1982) did not give a method for training the networks under an incremental, iterative process. But commonly to store pattern into the network biologically inspired Hebb training method is used. The main purpose of the Hopfield neural network is constraint the outputs of the network according to given weights. With this aspect, we can adjust our weights with the relations between them. For example, two nodes that tend, on average, to take on the same value over the training set so that the pairs 0, 0 or 1, 1 dominates; we say that the nodes are correlated. The pair 1, 1 will be reinforced by there being a positive internode weight. If, on the other hand, the two nodes tend, on average, to take on opposite values with pairs 0, 1 or 1, 0, then this will be reinforced by a negative internode weight. 2.4.1 The Hebb rule In 1949, neuropsychologist D. O. Hebb describes a possible way for biological neurons learn. And he said that: When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that As efficiency, as one of the cells firing B, is increased. This description can be adapted to our training algorithm in three steps: 1. Choose a pattern from the training set at random. 2. Present the components of this pattern at the outputs of the corresponding nodes of the net. 3. If two nodes have the same value then makes positive increment to the internode weight. If they have opposite values then make negative decrement to the weight. Iteration of these three steps many times constitutes a training algorithm whose learning rule may be written mathematically as;

Change in weight is only ever positive and only occurs if both nodes are firing (output 1). 2.5 Storage capacity

The storage prescription attempts to capture information about the mean correlation of components in the training set. As such, it must induce a weight set that is a compromise as far as any individual pattern is concerned. Clearly, as the number m of patterns increases, the chances of accurate storage must decrease since more trade-offs have to be made between pattern requirements. In some empirical work in his 1982 paper, Hopfield showed that about half the memories were stored accurately in a net of N nodes if m=0.15N. The other patterns did not get stored as stable states. In proving rigorously general results of this type, it is not possible to say anything about particular sets of patterns so that all results deal with probabilities and apply to a randomly selected training set. Thus McEliece et al. (1987) showed that for m<N/2logN, as N becomes very large, the probability that there is a single error in storing any one of the patterns becomes ever closer to zero. To give an idea of what this implies, for N=100 this result gives m=11. 3 THE EIGHT ROOKS PROBLEM Number of ways to place non-attacking rooks on a board that looks like a checkerboard; that is, no two rooks may be in the same row or column. The board is any subset of the squares of a rectangular board with m rows and n columns; we think of it as the squares in which one is allowed to put a rook. The board is the ordinary chessboard if all squares are allowed and m = n = 8 and a chessboard of any size if all squares are allowed and m = n. The rooks are arranged in such a way that there is no pair of rooks in the same row or column. In this sense, an arrangement is the positioning of rooks on a static, immovable board; the arrangement will (usually) be different if the board is rotated or reflected. For such a combinatorial problem we can derive a Simple Hopfield Neural Network. First we can think a chessboard with eight columns and eight rows. According to this set we can use a 16 node network for each place in chessboard to each node. The weights for the network are derived from the following considerations: Let xij represent the state of the unit corresponding to the square ij in the nxn board. The number of ones in column j is given by following function must be minimized: . If in each column only a single 1 is allowed, the

This expression can also be written as  States are binary so xi = xi2 ;   Our Hopfield energy function is;  energy function;       

And we can make some arrangements to our energy function to assimilate it to Hopfield

For every column we find the weight as a constant -2 and threshold value -1. The minimum of the function corresponds to the situation in which just one rook has been positioned in every column. Similarly, for the rows of the board we define the function E2 according to

  

  

 

We want to minimize the function E = E1 + E2. The general strategy is to reduce its analytical expression to a Hopfield form. The necessary algebraic steps can be avoided by noticing that the expression for E1 is the sum of n independent functions (one per column).

Network model seems like figure shown above, in the figure we can solve this problem for a 4 4 board. Each field is represented by a node. Only the connections of the first unit in the board are shown to avoid cluttering the diagram. The connections of each unit to all elements in the same row or column have the weight 2, all others have a weight zero. All units have the threshold 1. Any unit set to 1 inhibits any other units in the same row or column. If a row or column is all set to 0, when one of its elements is selected it will immediately switch its state to 1, since the total excitation (zero) is greater than the threshold 1. One of the solutions of this network is shown below. This solution is totally related to its initial starting conditions.

APPLICATIONS OF HOPFIELD NETWORKS As I mentioned before, when Hopfield was working for Bell industries, he proposed an

electrical realization of his model. He used amplifiers for Hopfield units and obtains complements of these units with inverters and fed back these to the inputs. Symbolic representation of this circuit is shown in figure. In this figure circles in contacts, are represented resistance and Hopfield adjust the weights with these resistance.

For example, contains a resistor with resistance r13 = 1/w13. The constants wij represent the weights of the Hopfield network between the units i and j. Inhibiting connections between one unit and another (that is, connections with negative weights) are simulated by connecting one inverted output of a unit to the other one. One of the most important implementation of Hopfield network is realized by optical way. The networks binary values are projected through the vertical lens to the left of the arrangement. The lens projects each value xi onto the corresponding row of an optical mask. Each row i in the mask is divided into fields which represent the n weights wi1, wi2 . . . win. on the mask, reflected light emitting or reflecting according to attended weight. The individual unit states are projected using light emitting diodes. The light going through the mask is collected by another lens and obtained a single value. The amount of light that goes through the mask is proportional to the product of xi and wij at each position ij of the mask.

In literature Hopfield neural network used for two purposes one of them is pattern recognition and the other is optimization problems. In pattern recognition, binary images are sampled and for each field a node will be attend. When image is stored to the network weights will be updated. For more patterns node number will be increase exponentially. For optimization problems can be written in an analytical form isomorphic to the Hopfield energy function, it can be solved by a Hopfield network. We can assume that every unit in the network is simulated by a small processor. The states of the units can be computed asynchronously by transmitting the current unit states from processor to processor. There is no need for expensive synchronization and the task is solved by a massively parallel system. This strategy can be applied to all those combinatorial problems for whose solution large mainframes have traditionally been used.

4.1

HNN in image processing Because of the usage of associative memory Hopfield neural network (HNN) finds

largely working area in image processing. In many studies Hopfield NN was used for image recognition such as hand writing recognition, face and gender recognition with genetic algorithm, edge detection, image segmentation, image restoration and image compression. In one study different characters are stored in HNN and when input is degraded and noise image HNN is enhance the image(it recognize the original image from the degraded image). Results are very successful with less recognized characters and under a fifty persent noise.

As seen in the results HNN is very successful in image recognition. But we shouldnt forget only binary images can be an input for Hopfield neural network. 4.2 Optimization problems As I mentioned before, if we can be write the given problem in an analytical form isomorphic to the Hopfield energy function, it can be solved by a Hopfield network. Hopfield and Tank were the first to try to solve the TSP using a neural network. But such a complex problem solved by HNN with binary neurons it strict a local minimum. So in 1984 Hopfield replaced binary neurons by neurons whose instantaneous activity increases with input and applied this to the TSP.

Results of the travelling sales man problem are shown above. In these figures first and second are continuous Hopfield model and the last one is the binary Hopfield model. As its clearly shown in the figure continuous HNN finds the global minimum but the binary HNN finds the local minimum. Heuristics have been proposed for the TSP, starting with the classical work by Kernighan and Lin. One of the first to deal with limits of the Hopfield model with constant size for the solution of the TSP was Abu-Mostafa. Although this almost meant the death of the traveling salesman, the Hopfield model and its stochastic variants have been applied in many other fields, such as psychology, simulation of ensembles of biological neural networks, and chaotic behavior of neural circuits. 5 CONCLUSION Hopfield NN is one of the most important milestones of recurrent networks. Some properties of this network is:  A recurrent network with all nodes connected to all other nodes  Nodes have binary outputs (either 0,1 or -1,1)  Weights between the nodes are symmetric wij = wji  No connection from a node to itself is allowed  Nodes are updated asynchronously (i.e. nodes are selected at random)  The network has no hidden nodes or layer Main advantages of this system are, do not need any kind of synchronization; they guarantee that a local minimum of the energy function will be reached. In pattern recognition HNN is very successful but with the increasing of the stored pattern number the network expands and the success of the network is decrease.

In simple optimization problems HNN with binary neurons can be used but it only finds local minimum. For more complex optimization problems like travelling salesman or rout planning we must use HNN with continuous neuron. There are a lot of areas that Hopfield used with different algorithms. But I think Hopfield can be very successful in classification studies. The studies in this task are much related and can be developed.

1) Hopfield J., 1982, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA 2) Rojas, R., 1996, Neural Networks A System Introduction, Springer-Verlag Berlin 3) Gurney, K., 1997, An Introduction to Neural Networks ,UCL Press, London 4) Hopfield J., 1985, Neural Computation of Decisions in Optimization Problems, Biological Cybernetics, Springer-Verlag 5) Boden M., 2001, A Guide to Recent Neural Networks and Backpropagation, Halmstad University