Dynamic Neural Networlks: An Overview

N.K. Sinha*, M.M. Gupta** and D. H. Rao***

*McMaster University, Hamill on, Canada **University of Saskatchewan, Saskatoon, Canada ***Gogte Institute of Technology. Belgaum, India

ABSTRACT
Over the last decade several advance< havc been made in the p d d i g m of artificial neurd d networks with specific emphasis on architectures a learning algorithms. However, most of the work is fixu.stxi on static ( f d o n m d ) neural networks. These neural networks respond instantaneously to the inputs, for they do not posses any time delay units. The use of time delays in neural networks is neurobiologically motivated, since it is well k m w n that signal delays arc omnipresent in the brain aal play an important role in neurobiological information processing. This conccpl has led to the development of dynamic neural networks. It is envisaged that dynamic neural networks, in addition to better represcntation of biological neural systems, offer better L omputationdl capabilities compared to their static counterparts. The objective of this paptx is to give an overview of dynamic neural structures.
1.

the network. This feedback implies that the network has local memory characteristics. The node equations in dynrmic networks are described by difhentd o r diffkwnce equation. Because of feedback paths fiom their outputs to the inputs, the response of dynamic neural networks is recursive. That is, the weights are adjusted, the output is then recalculated, and the prwem is repeated. For a stable network, successive iterations prtxluce smaller and smaller output changes until eventually the outputs become constant. Neural n e t w o r k s with feedback are particularly appropriate for system modeling, identification, control and filtering applications. h e These networks are important h u s e many of t systems that we wish to model in the real w r l d are non-linear dynamical systems. This is true, for example, in controls area in which we wish to model the fimvard or inverse dynamics of systems such as airplanes, rockets, sqmceLratl and robots 12 - 51. The dynamic neural structures, in general, can be classified into two categories. The first category encompasses the dynamic neural stnrctures dcvelopd based on the concept of single neuron dynamics as an extension of static neural networks. The " T e n t neural network developed by Hopfielld 161, Brain-State-in-a-Box developed by Anderson et i d 171, the-delay neural network &vdoped by Lang, Waibel and Hinton 181, dynamic neural unit (DNII) developed by Gupta and Rao 1 9 1 belong to thiis category. The .wmd category encompasses dynamxc neural :itructures which are developed based on the interactionof excitatoryand inhibitory or antagonistic e u r a l structures developed neural !;ubp)pulations. N by Wikon and Cowan [IO], Gupta and Knopf [ll], and Rao and Gupta 1121 belong to this category.

Introduction

Most of the neural network structures USXI prcsently for engineering applications are static (fimifiward) neural networks. These n e u r a l networks, having a number of neurons, respond instantaneously to the inputs. 'This form of static input-output mapping is well suited for pattern recognition applications wherc both the input vector x and the output vector y represent spatial patterns that are independent of time. The absence of fixxihack in static mural nctwtwks ensur&\ that networks an: conditionally stahlc:. 1 lowever, these networks svfier from the following Imitations 11 1: (i) In Wfiwd networks the information flows form Neuron A to H, to C, to D, a d never comes hack to A. (ii) The structure of an artificial neuron is not dynamic in nature and performs a simple summation operation, and (iii) The static neuron model does not take into account the time delays that affect the dynamics of the system. Time delays afe the inherent chracmtics of biological neurons. Unlike a static neural network, a dynamic neural network employs extensive fAiiiick betwen the neurons of a layer, andor between the layers of

The objective of this paper is to describe the above nientioned dynamic structures in brief. In view of this, the paper is organizedas follows. The first category of dyllarmc neural nehmks is described in Section 2. The second category, namely; dynamic networls based on neural subpopulations, is desQibed in Section 3, followed by the conclusions in the last section.

0-7803-5812-0/00/$10.0092000IEEE

491

2. Single Neuron Dynamics

2.1 Recurrent Neural Networks
The feedback, also known as recurrent, neural networks, intnduced by Hopfield, is one of the first dynamic neural network models. This model consists of a single layer Network N1, included in a feedbackconfiguration with a time delay as shown in Fig. 1. this figure, y(k) and y(k+l) represent the states of the neural network at instants k and k+ 1, xo
represents the initial value, W(k) &notes the vector

feed back upon themselves. The BSB operates by using the built-in positive ftuxlback to amplify an input pattern, until all neurons in the structure m driven into saturation. From the dynamics p i n t of view, the BSB model can be viewed as a discrete linear system in a saturated mode. The main difference between the BSB and a usual disaete m system is that the linear system is defined on % , while the BSB mtdel is &find on the closed ndimensional hypercube 131. The BSB model is shown in Fig.2, and is defined by the following set of equations.

of the neural weights, Yr I.] is the nonlinear activation function, ami z-l represents the unit delay operator. This feedback network represents a discxete-time dynamical system and can be & b e d by the following equation [R

where N1 [.I = Y

W XI . Given an initial value xo,

the dynamical system evolves to an equilibrium state if NI is suitably chosen. The set of initial conditions

in the neighborhood of ~0 which converge to the same equilibrium state is then identified with that
state. The term "associative memory" is used to Hopfield neural model has been widely u s e d

describesuch systems.
in many applications such as system identification ami control, robotics, machme vision, pattern
recognition, associative memories, combinatorial optimization problems. For mathematica! tractability, it is aSSumed that Hopfield nemal structure contains only single layer of neurons. Recently, however, multi-layered recurrent neural networks have been developed aml used for many applications. Furthermore, inputs, when exist, arc assumed to be constant 151. In spite of the interesting applications that recurrent neural network.. have been used for, the basic architecture of the neuron is static; that is, the neuron simply provides a w i g h t 4 integration of the synaptic inputs over a period of time. In other words, there are no dynamical elements within the structureof the neuron.

where W demtes a symmetric weight matrix, x(k) is the state vectorof the model at discrete time k, V1.1 is a piecewise linear function and [3 is a small positii e constant called the fedback factor. The BSR model functions as follows [31: An activation pattern x(0) is input into the BSB model as the initial state vector, and Eqn. 2(a) is used to compute y(0). Equation 2(b) is then used to truncate y(O), obtaining the updated state vector x( 1). This pmcedure is repeated until the BSB mtdel withes a stable state represented by a particular comer of the unit hypercube. Positive fixxiback on the HSR mtxlel causes the initial state vector x(0) to increase in Euclidean length with an increasing number of iterations until it hits a wall of the box (unit hypercube), then sliding along the wall and eventually ending up in a stable corner of the
box.
A natural application for the BSI3 nitdel IS clustering. This follows from the fict that thc stable corners of the unit hypercuh act as point attractors with well-behwed basins of attraction. Consequently, the BSR mtxicl niay he u s d as ail unsupewised clustering algorithm, with cach stakk: corner the unit hypercube represcnting a c.lu\ter qd relatecl data.

2 . 3 l'ime-Delay Neural Networks ('I'DNN)

It is possible to use a static network to process timc: series data by simply converting the tcrnporal !qucncc into a staic: pattern by utth~lding the sequence ovcr time. From a practical point 01 view, it is possible to unfold lhe sequence over a finite period of time 1 2 1 . '[his can be accomplished by feeding the input sequence into a tapped delay line into a static neural network architecture. An architecture like this is often ref& to as a 'rime-. Delay Neural Network ('I'DNN) 171. It should be noted, however, that the 'I'DNN is a j&&+hnwtd

2.2 Brain-State-in-a-Box Model

(BSB)

Neural

The Brain-State-in-a-Box (BSB) neural model, devdopwl by Anderson etal 171, is a positive feedback system with amplitude limitation. It consists of a highly interconnected set of neurons that

492

network it attains dynamic behavior by virtue of the fact that each synapse of the n e m k is designed as an FIR filter. For more details with regard to mathematical development and learning algorithm,
r e z a d e n s a r e ~ t [3]. o

neuron as the basic computing unit for dexribing neural i n f o " processing operations. Each computing unit in the network is based on an idaalivd neuron. An ideal neuron is assummi to respond optimally to the applied inputs. However,
experimental studies in neuro-physiology show that the response of a biological neuron appear random [7J, a d only by averaging many observations is it possible to obtain predictable results. However, mathematical analysis has shown that these random cells can transmit reliable information if they are sufficiently redundant in numbers. The total neural activity g e m d t d within a tissue layer is a result of spatially Itxalizxl assemblies of densely interconnected nerve cells Calild neumi' population, or neural mus.\. The neural population is comprised of neurons, and its properties have a generic resemblance to those of Wvic'lual neurons. But it is not identical to them, and its properties can not be predicted f r o m measurements on single neurons. This is due to the fact that the properties of neural population depend on various paranems of individual neurons a d also depend upon the interconnections hetween neurons 1101. The study of neural networks based o n single-neuron analysi\ precludes the above two facets of biological neural struclurw. The conceptual gap between the functions of single neurons and those of numbers of neurons, a neural mass, is still very wide. Fach neural population inay be hither divided into several coexisting suhpopulatiotis. A subpopulation contains a large class of similar nt:urons that lic in close spatial proximity. The most coininon neural mass is the mixturc o f excitaloty (posi tivc) and h1hihifor-y(negative) subpopulations of neurc:)ns. 'ne excitatory neural subpopulation increases the electro-chemical potential of the postsynaptic neuron, while the inhihitory suhpipulation d u c the ~ electctro-chemical potential. The mini rnuni topoloby of such a neural mass contain! cxcitiittny (positive), inhihitory (negative), excitatory inhibitory (synaptic connection from excitattny to inhihitory), and inhibitory - excitattny (synaptic conrmtion from inhibitory to excitatory) lidback loops. B d on this hypothesis, Gupb and K.nopf prqmed a neural model n a d P-N neural procmsor I I 1 I for machine vision applications, and Rao anl Gupta proposed dynarnic neural processor (DNP) fix robotics and control applications 1151. A brief dmxiption of these two dynamic neural structures is given below.

The T D N neural strucutt: has been used in many applications. For example, Sejnowski and R0S-g [13] used TDNN for t e x t - t O - F h conversion, while Waibel et a1 [141 employed this structure for phoneme recognition. Narendta and partbasarthy [5] have applied this type of neural structure for system identification and control of nodinea~ dynamical systems.

2.4 Dynamic Neural Unit (DNU)

The dynapic m m l unit (DNU), proposed by Gupta and Rao (61, is a dynamic model of the biological neuron. It consists of a seconl-order dynamics whose output constitutes the argument to a time-varying nonlinear activation function. Thus, the DNU performs two distinct operations: (i) the synaptic operarion and (ii) the somatic opemtion. The first operation cOrreSpOnds to adaptation of i k d h m d and feedback synaptic weights, while the second COrreSpOMiS to adaptation of gain (slope) of the nonlinear activation function. The DNU consists of M a y elements, feedand f d h k paths weighted by the synaptic weights aff and hfi respectively representing a xxond-txder structure followed by a nonlinear activation function as shown in Fig. 3. In DNU, vl(k) is the output of' the dynamic structure, u(k) is the neural output, and aff

= [ao, a,, 5 1 and bfb = Ibl, b21 are the vectors of adaptable tbdf"d and fedback weights respectively.The nonlinear mapping operation on vl(k) yields a neural output U@)given by:

where Y [.] is some nonlinear activation function, usually the sigmoidal function and gs is the somatic

gain which controls slope of the activation function. Details of this neural structure, the development of learning algorithm and its application to the control of unknown nonlinear dynamic systems are discussed
in [9].

3. Dynamic Neural Structures Based o n Neural Subpopulations

The neural network structures kscribwi in the earlier section consider the behavior of a single

3.1

P-N Neural Processor

The computational role performed by the processor, named the Positive-Negative (PN) r i e d

493

processor, emulates the spatio-temporal information processing capabilities of certain neural activity fields found along the human visual pathway. The statespace model of this visual information prmessor correqwnds to a bi-layered two-dimensional array of densely interconnected nonlinear processing ekmenrs (PES). An individual PE represents the neural activity exhibited by a spatially localized subpopulation of excitatory or inhibitory nerve cells. A for the statespace model of the PN m a l processor is shown in Fig. 4.
~~

The functional dynamics exhibited by a neural computing unit, the DNU, is defined by a second-order difkmce equation. The state variables uE(k+l) and U (k+l), generated at time (k+l) by the I excitatory and id&&axy n e d u n i t s of t h e proposed neural processor, are modeled by the nonlinear functional relationships:

Two types of inputs are received by the constituent PES. One type arises from the external inputs originating in the signal space S(k), and the second type originates from the lateral and recurrent connections between the PES within a common network X(k). The spatial transformatir:n of the external input signal is given by the matrix A of the fe@brwud subnet, and the strength of lateral a d recurrent connections between individual PES is given by the matrix W of the fedbucksubnet. A variety of information processing operations asstxiated with the early stages of machine visior can be realized by the basic PN new& processor architecture. These diverse operations a~ achieved by selectively programming the system matrices A and W, and the parameters of the nonlincar mapping opmtor @[-I. The coefficients of A in the fkedfiwd subnet are made to act as a linear spatial filter that either smooth or enhance features embedded within the external input S(k). The cwfficient- wlcctd for W ili the feedback subnet, in conjunction with properly chosen parameters for @I- 1, will determine the computational role of the processor in terms of spatio-temporal filtering (STF), motion detection, short-term visual mcmory (STVM), spatiotemporal stabilization (STS) and pulse fi.txluency modulation (PFM).

where v (k) and v{k) represent the proption of

neurons in the neural unit that m i v e inputs gmter than an intrinsic threshold, and E and I represent the nonlinear excitatory and inhibitory actions of the neurons. The neurons that w i v e inputs greater than a threshold value is given by a nonlinear function of vh(k), Y [vh(k)] . The total inputs incident on t h e excitatory and inhibitory neural units are respectively

3.2 Dynamic Neural Processor (DN P)

The dynamic neural processor (DNP) is developed based on physiological evidence that nervous activity of any complexity depends upon the interaction of the excitatory and inhibitory neural subpopulations. The DNP thus functionally mimics the aggregate dynamic properties of a neural population or neural mass. The DNP comprises of two basic nodes called the dynumic nczrrul units (briefly desctihed in the preceing subsection) and arr: coupled in excitatory and inhibitory modes. The DNU embodies delay eloments, tixxlforward a d fixxlkick weights tblloweci by a time-varying nonlinear activation function; thus different from the conventionally assumed structure of neurons. The morpholoLy of DNP is shown in Fig. 6.

where w and w are the weights associated with the E I excitatmy and inhibitory neural inputs respectively, w and w represent the self-synaptic connection EE n strengths, w and w represent the inter-neuron 1E E1 synaptic strengths, and 8 and 8 repsent the E I thresholds of excitattny and inhibitory mumm respectively. The DNP has been used for computational purposes, such as functional approximation, computation of inverse kinematic transformations of robots and control of unknown nonlinear dynamic systems [ 151.

4. Conclusions
Static neuron model used in artificial I networks ignores many of the characteristics of its biological counterpart, and is a grossly simplified version. As the name indicates, static (feedfhmd)

494

neural network? do not incorporate feedback On the other hand, dynamic neural networks employ extensive f d b k between the neurons of a layer, and/or beheen the layers of the network. This feedback implies that dynamic neural networks have local memory characteristics. The node equations in dynamic networks are described by differential or difference equations. Neural networks with feedback are particularly appropriate for system modeling, identification, control and filtering applications. Recently, many researchers are focusing their efforts in developing dynamic neural structures and dyraarmc learning algorithms. In this p w we have presented an overview of dynamic neural structures.

KJ. Lang, A.H. Waibel and G.E. Hinton,

"A Time-Delay Neural Network Architecture

f r o m Isolated Word Recognition", Neud Networks, Vol. 3, No. 1, pp. 2 3 4 , 1 9 9 0 .

M.M. Gupta and D.H. b o , " m c Neural Units With Applications to the Control of Unknown Nonlinear Systems", Ihe Joumal o f Intelligent and Fuzzy Sy,stems, Vol. 1, No. 1, pp. 73-92, Jan. 1993. H.R. Wilson and J.D. Cowan, "Excitatory ami Inhibitory htemctions in Locaked Populations of Model Neurons", Biophysiml J a ~ t n a lVol. , 12, pp. 1-24,1972. M M. G q t a and G.K. Knopf, "A Multitask Visual Infurmation Processor with a Bilologically Motivated Design", J m d of Vimal Communication and Image Representarion, 3, No. 3, pp. 230-246, Sept. 1992. D.H. Rao and MM. Gupta, "Functional Approximation Using Dynamic Neural

References
J. J. Hopfield, "Artificial Neural Networks are Coming", Z E E E w e n , An Intewiew by W. Myers, pp. 3-6, April 1990. D.R. Hush and B.G. Home, "Progress in Supervised Neural Networks", ZEEE Signal Processing Magazine, January, Volume 10, NO. 1, pp. 8-39, 1993.
S . Haykin, Neural Networks: A Comprehensive Foundation, Z Z E Press. 1994.

MM. Gupta and D.H. Rao, "Neuro-Conlrol

Systems: A Tutorial", in NeuroControl Systems: Theory and Applications, Eds., M.M. Gupta and D.H. b o , pp. 1 - 43, IEEE Press, March 1994.

K.S. N a " and K. Parthamthy, "Identification and Control of Dynam~cal Systems Using Neural Networks", ZEEE Trans. on N e u r a l Networks, pp. 4-27, Vol. 1, No. 1, March 1990.
Hopfield, "Neurons with (;lraded Response Have Collective Computational Properties Like of Those Two-State Neurons", Proceedings of the Narional Academy of Sciences, Vol. 81, pp. 3088-. 3092,1984. J.A. Andason,J.W. Silverstein, S.A. Ritz and R.S. Jones, "Distinctive Features, Categorical Perception, and Probability Laming: Some Applications of a Neural Model", Psychological Review, vol. 84, pp. 413-451, 1977.

Pr~xesm", Inte"d Joltmal of hhml Mhs-Pamllel Computing and Informath systems, Vol. 5 , pp. 573-592,1994.

T. Sejnowski and C.R. Rosenberg, "NETtalk:A Neural NetworkThat Learns t o Relad Aloud", Tech. Report JHWEECS86/01, John Hopkins University, 1986. A. Waibel, T. Hanazawa, G. Hinton, K. and KJ. Lang, "Phoneme Recognition Using Time-Delay Neural Networks", IEEE l).ans. on Acoustics, Speech andsignal Processing, Vol. 37, No. 3, pp. 328-339, March 1989.

JJ.

Shikano

M.M. Gupta and D.H. bo, "Dynamic Nwral Processor and Its Applications to Robotics and Control", in Intelligent Control, ZEEE Z-ks, Eds., M.M. Gupta and N.K. Sinha, pp. 515-545,19%.

495

Static nard n c l w r k

lk)
Fig. 2 Block diagram oflhc 0SB modcl

Fig. 4: Thc Y N ncunl pnrcssor.

Fig. 5 : 'Ilic dyriamic ricuml prwcssor (DN1').

496